system

Abstract

We provide the system. [Solution] A sensor means for acquiring usage data, A data analysis means for analyzing data acquired from the aforementioned sensor means and learning the user's lifestyle patterns, Information and communication means for proposing optimal operating methods and energy management to the user based on the results learned by the aforementioned data analysis means, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, various household appliances are widespread, but many of them rely on user operations, and there is no efficient energy management or operation optimization adapted to the user's lifestyle. Therefore, it is required to reduce waste of energy consumption and labor of operation and improve the convenience for users.

Means for Solving the Problems

[0005] The present invention provides a system comprising sensor means for acquiring usage data, data analysis means for analyzing the data to learn the user's lifestyle patterns, and information communication means for proposing optimal operation methods and energy management based on the analysis results. This enables efficient operation of home appliances that suits the user's lifestyle, thereby optimizing energy consumption and improving user convenience.

[0006] A "sensor device" is a device used to acquire data on the usage status of home appliances and environmental information.

[0007] "Data analysis means" refers to algorithms and processing devices that analyze data acquired from sensor means and learn the user's lifestyle patterns.

[0008] "Information and communication means" refers to a communication device or interface that proposes optimal operating methods and energy management to the user based on the analysis results obtained by data analysis means.

[0009] "Usage data" refers to data that shows how home appliances are being used, including, for example, the time of use, frequency of use, and operating method.

[0010] "User lifestyle patterns" refer to the daily behavioral patterns of the user, such as when and how often they use home appliances.

[0011] "Operation suggestions" refer to the optimal device operation procedures and energy-saving settings suggested to the user based on analyzed usage data.

[0012] "Energy management" refers to the process of controlling and adjusting household appliances to use energy efficiently. [Brief explanation of the drawing]

[0013] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is constructed as a system equipped with sensor means, data analysis means, and information communication means. First, the sensor means built into the terminal acquires usage data of the home appliance in real time. This includes information such as usage time, operating mode, and ambient environmental conditions (e.g., room temperature and illuminance). This allows the terminal to transmit detailed usage data to the server.

[0035] Next, the server receives this data and analyzes it in detail using data analysis tools. By using machine learning algorithms, the server learns and models the user's lifestyle patterns. For example, it can recognize patterns in which users use specific home appliances at specific times of the day.

[0036] Furthermore, based on the analysis results obtained, the server generates suggestions for optimal operation methods and improvements to energy efficiency. These suggestions are transmitted to the terminal via information and communication means and notified to the user via the terminal's display or a smartphone app. For example, if the server detects a pattern of using the coffee maker at a certain time in the morning, it can suggest setting it to automatically turn on a little before that time.

[0037] As a result, users can reduce energy waste and improve convenience by performing the suggested actions. Furthermore, user selections and actions are recorded again via sensors and fed back into subsequent analysis. This allows the system to continuously learn from user behavior and further optimize its performance.

[0038] Thus, the system of the present invention is implemented as a smart home appliance solution that continuously evolves to match the user's lifestyle, achieving improvements in convenience and energy efficiency.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The device collects real-time usage data of home appliances through sensors. This data includes usage time, device operating status, and ambient environmental conditions.

[0042] Step 2:

[0043] The device sends the collected data to the server at regular intervals. This data is stored in a database for use in subsequent analysis.

[0044] Step 3:

[0045] The server performs initial data cleansing on the received data. It removes noise and outliers and formats the data in a way that is suitable for analysis.

[0046] Step 4:

[0047] The server feeds cleansed data to machine learning algorithms to analyze and model the user's lifestyle patterns. This allows it to recognize usage trends at specific times and under specific conditions.

[0048] Step 5:

[0049] The server generates operation suggestions tailored to the user based on the analysis results. This includes automatic setting of operation schedules aimed at improving energy efficiency and suggestions for energy-saving modes.

[0050] Step 6:

[0051] The server sends the generated operation suggestions to the terminal via information and communication means. The terminal notifies the user of the suggestions through the user interface.

[0052] Step 7:

[0053] The user can either accept the suggestion or make their own adjustments. The results of this operation are again recorded by the device's sensors and fed back to the server.

[0054] Step 8:

[0055] The server uses the new feedback data to update and improve the user's model, thereby improving the accuracy of future suggestions.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] Modern home electronic devices are diverse, and their efficient use and energy conservation are essential. However, it is difficult for users to recognize their own behavioral patterns and manually configure devices appropriately, often resulting in wasted energy. To solve this, a system is needed that automatically learns user behavior and suggests optimized operation.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes detection means for collecting usage information, information processing means for analyzing the information collected from the detection means and modeling the user's behavioral tendencies, communication means for proposing efficient operating methods and energy-saving strategies to the user based on the model obtained by the information processing means, and proposal generation means for generating proposals using a generated AI model. As a result, the user can achieve efficient and energy-efficient use of the equipment without having to recognize their own behavioral patterns.

[0061] "Usage information" refers to information such as time, mode, and environmental conditions related to the use of home appliances and other devices.

[0062] "Detection means" refers to a system that acquires usage information using sensors or other detection devices.

[0063] "Information processing means" refers to technologies and devices used to analyze collected usage information and model user behavior trends and patterns.

[0064] "Communication methods" refer to network technologies and devices used to deliver suggestions to users regarding energy management and operation methods based on analysis results.

[0065] A "generative AI model" refers to an algorithm that uses artificial intelligence to learn from data and generate new suggestions.

[0066] "Suggestion generation method" refers to a system that uses a generation AI model to suggest the optimal operating method according to the user's behavior and usage situation.

[0067] This invention is a system for efficiently managing the usage status of home appliances using terminals installed in each home. The terminals are equipped with various sensors for detecting temperature, illuminance, motion, etc., and these sensors collect usage information in real time.

[0068] The terminal transfers the collected information to the server. The server is equipped with information processing capabilities and uses machine learning algorithms to analyze user behavior trends and usage patterns from the collected data. This analysis makes it possible to model patterns such as the frequent use of specific home appliances during specific time periods.

[0069] Furthermore, the server utilizes the generated AI model to create optimal operation suggestions based on the model obtained from the information processing means. These suggestions are transmitted to the terminal via communication means and presented to the user through the terminal's display or an application installed on the user's smartphone. For example, if a user has a habit of using a coffee maker at a specific time every morning, the server can generate a suggestion to automatically turn on the power at that time.

[0070] This allows users to improve the convenience of their daily lives while saving energy simply by performing the suggested actions. Furthermore, the actions performed by the user are fed back through the terminal and sent to the server. This feedback information is used for subsequent analysis, further enhancing the overall system and improving its ability to provide suggestions tailored to individual users.

[0071] An example of a prompt message might be, "Based on my morning routine, please provide suggestions for improving energy efficiency." This system creates an environment where users can manage their home appliances efficiently and smartly without any hassle.

[0072] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0073] Step 1:

[0074] The device uses its built-in sensors to collect information on the usage status of home appliances. The input data includes environmental information such as the appliance's usage time, mode, room temperature, and illuminance. The device periodically scans this information and outputs the data in streaming format. Specifically, the device acquires information at regular intervals and temporarily stores it in local memory.

[0075] Step 2:

[0076] The terminal sends the usage information it collects to the server. The input data is the usage information output from the terminal. The terminal encodes this data as a digital signal and transfers it to the server using a secure communication protocol. Specifically, the terminal performs batch transfers when a certain amount of data is reached.

[0077] Step 3:

[0078] The server analyzes the usage information it receives using information processing tools. The input data is usage information sent from the terminal. The server uses machine learning algorithms to analyze the data and derive and output user behavior patterns. For example, it may extract patterns in which specific home appliances are frequently used during certain time periods.

[0079] Step 4:

[0080] The server generates suggestions using a generative AI model based on the analysis results. The input data is the user's behavior patterns derived from the analysis. The server inputs this data into the generative AI model and outputs suggestions for optimal operating methods and energy-saving measures. Specifically, the server compares various candidate settings and selects the most effective one.

[0081] Step 5:

[0082] The server generates suggestions and sends them to the terminal via a communication method, notifying the user. The input data is the generated suggestions. The server generates a notification message and outputs it to the user's display or smartphone app via the terminal. Specifically, a notification is made through the user interface based on the message content.

[0083] Step 6:

[0084] The user operates the device to take action based on the suggestions. The input data is the suggestions received via the device. The user reviews the suggestions and performs specific actions through the device's UI. Specific actions include pressing buttons to change the settings of home appliances based on the suggestions.

[0085] Step 7:

[0086] The device collects user behavior data again via sensors and feeds it back to the server. The input data is the actual result of the user's actions. The device analyzes the obtained data and completes the feedback process by sending the results back to the server. Specifically, the device acquires new data from the sensors and sends it to the server in batches.

[0087] (Application Example 1)

[0088] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] In modern industry, optimizing machine operation and resource management are crucial challenges. However, currently, many industrial machines are not operated efficiently based on actual usage patterns, potentially leading to energy waste and increased operating costs. To solve this problem, a system is needed that monitors machine operating status in real time and proposes efficient operating methods based on that data.

[0090] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0091] In this invention, the server includes a detection means for acquiring usage data, a mathematical analysis means for analyzing the data acquired from the detection means and learning the user's behavior patterns, an information and communication means for proposing optimal operating methods and resource management to the user based on the results learned by the mathematical analysis means, and a function of the information and communication means for analyzing the operating status of industrial machinery and generating suggestions to improve efficiency. This makes it possible to improve the operating efficiency of industrial machinery and reduce unnecessary energy consumption.

[0092] "Usage data" refers to information indicating the operating status of industrial machinery, including, for example, operating hours, operating mode, and environmental conditions.

[0093] "Detection means" refers to a device or method for acquiring real-time usage data of industrial machinery.

[0094] "Mathematical analysis means" refers to methods and techniques for learning machine operation patterns from acquired usage data and deriving efficient operating methods.

[0095] "Information and communication means" refers to communication technology or devices used to transmit to the user, based on analysis results, suggestions for optimal operation methods and resource management.

[0096] "Behavioral patterns" refer to the repetitive and predictable ways in which industrial machinery is used or exhibits certain tendencies under specific conditions.

[0097] "Resource management" refers to management methods for efficiently operating industrial machinery while minimizing the consumption of energy and materials associated with its use.

[0098] The system for realizing this invention is configured to monitor and optimize the efficient operation of industrial machinery. The server acquires real-time usage data of the industrial machinery using detection means. This includes the machine's operating time, operating mode, and ambient environmental conditions.

[0099] The acquired data is sent to a server and analyzed using mathematical analysis tools. The server uses machine learning algorithms to learn machine usage patterns and model efficient operating methods.

[0100] Based on modeled usage patterns, the system sends suggestions for optimal operation methods and resource management to the terminal via information and communication means, notifying the user. For example, if the server recognizes that operating a machine during a specific time period is energy-efficient, it can suggest shifting the operation to that time period.

[0101] This system allows users to perform suggested operations and reduce energy consumption. The user's results are then fed back to the server via a detection mechanism, and the model is updated by a mathematical analysis mechanism. Through this process, the system continuously learns and generates even more optimized suggestions.

[0102] As a specific example, in one manufacturing plant, daytime temperature increases were leading to decreased machine efficiency. By introducing this system, the server detected patterns and suggested energy-efficient nighttime operations. This effectively reduced the plant's energy costs.

[0103] An example of a prompt to input into the generative AI model is: "Analyze the operating patterns of factory robots and test and report the optimal suggestions for reducing energy consumption in a generative lab environment."

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The server acquires usage data from industrial machinery through detection means. This data includes information such as operating hours, operating mode, and environmental conditions. The acquired data is stored in a database and used as input for subsequent analysis processes.

[0107] Step 2:

[0108] The server inputs the acquired usage data into a mathematical analysis system and analyzes the data using machine learning algorithms. In this process, the server identifies machine usage patterns and generates a model to derive efficient operating methods. As a result, the characteristics of the main usage patterns are output.

[0109] Step 3:

[0110] Based on the generated model, the server sends suggestions for optimal operation methods and resource management to the terminal via information and communication means. For example, it may notify the terminal of energy-efficient operating hours. This allows the terminal to present specific improvement suggestions to the user.

[0111] Step 4:

[0112] The user executes the suggestions received through the terminal. The resulting usage data is then sent back to the server via the detection mechanism. This data is used as feedback and incorporated into the model update process.

[0113] Step 5:

[0114] The server analyzes the feedback data and updates the existing model using mathematical analysis tools. The updated model is used as the basis for future proposal generation. This allows the system to be continuously optimized and provide users with even more effective proposals.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] This invention combines a system that learns the user's lifestyle patterns and proposes optimal operating methods and energy management with an emotion engine that recognizes the user's emotions. This system comprises sensor means, data analysis means, information communication means, and an emotion engine.

[0117] First, the sensor built into the device collects data on the usage of the home appliance. This data includes not only device usage information but also information about the usage environment. This data is sent to a server and stored in a database.

[0118] Next, the server uses data analysis tools to analyze the collected data and model the user's lifestyle patterns. In addition, an emotion engine estimates the user's emotional state from their voice and facial expressions. This estimation involves analyzing audio and video data collected, for example, via a camera or microphone, to obtain information such as whether the user is relaxed or stressed.

[0119] The server then takes into account both the user's lifestyle and emotional state to suggest the most suitable operating method. For example, if it estimates that the user is tired, it can suggest automatically adjusting the lighting to a softer glow and playing soothing music. This suggestion is sent to the terminal via information and communication, and the terminal notifies the user. The user can then review the suggestion and choose whether or not to implement it.

[0120] Furthermore, the selected action and the subsequent emotional state are fed back from the sensors and analyzed again on the server. This feedback helps to further refine the next suggestion, improving the system's accuracy and user satisfaction.

[0121] In this way, by learning and adapting to the user's lifestyle patterns and emotions in real time, it becomes possible to provide a more human-like environment. This invention is implemented as a next-generation smart home appliance system that simultaneously achieves user comfort and energy efficiency.

[0122] The following describes the processing flow.

[0123] Step 1:

[0124] The device collects real-time data on the usage of home appliances and environmental information through sensors. Furthermore, it simultaneously acquires data to detect the user's emotional state through voice recognition and video analysis.

[0125] Step 2:

[0126] The device transmits collected usage and emotional state data to the server. The server securely stores this data and prepares it for subsequent analysis.

[0127] Step 3:

[0128] The server uses data analysis tools to analyze the received data and model the user's lifestyle patterns. Simultaneously, it operates an emotion engine to evaluate the user's mental state based on the emotion data.

[0129] Step 4:

[0130] The server synthesizes the analysis results and generates optimal action suggestions tailored to the user's state and habits. For example, if the user is showing signs of fatigue, it may include specific instructions such as changing the lighting to a warmer color.

[0131] Step 5:

[0132] The server sends the generated operation suggestions to the terminal using information and communication means. The terminal notifies the user of the suggestions through the user interface. The notification is displayed as a display or audio alert.

[0133] Step 6:

[0134] The user can choose to perform an action based on the received suggestion or ignore the suggestion. The results of this choice are collected again through sensors.

[0135] Step 7:

[0136] The device then sends the user's actions and subsequent emotional state back to the server. The server uses this feedback data to update the model and improve the accuracy of future suggestions. This allows the system to continuously learn and better adapt to the user's needs.

[0137] (Example 2)

[0138] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0139] In recent years, there has been a growing demand for optimized operation efficiency and resource usage of devices in homes and offices. However, conventional systems have struggled to meet individual user needs because they rely solely on regular operating schedules without considering the user's emotional state. Furthermore, the lack of sufficient functionality for continuous system improvement through feedback has prevented them from maximizing user comfort.

[0140] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0141] In this invention, the server includes a detection device for acquiring information related to usage status, a data processing device for analyzing the information acquired from the detection device and learning the user's behavior patterns, an emotion recognition device for recognizing the user's emotional state in addition to the results learned by the data processing device, and a communication device for presenting the user with the optimal operating method and resource management based on the emotional state recognized by the emotion recognition device and the learning results. This enables optimized device operation tailored to the emotional state of each individual user and realizes continuous system improvement using feedback.

[0142] A "detection device" is a device used to acquire information related to usage conditions.

[0143] A "data processing device" is a device used to analyze acquired information and learn the user's behavioral patterns.

[0144] An "emotion recognition device" is a device used to recognize the emotional state of a user, and it has the function of estimating emotions from voice and facial expressions.

[0145] A "communication device" is a device used to present the user with the optimal operating method and resource management based on the analysis results and emotion recognition results.

[0146] This invention is a system for optimizing the operation of devices in the user's living environment and adapting to the user's emotional state. Specifically, this objective is achieved by combining a detection device, a data processing device, an emotion recognition device, and a communication device.

[0147] The terminal collects information about the usage status of home appliances and the environment through a detection device equipped with multiple sensors. This information includes the on / off status of lights, temperature settings, and volume levels. This data is transmitted to a server using wireless communication technology.

[0148] The server analyzes the information it receives using a data processing device and learns user behavior patterns using machine learning algorithms. This involves techniques such as time series analysis and clustering. For example, open-source libraries and commercial cloud-based AI services can be utilized.

[0149] The server further uses an emotion recognition device to estimate the emotional state from the voice and facial expression data collected by the terminal. A voice recognition API is used for voice analysis, and image processing software is used for facial expression analysis. This allows the server to obtain information such as whether the user is relaxed or stressed.

[0150] The optimized operating method is transmitted to the terminal via a communication device based on the analysis results and emotion recognition results. The terminal notifies the user of this information and displays a suggestion on the screen, for example, "To match your current mood, we will soften the lighting and play relaxing music." The user chooses whether to accept the suggestion, and the result of that choice is sent back to the server as feedback.

[0151] As a concrete example, if the server detects that a user is experiencing stress, it suggests changing the lighting to a warmer color and playing relaxing music. This suggestion enables the creation of an optimal environment based on the user's emotions.

[0152] An example of a prompt for a generating AI model could be: "Please describe a system that suggests the most relaxing appliance settings for a user based on appliance usage data and voice / facial expression data. What specific elements does the system analyze?"

[0153] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0154] Step 1:

[0155] The device uses multiple built-in sensors to collect data on the surrounding environment and equipment usage. Inputs include the on / off status of lights, air conditioner temperature settings, volume levels, room temperature, and humidity. This information is collected asynchronously and transmitted to the server wirelessly as output. Specifically, the sensors sense constantly changing environmental data in real time and convert it into digital signals.

[0156] Step 2:

[0157] The server stores the received usage and environmental data in a database. It receives various sensor data transmitted from terminals as input. As output, it organizes this data chronologically and stores it in the database in a format suitable for long-term analysis. Specifically, the server eliminates data duplication and inserts it into tables within the database.

[0158] Step 3:

[0159] The server analyzes stored data and applies machine learning algorithms to learn user behavior patterns. It takes raw usage data stored in a database as input. The output models patterns such as when and what devices users prefer to use. Specifically, it uses Python libraries and cloud service APIs to perform iterative learning using clustering techniques and extract patterns.

[0160] Step 4:

[0161] The device uses its built-in camera and microphone to collect the user's voice and facial expressions. It acquires ambient sounds and visual data from the user's environment as input. It sends audio waveform data and image frames to the server as output. Specifically, the device compresses audio data appropriately and selects and transmits a portion of the image data from a continuous frame to reduce communication load.

[0162] Step 5:

[0163] The server uses an emotion recognition device to estimate emotional states from transmitted audio and facial expression data. It receives audio waveforms and image data transmitted by the terminal as input. As output, the emotional state (e.g., relaxed, stressed) is quantified and stored as analysis results. Specifically, it uses speech recognition APIs and image processing libraries to analyze voice tone and facial features, and reflects the estimated emotion in the database.

[0164] Step 6:

[0165] The server generates optimal operating methods and resource management for the user based on the obtained behavioral patterns and emotional states. It integrates behavioral models and emotional data as input. Outputs include suggestions such as adjusting lighting color temperature or creating music playlists. In terms of specific actions, it selects the most appropriate control signal from multiple candidates and sends the instruction to the terminal via a communication device.

[0166] Step 7:

[0167] The user receives a notification from the device and chooses whether to perform the suggested action. As input, they visually confirm the options displayed on the device screen. As output, they determine their selection through a simple interface operation, and the result is sent to the server. In terms of specific action, the selection is made via the touchscreen, and the selection data is re-introduced into the system as feedback.

[0168] Step 8:

[0169] The server analyzes feedback data and updates its analysis model to improve the accuracy of the system's suggestions. It analyzes user-selected data and subsequently collected sensor data as input. The output is a model update that makes the next suggestion more personalized. Specifically, the model is periodically retrained to reflect the latest data, ensuring the system is always up-to-date.

[0170] (Application Example 2)

[0171] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0172] To improve the customer experience in modern brick-and-mortar stores, personalized customer service is required. However, conventional technology struggles to perceive and reflect customer emotions and individual preferences in real time. Furthermore, the challenge lies in creating an environment optimized for each individual customer while increasing the energy efficiency of store operations.

[0173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0174] In this invention, the server includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for suggesting optimal operating methods and energy management to the user, and an emotion estimation means for estimating the user's emotional state. This makes it possible to provide individually optimized customer service methods, product suggestions, and store environment adjustments based on the emotions and behavior of customers.

[0175] "Usage data" refers to information about user behavior and the environment acquired by detection devices.

[0176] A "detection device" refers to a device such as a sensor or camera used to acquire usage data.

[0177] "Information analysis methods" refer to technologies that analyze acquired data to estimate user behavior patterns and emotional states.

[0178] "Means of communication" refer to methods and devices for providing information and suggestions to users based on analysis results.

[0179] "Emotion estimation methods" are technologies that evaluate and infer a user's emotional state from voice and facial expression data.

[0180] A "user interface" refers to the screen display or device that allows a user to receive information and perform actions.

[0181] This invention is a system for improving the customer experience in physical stores. The system includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for providing optimal operations and suggestions, and an emotion estimation means for understanding the customer's emotional state.

[0182] The server aggregates data acquired through detection devices, and information analysis tools analyze this data. Specifically, smart glasses and in-store sensors record customer movements and facial expressions via cameras and microphones, and technology is used to estimate emotions from voice and facial features based on this data. Emotion analysis software such as Microsoft® Azure®'s Emotion API is used to estimate the customer's level of relaxation and the product categories they are interested in from the analyzed data.

[0183] Based on the emotional state and behavioral patterns it obtains, the server instructs store staff via communication channels on the most appropriate customer service methods and product recommendations. These instructions are displayed in real time on the smart glasses worn by the staff. For example, if it is estimated that a customer is relaxing while holding a book, the staff can suggest calming music or recommend gentle reading materials as related products.

[0184] As a concrete example, when a customer enters a store's cafe and takes a seat, smart glasses inform the staff that "the customer is calm" based on their facial expression. Based on this information, the staff can offer the customer the most suitable cafe menu or a quiet environment for reading. In this way, it becomes possible to create an optimal environment tailored to the customer's emotional state.

[0185] Example prompt: "Please tell me how to estimate the customer's level of relaxation based on their facial expression data and adjust the music and product suggestions accordingly."

[0186] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0187] Step 1:

[0188] The server collects customer usage data through detection devices installed in physical stores, such as cameras and microphones. Customer facial expressions and voice data are captured as input. The server stores this data and formats it as basic data for subsequent processing.

[0189] Step 2:

[0190] The server uses emotion analysis software (e.g., Microsoft Azure Emotion API) to analyze the collected facial and voice data. Using the data formatted in the previous step as input, the server performs emotion analysis. In this analysis process, the data is decomposed into features, and the customer's emotional state (e.g., relaxed, excited, focused) is estimated based on the original data.

[0191] Step 3:

[0192] The server uses information analysis tools to generate a model based on the customer's emotional state and behavioral patterns, and calculates the optimal customer service method or suggested action. It uses the output of emotional analysis as input to infer the environment settings and product information that the customer is presumed to desire. At this stage, data analysis and machine learning models are applied.

[0193] Step 4:

[0194] The server transmits the generated suggestions to the store staff's smart glasses via a communication channel. The input is the optimized suggestions from the previous step. The server visualizes the data so that staff can view it in real time via the display and instructs them to take specific actions (e.g., adjust background music, recommend specific products).

[0195] Step 5:

[0196] Users interact with customers using suggested service methods and product information. Input is suggested information from the server. Based on this, users provide services tailored to customer needs and emotions, improving the customer experience.

[0197] Step 6:

[0198] The terminal acquires the results of the interaction with the customer as data again through a detection device and feeds it back to the server. The input consists of the user's customer service results and subsequent environmental data. The server uses this feedback information to improve the accuracy of the system by incorporating it into future analyses and suggestions.

[0199] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0200] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0201] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0202] [Second Embodiment]

[0203] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0204] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0205] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0206] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0207] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0208] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0209] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0210] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0211] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0212] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0213] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0214] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0215] This invention is constructed as a system equipped with sensor means, data analysis means, and information communication means. First, the sensor means built into the terminal acquires usage data of the home appliance in real time. This includes information such as usage time, operating mode, and ambient environmental conditions (e.g., room temperature and illuminance). This allows the terminal to transmit detailed usage data to the server.

[0216] Next, the server receives this data and analyzes it in detail using data analysis tools. By using machine learning algorithms, the server learns and models the user's lifestyle patterns. For example, it can recognize patterns in which users use specific home appliances at specific times of the day.

[0217] Furthermore, based on the analysis results obtained, the server generates suggestions for optimal operation methods and improvements to energy efficiency. These suggestions are transmitted to the terminal via information and communication means and notified to the user via the terminal's display or a smartphone app. For example, if the server detects a pattern of using the coffee maker at a certain time in the morning, it can suggest setting it to automatically turn on a little before that time.

[0218] As a result, users can reduce energy waste and improve convenience by performing the suggested actions. Furthermore, user selections and actions are recorded again via sensors and fed back into subsequent analysis. This allows the system to continuously learn from user behavior and further optimize its performance.

[0219] Thus, the system of the present invention is implemented as a smart home appliance solution that continuously evolves to match the user's lifestyle, achieving improvements in convenience and energy efficiency.

[0220] The following describes the processing flow.

[0221] Step 1:

[0222] The device collects real-time usage data of home appliances through sensors. This data includes usage time, device operating status, and ambient environmental conditions.

[0223] Step 2:

[0224] The device sends the collected data to the server at regular intervals. This data is stored in a database for use in subsequent analysis.

[0225] Step 3:

[0226] The server performs initial data cleansing on the received data. It removes noise and outliers and formats the data in a way that is suitable for analysis.

[0227] Step 4:

[0228] The server feeds cleansed data to machine learning algorithms to analyze and model the user's lifestyle patterns. This allows it to recognize usage trends at specific times and under specific conditions.

[0229] Step 5:

[0230] The server generates operation suggestions tailored to the user based on the analysis results. This includes automatic setting of operation schedules aimed at improving energy efficiency and suggestions for energy-saving modes.

[0231] Step 6:

[0232] The server sends the generated operation suggestions to the terminal via information and communication means. The terminal notifies the user of the suggestions through the user interface.

[0233] Step 7:

[0234] The user can either accept the suggestion or make their own adjustments. The results of this operation are again recorded by the device's sensors and fed back to the server.

[0235] Step 8:

[0236] The server uses the new feedback data to update and improve the user's model, thereby improving the accuracy of future suggestions.

[0237] (Example 1)

[0238] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0239] Modern home electronic devices are diverse, and their efficient use and energy conservation are essential. However, it is difficult for users to recognize their own behavioral patterns and manually configure devices appropriately, often resulting in wasted energy. To solve this, a system is needed that automatically learns user behavior and suggests optimized operation.

[0240] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0241] In this invention, the server includes detection means for collecting usage information, information processing means for analyzing the information collected from the detection means and modeling the user's behavioral tendencies, communication means for proposing efficient operating methods and energy-saving strategies to the user based on the model obtained by the information processing means, and proposal generation means for generating proposals using a generated AI model. As a result, the user can achieve efficient and energy-efficient use of the equipment without having to recognize their own behavioral patterns.

[0242] "Usage information" refers to information such as time, mode, and environmental conditions related to the use of home appliances and other devices.

[0243] "Detection means" refers to a system that acquires usage information using sensors or other detection devices.

[0244] "Information processing means" refers to technologies and devices used to analyze collected usage information and model user behavior trends and patterns.

[0245] "Communication methods" refer to network technologies and devices used to deliver suggestions to users regarding energy management and operation methods based on analysis results.

[0246] A "generative AI model" refers to an algorithm that uses artificial intelligence to learn from data and generate new suggestions.

[0247] "Suggestion generation method" refers to a system that uses a generation AI model to suggest the optimal operating method according to the user's behavior and usage situation.

[0248] This invention is a system for efficiently managing the usage status of home appliances using terminals installed in each home. The terminals are equipped with various sensors for detecting temperature, illuminance, motion, etc., and these sensors collect usage information in real time.

[0249] The terminal transfers the collected information to the server. The server is equipped with information processing capabilities and uses machine learning algorithms to analyze user behavior trends and usage patterns from the collected data. This analysis makes it possible to model patterns such as the frequent use of specific home appliances during specific time periods.

[0250] Furthermore, the server utilizes the generated AI model to create optimal operation suggestions based on the model obtained from the information processing means. These suggestions are transmitted to the terminal via communication means and presented to the user through the terminal's display or an application installed on the user's smartphone. For example, if a user has a habit of using a coffee maker at a specific time every morning, the server can generate a suggestion to automatically turn on the power at that time.

[0251] This allows users to improve the convenience of their daily lives while saving energy simply by performing the suggested actions. Furthermore, the actions performed by the user are fed back through the terminal and sent to the server. This feedback information is used for subsequent analysis, further enhancing the overall system and improving its ability to provide suggestions tailored to individual users.

[0252] An example of a prompt message might be, "Based on my morning routine, please provide suggestions for improving energy efficiency." This system creates an environment where users can manage their home appliances efficiently and smartly without any hassle.

[0253] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0254] Step 1:

[0255] The device uses its built-in sensors to collect information on the usage status of home appliances. The input data includes environmental information such as the appliance's usage time, mode, room temperature, and illuminance. The device periodically scans this information and outputs the data in streaming format. Specifically, the device acquires information at regular intervals and temporarily stores it in local memory.

[0256] Step 2:

[0257] The terminal sends the usage information it collects to the server. The input data is the usage information output from the terminal. The terminal encodes this data as a digital signal and transfers it to the server using a secure communication protocol. Specifically, the terminal performs batch transfers when a certain amount of data is reached.

[0258] Step 3:

[0259] The server analyzes the usage information it receives using information processing tools. The input data is usage information sent from the terminal. The server uses machine learning algorithms to analyze the data and derive and output user behavior patterns. For example, it may extract patterns in which specific home appliances are frequently used during certain time periods.

[0260] Step 4:

[0261] The server generates suggestions using a generative AI model based on the analysis results. The input data is the user's behavior patterns derived from the analysis. The server inputs this data into the generative AI model and outputs suggestions for optimal operating methods and energy-saving measures. Specifically, the server compares various candidate settings and selects the most effective one.

[0262] Step 5:

[0263] The server generates suggestions and sends them to the terminal via a communication method, notifying the user. The input data is the generated suggestions. The server generates a notification message and outputs it to the user's display or smartphone app via the terminal. Specifically, a notification is made through the user interface based on the message content.

[0264] Step 6:

[0265] The user operates the device to take action based on the suggestions. The input data is the suggestions received via the device. The user reviews the suggestions and performs specific actions through the device's UI. Specific actions include pressing buttons to change the settings of home appliances based on the suggestions.

[0266] Step 7:

[0267] The device collects user behavior data again via sensors and feeds it back to the server. The input data is the actual result of the user's actions. The device analyzes the obtained data and completes the feedback process by sending the results back to the server. Specifically, the device acquires new data from the sensors and sends it to the server in batches.

[0268] (Application Example 1)

[0269] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] In modern industry, optimizing machine operation and resource management are crucial challenges. However, currently, many industrial machines are not operated efficiently based on actual usage patterns, potentially leading to energy waste and increased operating costs. To solve this problem, a system is needed that monitors machine operating status in real time and proposes efficient operating methods based on that data.

[0271] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0272] In this invention, the server includes a detection means for acquiring usage data, a mathematical analysis means for analyzing the data acquired from the detection means and learning the user's behavior patterns, an information and communication means for proposing optimal operating methods and resource management to the user based on the results learned by the mathematical analysis means, and a function of the information and communication means for analyzing the operating status of industrial machinery and generating suggestions to improve efficiency. This makes it possible to improve the operating efficiency of industrial machinery and reduce unnecessary energy consumption.

[0273] "Usage data" refers to information indicating the operating status of industrial machinery, including, for example, operating hours, operating mode, and environmental conditions.

[0274] "Detection means" refers to a device or method for acquiring real-time usage data of industrial machinery.

[0275] "Mathematical analysis means" refers to methods and techniques for learning machine operation patterns from acquired usage data and deriving efficient operating methods.

[0276] "Information and communication means" refers to communication technology or devices used to transmit to the user, based on analysis results, suggestions for optimal operation methods and resource management.

[0277] "Behavioral patterns" refer to the repetitive and predictable ways in which industrial machinery is used or exhibits certain tendencies under specific conditions.

[0278] "Resource management" refers to management methods for efficiently operating industrial machinery while minimizing the consumption of energy and materials associated with its use.

[0279] The system for realizing this invention is configured to monitor and optimize the efficient operation of industrial machinery. The server acquires real-time usage data of the industrial machinery using detection means. This includes the machine's operating time, operating mode, and ambient environmental conditions.

[0280] The acquired data is sent to a server and analyzed using mathematical analysis tools. The server uses machine learning algorithms to learn machine usage patterns and model efficient operating methods.

[0281] Based on modeled usage patterns, the system sends suggestions for optimal operation methods and resource management to the terminal via information and communication means, notifying the user. For example, if the server recognizes that operating a machine during a specific time period is energy-efficient, it can suggest shifting the operation to that time period.

[0282] With this system, the user can execute the proposed operations to reduce energy consumption. Also, the execution results of the user are fed back to the server through the detection means, and the model is updated by the mathematical analysis means. Through this process, the system continuously learns and generates more optimized proposals.

[0283] As a specific example, in a manufacturing factory, the temperature rise during the day was causing a decrease in machine efficiency. By introducing this system, according to the patterns detected by the server, it proposed energy-efficient night operations. As a result, the factory's energy costs could be effectively reduced.

[0284] An example of a prompt sentence to input into the generative AI model is "Analyze the operating patterns of factory robots, generate optimal proposals to suppress energy consumption, test them in a laboratory environment, and report."

[0285] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0286] Step 1:

[0287] The server acquires usage data from industrial machines through the detection means. This includes information such as operating time, operation mode, environmental conditions, etc. The acquired data is stored in a database and used as input for subsequent analysis processing.

[0288] Step 2:

[0289] The server inputs the acquired usage data into the mathematical analysis means and analyzes the data using machine learning algorithms. In this process, the server identifies the usage patterns of the machine and generates a model for deriving efficient operation methods. As a result, the characteristics of the main usage patterns are output.

[0290] Step 3:

[0291] Based on the generated model, the server sends suggestions for optimal operation methods and resource management to the terminal via information and communication means. For example, it may notify the terminal of energy-efficient operating hours. This allows the terminal to present specific improvement suggestions to the user.

[0292] Step 4:

[0293] The user executes the suggestions received through the terminal. The resulting usage data is then sent back to the server via the detection mechanism. This data is used as feedback and incorporated into the model update process.

[0294] Step 5:

[0295] The server analyzes the feedback data and updates the existing model using mathematical analysis tools. The updated model is used as the basis for future proposal generation. This allows the system to be continuously optimized and provide users with even more effective proposals.

[0296] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0297] This invention combines a system that learns the user's lifestyle patterns and proposes optimal operating methods and energy management with an emotion engine that recognizes the user's emotions. This system comprises sensor means, data analysis means, information communication means, and an emotion engine.

[0298] First, the sensor built into the device collects data on the usage of the home appliance. This data includes not only device usage information but also information about the usage environment. This data is sent to a server and stored in a database.

[0299] Next, the server uses data analysis tools to analyze the collected data and model the user's lifestyle patterns. In addition, an emotion engine estimates the user's emotional state from their voice and facial expressions. This estimation involves analyzing audio and video data collected, for example, via a camera or microphone, to obtain information such as whether the user is relaxed or stressed.

[0300] The server then takes into account both the user's lifestyle and emotional state to suggest the most suitable operating method. For example, if it estimates that the user is tired, it can suggest automatically adjusting the lighting to a softer glow and playing soothing music. This suggestion is sent to the terminal via information and communication, and the terminal notifies the user. The user can then review the suggestion and choose whether or not to implement it.

[0301] Furthermore, the selected action and subsequent emotional state are fed back from the sensors and analyzed again on the server. This feedback helps to further refine the next suggestion, improving the system's accuracy and user satisfaction.

[0302] In this way, by learning and adapting to the user's lifestyle patterns and emotions in real time, it becomes possible to provide a more human-like environment. This invention is implemented as a next-generation smart home appliance system that simultaneously achieves user comfort and energy efficiency.

[0303] The following describes the processing flow.

[0304] Step 1:

[0305] The device collects real-time data on the usage of home appliances and environmental information through sensors. Furthermore, it simultaneously acquires data to detect the user's emotional state through voice recognition and video analysis.

[0306] Step 2:

[0307] The terminal transmits the collected usage data and emotional state data to the server. The server securely stores these data in preparation for subsequent analysis.

[0308] Step 3:

[0309] The server uses data analysis means to analyze the received data and model the user's life pattern. At the same time, it operates the emotion engine to evaluate the user's mental state based on the emotion data.

[0310] Step 4:

[0311] The server synthesizes the analysis results and generates an optimal operation proposal according to the user's state and habits. For example, when the user shows fatigue, it includes specific operation instructions such as changing the lighting to warm color.

[0312] Step 5:

[0313] The server transmits the generated operation proposal to the terminal using information communication means. The terminal notifies the user of the proposal content through the user interface. The notification is displayed as a display or a voice alert.

[0314] Step 6:

[0315] The user can choose to execute the operation based on the received proposal or ignore the proposal. The results obtained from this selection are recollected through the sensor.

[0316] Step 7:

[0317] The terminal transmits the user's operation result and subsequent emotional state to the server again. The server uses this feedback data to update the model and improve the accuracy of the next proposal. Thereby, the system continuously learns and better adapts to the user's needs.

[0318] (Example 2)

[0319] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0320] In recent years, there has been a growing demand for optimized operation efficiency and resource usage of devices in homes and offices. However, conventional systems have struggled to meet individual user needs because they rely solely on regular operating schedules without considering the user's emotional state. Furthermore, the lack of sufficient functionality for continuous system improvement through feedback has prevented them from maximizing user comfort.

[0321] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0322] In this invention, the server includes a detection device for acquiring information related to usage status, a data processing device for analyzing the information acquired from the detection device and learning the user's behavior patterns, an emotion recognition device for recognizing the user's emotional state in addition to the results learned by the data processing device, and a communication device for presenting the user with the optimal operating method and resource management based on the emotional state recognized by the emotion recognition device and the learning results. This enables optimized device operation tailored to the emotional state of each individual user and realizes continuous system improvement using feedback.

[0323] A "detection device" is a device used to acquire information related to usage conditions.

[0324] A "data processing device" is a device used to analyze acquired information and learn the user's behavioral patterns.

[0325] An "emotion recognition device" is a device used to recognize the emotional state of a user, and it has the function of estimating emotions from voice and facial expressions.

[0326] A "communication device" is a device used to present the user with the optimal operating method and resource management based on the analysis results and emotion recognition results.

[0327] This invention is a system for optimizing the operation of devices in the user's living environment and adapting to the user's emotional state. Specifically, this objective is achieved by combining a detection device, a data processing device, an emotion recognition device, and a communication device.

[0328] The terminal collects information about the usage status of home appliances and the environment through a detection device equipped with multiple sensors. This information includes the on / off status of lights, temperature settings, and volume levels. This data is transmitted to a server using wireless communication technology.

[0329] The server analyzes the information it receives using a data processing device and learns user behavior patterns using machine learning algorithms. This involves techniques such as time series analysis and clustering. For example, open-source libraries and commercial cloud-based AI services can be utilized.

[0330] The server further uses an emotion recognition device to estimate the emotional state from the voice and facial expression data collected by the terminal. A voice recognition API is used for voice analysis, and image processing software is used for facial expression analysis. This allows the server to obtain information such as whether the user is relaxed or stressed.

[0331] The optimized operating method is transmitted to the terminal via a communication device based on the analysis results and emotion recognition results. The terminal notifies the user of this information and displays a suggestion on the screen, for example, "To match your current mood, we will soften the lighting and play relaxing music." The user chooses whether to accept the suggestion, and the result of that choice is sent back to the server as feedback.

[0332] As a concrete example, if the server detects that a user is experiencing stress, it suggests changing the lighting to a warmer color and playing relaxing music. This suggestion enables the creation of an optimal environment based on the user's emotions.

[0333] An example of a prompt for a generating AI model could be: "Please describe a system that suggests the most relaxing appliance settings for a user based on appliance usage data and voice / facial expression data. What specific elements does the system analyze?"

[0334] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0335] Step 1:

[0336] The device uses multiple built-in sensors to collect data on the surrounding environment and equipment usage. Inputs include the on / off status of lights, air conditioner temperature settings, volume levels, room temperature, and humidity. This information is collected asynchronously and transmitted to the server wirelessly as output. Specifically, the sensors sense constantly changing environmental data in real time and convert it into digital signals.

[0337] Step 2:

[0338] The server stores the received usage and environmental data in a database. It receives various sensor data transmitted from terminals as input. As output, it organizes this data chronologically and stores it in the database in a format suitable for long-term analysis. Specifically, the server eliminates data duplication and inserts it into tables within the database.

[0339] Step 3:

[0340] The server analyzes stored data and applies machine learning algorithms to learn user behavior patterns. It takes raw usage data stored in a database as input. The output models patterns such as when and what devices users prefer to use. Specifically, it uses Python libraries and cloud service APIs to perform iterative learning using clustering techniques and extract patterns.

[0341] Step 4:

[0342] The device uses its built-in camera and microphone to collect the user's voice and facial expressions. It acquires ambient sounds and visual data from the user's environment as input. It sends audio waveform data and image frames to the server as output. Specifically, the device compresses audio data appropriately and selects and transmits a portion of the image data from a continuous frame to reduce communication load.

[0343] Step 5:

[0344] The server uses an emotion recognition device to estimate emotional states from transmitted audio and facial expression data. It receives audio waveforms and image data transmitted by the terminal as input. As output, the emotional state (e.g., relaxed, stressed) is quantified and stored as analysis results. Specifically, it uses speech recognition APIs and image processing libraries to analyze voice tone and facial features, and reflects the estimated emotion in the database.

[0345] Step 6:

[0346] The server generates optimal operating methods and resource management for the user based on the obtained behavioral patterns and emotional states. It integrates behavioral models and emotional data as input. Outputs include suggestions such as adjusting lighting color temperature or creating music playlists. In terms of specific actions, it selects the most appropriate control signal from multiple candidates and sends the instruction to the terminal via a communication device.

[0347] Step 7:

[0348] The user receives a notification from the device and chooses whether to perform the suggested action. As input, they visually confirm the options displayed on the device screen. As output, they determine their selection through a simple interface operation, and the result is sent to the server. In terms of specific action, the selection is made via the touchscreen, and the selection data is re-introduced into the system as feedback.

[0349] Step 8:

[0350] The server analyzes feedback data and updates its analysis model to improve the accuracy of the system's suggestions. It analyzes user-selected data and subsequently collected sensor data as input. The output is a model update that makes the next suggestion more personalized. Specifically, the model is periodically retrained to reflect the latest data, ensuring the system is always up-to-date.

[0351] (Application Example 2)

[0352] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0353] To improve the customer experience in modern brick-and-mortar stores, personalized customer service is required. However, conventional technology struggles to perceive and reflect customer emotions and individual preferences in real time. Furthermore, the challenge lies in creating an environment optimized for each individual customer while increasing the energy efficiency of store operations.

[0354] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0355] In this invention, the server includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for suggesting optimal operating methods and energy management to the user, and an emotion estimation means for estimating the user's emotional state. This makes it possible to provide individually optimized customer service methods, product suggestions, and store environment adjustments based on the emotions and behavior of customers.

[0356] "Usage data" refers to information about user behavior and the environment acquired by detection devices.

[0357] A "detection device" refers to a device such as a sensor or camera used to acquire usage data.

[0358] "Information analysis methods" refer to technologies that analyze acquired data to estimate user behavior patterns and emotional states.

[0359] "Means of communication" refer to methods and devices for providing information and suggestions to users based on analysis results.

[0360] "Emotion estimation methods" are technologies that evaluate and infer a user's emotional state from voice and facial expression data.

[0361] A "user interface" refers to the screen display or device that allows a user to receive information and perform actions.

[0362] This invention is a system for improving the customer experience in physical stores. The system includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for providing optimal operations and suggestions, and an emotion estimation means for understanding the customer's emotional state.

[0363] The server aggregates data acquired through detection devices, and information analysis tools analyze this data. Specifically, smart glasses and in-store sensors record customer movements and facial expressions via cameras and microphones, and technology is used to estimate emotions from voice and facial features based on this data. Emotion analysis software, such as Microsoft Azure's Emotion API, is used to estimate the customer's level of relaxation and the product categories they are interested in from the analyzed data.

[0364] Based on the emotional state and behavioral patterns it obtains, the server instructs store staff via communication channels on the most appropriate customer service methods and product recommendations. These instructions are displayed in real time on the smart glasses worn by the staff. For example, if it is estimated that a customer is relaxing while holding a book, the staff can suggest calming music or recommend gentle reading materials as related products.

[0365] As a concrete example, when a customer enters a store's cafe and takes a seat, smart glasses inform the staff that "the customer is calm" based on their facial expression. Based on this information, the staff can offer the customer the most suitable cafe menu or a quiet environment for reading. In this way, it becomes possible to create an optimal environment tailored to the customer's emotional state.

[0366] Example prompt: "Please tell me how to estimate the customer's level of relaxation based on their facial expression data and adjust the music and product suggestions accordingly."

[0367] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0368] Step 1:

[0369] The server collects customer usage data through detection devices installed in physical stores, such as cameras and microphones. Customer facial expressions and voice data are captured as input. The server stores this data and formats it as basic data for subsequent processing.

[0370] Step 2:

[0371] The server uses emotion analysis software (e.g., Microsoft Azure Emotion API) to analyze the collected facial and voice data. Using the data formatted in the previous step as input, the server performs emotion analysis. In this analysis process, the data is decomposed into features, and the customer's emotional state (e.g., relaxed, excited, focused) is estimated based on the original data.

[0372] Step 3:

[0373] The server uses information analysis tools to generate a model based on the customer's emotional state and behavioral patterns, and calculates the optimal customer service method or suggested action. It uses the output of emotional analysis as input to infer the environment settings and product information that the customer is presumed to desire. At this stage, data analysis and machine learning models are applied.

[0374] Step 4:

[0375] The server transmits the generated suggestions to the store staff's smart glasses via a communication channel. The input is the optimized suggestions from the previous step. The server visualizes the data so that staff can view it in real time via the display and instructs them to take specific actions (e.g., adjust background music, recommend specific products).

[0376] Step 5:

[0377] Users interact with customers using suggested service methods and product information. Input is suggested information from the server. Based on this, users provide services tailored to customer needs and emotions, improving the customer experience.

[0378] Step 6:

[0379] The terminal acquires the results of the interaction with the customer as data again through a detection device and feeds it back to the server. The input consists of the user's customer service results and subsequent environmental data. The server uses this feedback information to improve the accuracy of the system by incorporating it into future analyses and suggestions.

[0380] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0381] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0382] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0383] [Third Embodiment]

[0384] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0385] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0386] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0387] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0388] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0389] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0390] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0391] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0392] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0393] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0394] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0395] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0396] This invention is constructed as a system equipped with sensor means, data analysis means, and information communication means. First, the sensor means built into the terminal acquires usage data of the home appliance in real time. This includes information such as usage time, operating mode, and ambient environmental conditions (e.g., room temperature and illuminance). This allows the terminal to transmit detailed usage data to the server.

[0397] Next, the server receives this data and analyzes it in detail using data analysis tools. By using machine learning algorithms, the server learns and models the user's lifestyle patterns. For example, it can recognize patterns in which users use specific home appliances at specific times of the day.

[0398] Furthermore, based on the analysis results obtained, the server generates suggestions for optimal operation methods and improvements to energy efficiency. These suggestions are transmitted to the terminal via information and communication means and notified to the user via the terminal's display or a smartphone app. For example, if the server detects a pattern of using the coffee maker at a certain time in the morning, it can suggest setting it to automatically turn on a little before that time.

[0399] As a result, users can reduce energy waste and improve convenience by performing the suggested actions. Furthermore, user selections and actions are recorded again via sensors and fed back into subsequent analysis. This allows the system to continuously learn from user behavior and further optimize its performance.

[0400] Thus, the system of the present invention is implemented as a smart home appliance solution that continuously evolves to match the user's lifestyle, achieving improvements in convenience and energy efficiency.

[0401] The following describes the processing flow.

[0402] Step 1:

[0403] The device collects real-time usage data of home appliances through sensors. This data includes usage time, device operating status, and ambient environmental conditions.

[0404] Step 2:

[0405] The device sends the collected data to the server at regular intervals. This data is stored in a database for use in subsequent analysis.

[0406] Step 3:

[0407] The server performs initial data cleansing on the received data. It removes noise and outliers and formats the data in a way that is suitable for analysis.

[0408] Step 4:

[0409] The server feeds cleansed data to machine learning algorithms to analyze and model the user's lifestyle patterns. This allows it to recognize usage trends at specific times and under specific conditions.

[0410] Step 5:

[0411] The server generates operation suggestions tailored to the user based on the analysis results. This includes automatic setting of operation schedules aimed at improving energy efficiency and suggestions for energy-saving modes.

[0412] Step 6:

[0413] The server sends the generated operation suggestions to the terminal via information and communication means. The terminal notifies the user of the suggestions through the user interface.

[0414] Step 7:

[0415] The user can either accept the suggestion or make their own adjustments. The results of this operation are again recorded by the device's sensors and fed back to the server.

[0416] Step 8:

[0417] The server uses the new feedback data to update and improve the user's model, thereby improving the accuracy of future suggestions.

[0418] (Example 1)

[0419] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0420] Modern home electronic devices are diverse, and their efficient use and energy conservation are essential. However, it is difficult for users to recognize their own behavioral patterns and manually configure devices appropriately, often resulting in wasted energy. To solve this, a system is needed that automatically learns user behavior and suggests optimized operation.

[0421] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0422] In this invention, the server includes detection means for collecting usage information, information processing means for analyzing the information collected from the detection means and modeling the user's behavioral tendencies, communication means for proposing efficient operating methods and energy-saving strategies to the user based on the model obtained by the information processing means, and proposal generation means for generating proposals using a generated AI model. As a result, the user can achieve efficient and energy-efficient use of the equipment without having to recognize their own behavioral patterns.

[0423] "Usage information" refers to information such as time, mode, and environmental conditions related to the use of home appliances and other devices.

[0424] "Detection means" refers to a system that acquires usage information using sensors or other detection devices.

[0425] "Information processing means" refers to technologies and devices used to analyze collected usage information and model user behavior trends and patterns.

[0426] "Communication methods" refer to network technologies and devices used to deliver suggestions to users regarding energy management and operation methods based on analysis results.

[0427] A "generative AI model" refers to an algorithm that uses artificial intelligence to learn from data and generate new suggestions.

[0428] "Suggestion generation method" refers to a system that uses a generation AI model to suggest the optimal operating method according to the user's behavior and usage situation.

[0429] This invention is a system for efficiently managing the usage status of home appliances using terminals installed in each home. The terminals are equipped with various sensors for detecting temperature, illuminance, motion, etc., and these sensors collect usage information in real time.

[0430] The terminal transfers the collected information to the server. The server is equipped with information processing capabilities and uses machine learning algorithms to analyze user behavior trends and usage patterns from the collected data. This analysis makes it possible to model patterns such as the frequent use of specific home appliances during specific time periods.

[0431] Furthermore, the server utilizes the generated AI model to create optimal operation suggestions based on the model obtained from the information processing means. These suggestions are transmitted to the terminal via communication means and presented to the user through the terminal's display or an application installed on the user's smartphone. For example, if a user has a habit of using a coffee maker at a specific time every morning, the server can generate a suggestion to automatically turn on the power at that time.

[0432] This allows users to improve the convenience of their daily lives while saving energy simply by performing the suggested actions. Furthermore, the actions performed by the user are fed back through the terminal and sent to the server. This feedback information is used for subsequent analysis, further enhancing the overall system and improving its ability to provide suggestions tailored to individual users.

[0433] An example of a prompt message might be, "Based on my morning routine, please provide suggestions for improving energy efficiency." This system creates an environment where users can manage their home appliances efficiently and smartly without any hassle.

[0434] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0435] Step 1:

[0436] The device uses its built-in sensors to collect information on the usage status of home appliances. The input data includes environmental information such as the appliance's usage time, mode, room temperature, and illuminance. The device periodically scans this information and outputs the data in streaming format. Specifically, the device acquires information at regular intervals and temporarily stores it in local memory.

[0437] Step 2:

[0438] The terminal sends the usage information it collects to the server. The input data is the usage information output from the terminal. The terminal encodes this data as a digital signal and transfers it to the server using a secure communication protocol. Specifically, the terminal performs batch transfers when a certain amount of data is reached.

[0439] Step 3:

[0440] The server analyzes the usage information it receives using information processing tools. The input data is usage information sent from the terminal. The server uses machine learning algorithms to analyze the data and derive and output user behavior patterns. For example, it may extract patterns in which specific home appliances are frequently used during certain time periods.

[0441] Step 4:

[0442] The server generates suggestions using a generative AI model based on the analysis results. The input data is the user's behavior patterns derived from the analysis. The server inputs this data into the generative AI model and outputs suggestions for optimal operating methods and energy-saving measures. Specifically, the server compares various candidate settings and selects the most effective one.

[0443] Step 5:

[0444] The server generates suggestions and sends them to the terminal via a communication method, notifying the user. The input data is the generated suggestions. The server generates a notification message and outputs it to the user's display or smartphone app via the terminal. Specifically, a notification is made through the user interface based on the message content.

[0445] Step 6:

[0446] The user operates the device to take action based on the suggestions. The input data is the suggestions received via the device. The user reviews the suggestions and performs specific actions through the device's UI. Specific actions include pressing buttons to change the settings of home appliances based on the suggestions.

[0447] Step 7:

[0448] The device collects user behavior data again via sensors and feeds it back to the server. The input data is the actual result of the user's actions. The device analyzes the obtained data and completes the feedback process by sending the results back to the server. Specifically, the device acquires new data from the sensors and sends it to the server in batches.

[0449] (Application Example 1)

[0450] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0451] In modern industry, optimizing machine operation and resource management are crucial challenges. However, currently, many industrial machines are not operated efficiently based on actual usage patterns, potentially leading to energy waste and increased operating costs. To solve this problem, a system is needed that monitors machine operating status in real time and proposes efficient operating methods based on that data.

[0452] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0453] In this invention, the server includes a detection means for acquiring usage data, a mathematical analysis means for analyzing the data acquired from the detection means and learning the user's behavior patterns, an information and communication means for proposing optimal operating methods and resource management to the user based on the results learned by the mathematical analysis means, and a function of the information and communication means for analyzing the operating status of industrial machinery and generating suggestions to improve efficiency. This makes it possible to improve the operating efficiency of industrial machinery and reduce unnecessary energy consumption.

[0454] "Usage data" refers to information indicating the operating status of industrial machinery, including, for example, operating hours, operating mode, and environmental conditions.

[0455] "Detection means" refers to a device or method for acquiring real-time usage data of industrial machinery.

[0456] "Mathematical analysis means" refers to methods and techniques for learning machine operation patterns from acquired usage data and deriving efficient operating methods.

[0457] "Information and communication means" refers to communication technology or devices used to transmit to the user, based on analysis results, suggestions for optimal operation methods and resource management.

[0458] "Behavioral patterns" refer to the repetitive and predictable ways in which industrial machinery is used or exhibits certain tendencies under specific conditions.

[0459] "Resource management" refers to management methods for efficiently operating industrial machinery while minimizing the consumption of energy and materials associated with its use.

[0460] The system for realizing this invention is configured to monitor and optimize the efficient operation of industrial machinery. The server acquires real-time usage data of the industrial machinery using detection means. This includes the machine's operating time, operating mode, and ambient environmental conditions.

[0461] The acquired data is sent to a server and analyzed using mathematical analysis tools. The server uses machine learning algorithms to learn machine usage patterns and model efficient operating methods.

[0462] Based on modeled usage patterns, the system sends suggestions for optimal operation methods and resource management to the terminal via information and communication means, notifying the user. For example, if the server recognizes that operating a machine during a specific time period is energy-efficient, it can suggest shifting the operation to that time period.

[0463] This system allows users to perform suggested operations and reduce energy consumption. The user's results are then fed back to the server via a detection mechanism, and the model is updated by a mathematical analysis mechanism. Through this process, the system continuously learns and generates even more optimized suggestions.

[0464] As a specific example, in one manufacturing plant, daytime temperature increases were leading to decreased machine efficiency. By introducing this system, the server detected patterns and suggested energy-efficient nighttime operations. This effectively reduced the plant's energy costs.

[0465] An example of a prompt to input into the generative AI model is: "Analyze the operating patterns of factory robots and test and report the optimal suggestions for reducing energy consumption in a generative lab environment."

[0466] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0467] Step 1:

[0468] The server acquires usage data from industrial machinery through detection means. This data includes information such as operating hours, operating mode, and environmental conditions. The acquired data is stored in a database and used as input for subsequent analysis processes.

[0469] Step 2:

[0470] The server inputs the acquired usage data into a mathematical analysis system and analyzes the data using machine learning algorithms. In this process, the server identifies machine usage patterns and generates a model to derive efficient operating methods. As a result, the characteristics of the main usage patterns are output.

[0471] Step 3:

[0472] Based on the generated model, the server sends suggestions for optimal operation methods and resource management to the terminal via information and communication means. For example, it may notify the terminal of energy-efficient operating hours. This allows the terminal to present specific improvement suggestions to the user.

[0473] Step 4:

[0474] The user executes the suggestions received through the terminal. The resulting usage data is then sent back to the server via the detection mechanism. This data is used as feedback and incorporated into the model update process.

[0475] Step 5:

[0476] The server analyzes the feedback data and updates the existing model using mathematical analysis tools. The updated model is used as the basis for future proposal generation. This allows the system to be continuously optimized and provide users with even more effective proposals.

[0477] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0478] This invention combines a system that learns the user's lifestyle patterns and proposes optimal operating methods and energy management with an emotion engine that recognizes the user's emotions. This system comprises sensor means, data analysis means, information communication means, and an emotion engine.

[0479] First, the sensor built into the device collects data on the usage of the home appliance. This data includes not only device usage information but also information about the usage environment. This data is sent to a server and stored in a database.

[0480] Next, the server uses data analysis tools to analyze the collected data and model the user's lifestyle patterns. In addition, an emotion engine estimates the user's emotional state from their voice and facial expressions. This estimation involves analyzing audio and video data collected, for example, via a camera or microphone, to obtain information such as whether the user is relaxed or stressed.

[0481] The server then takes into account both the user's lifestyle and emotional state to suggest the most suitable operating method. For example, if it estimates that the user is tired, it can suggest automatically adjusting the lighting to a softer glow and playing soothing music. This suggestion is sent to the terminal via information and communication, and the terminal notifies the user. The user can then review the suggestion and choose whether or not to implement it.

[0482] Furthermore, the selected action and the subsequent emotional state are fed back from the sensors and analyzed again on the server. This feedback helps to further refine the next suggestion, improving the system's accuracy and user satisfaction.

[0483] In this way, by learning and adapting to the user's lifestyle patterns and emotions in real time, it becomes possible to provide a more human-like environment. This invention is implemented as a next-generation smart home appliance system that simultaneously achieves user comfort and energy efficiency.

[0484] The following describes the processing flow.

[0485] Step 1:

[0486] The device collects real-time data on the usage of home appliances and environmental information through sensors. Furthermore, it simultaneously acquires data to detect the user's emotional state through voice recognition and video analysis.

[0487] Step 2:

[0488] The device transmits collected usage and emotional state data to the server. The server securely stores this data and prepares it for subsequent analysis.

[0489] Step 3:

[0490] The server uses data analysis tools to analyze the received data and model the user's lifestyle patterns. Simultaneously, it operates an emotion engine to evaluate the user's mental state based on the emotion data.

[0491] Step 4:

[0492] The server synthesizes the analysis results and generates optimal action suggestions tailored to the user's state and habits. For example, if the user is showing signs of fatigue, it may include specific instructions such as changing the lighting to a warmer color.

[0493] Step 5:

[0494] The server sends the generated operation suggestions to the terminal using information and communication means. The terminal notifies the user of the suggestions through the user interface. The notification is displayed as a display or audio alert.

[0495] Step 6:

[0496] The user can choose to perform an action based on the received suggestion or ignore the suggestion. The results of this choice are collected again through sensors.

[0497] Step 7:

[0498] The device then sends the user's actions and subsequent emotional state back to the server. The server uses this feedback data to update the model and improve the accuracy of future suggestions. This allows the system to continuously learn and better adapt to the user's needs.

[0499] (Example 2)

[0500] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0501] In recent years, there has been a growing demand for optimized operation efficiency and resource usage of devices in homes and offices. However, conventional systems have struggled to meet individual user needs because they rely solely on regular operating schedules without considering the user's emotional state. Furthermore, the lack of sufficient functionality for continuous system improvement through feedback has prevented them from maximizing user comfort.

[0502] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0503] In this invention, the server includes a detection device for acquiring information related to usage status, a data processing device for analyzing the information acquired from the detection device and learning the user's behavior patterns, an emotion recognition device for recognizing the user's emotional state in addition to the results learned by the data processing device, and a communication device for presenting the user with the optimal operating method and resource management based on the emotional state recognized by the emotion recognition device and the learning results. This enables optimized device operation tailored to the emotional state of each individual user and realizes continuous system improvement using feedback.

[0504] A "detection device" is a device used to acquire information related to usage conditions.

[0505] A "data processing device" is a device used to analyze acquired information and learn the user's behavioral patterns.

[0506] An "emotion recognition device" is a device used to recognize the emotional state of a user, and it has the function of estimating emotions from voice and facial expressions.

[0507] A "communication device" is a device used to present the user with the optimal operating method and resource management based on the analysis results and emotion recognition results.

[0508] This invention is a system for optimizing the operation of devices in the user's living environment and adapting to the user's emotional state. Specifically, this objective is achieved by combining a detection device, a data processing device, an emotion recognition device, and a communication device.

[0509] The terminal collects information about the usage status of home appliances and the environment through a detection device equipped with multiple sensors. This information includes the on / off status of lights, temperature settings, and volume levels. This data is transmitted to a server using wireless communication technology.

[0510] The server analyzes the information it receives using a data processing device and learns user behavior patterns using machine learning algorithms. This involves techniques such as time series analysis and clustering. For example, open-source libraries and commercial cloud-based AI services can be utilized.

[0511] The server further uses an emotion recognition device to estimate the emotional state from the voice and facial expression data collected by the terminal. A voice recognition API is used for voice analysis, and image processing software is used for facial expression analysis. This allows the server to obtain information such as whether the user is relaxed or stressed.

[0512] The optimized operating method is transmitted to the terminal via a communication device based on the analysis results and emotion recognition results. The terminal notifies the user of this information and displays a suggestion on the screen, for example, "To match your current mood, we will soften the lighting and play relaxing music." The user chooses whether to accept the suggestion, and the result of that choice is sent back to the server as feedback.

[0513] As a concrete example, if the server detects that a user is experiencing stress, it suggests changing the lighting to a warmer color and playing relaxing music. This suggestion enables the creation of an optimal environment based on the user's emotions.

[0514] An example of a prompt for a generating AI model could be: "Please describe a system that suggests the most relaxing appliance settings for a user based on appliance usage data and voice / facial expression data. What specific elements does the system analyze?"

[0515] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0516] Step 1:

[0517] The device uses multiple built-in sensors to collect data on the surrounding environment and equipment usage. Inputs include the on / off status of lights, air conditioner temperature settings, volume levels, room temperature, and humidity. This information is collected asynchronously and transmitted to the server wirelessly as output. Specifically, the sensors sense constantly changing environmental data in real time and convert it into digital signals.

[0518] Step 2:

[0519] The server stores the received usage and environmental data in a database. It receives various sensor data transmitted from terminals as input. As output, it organizes this data chronologically and stores it in the database in a format suitable for long-term analysis. Specifically, the server eliminates data duplication and inserts it into tables within the database.

[0520] Step 3:

[0521] The server analyzes stored data and applies machine learning algorithms to learn user behavior patterns. It takes raw usage data stored in a database as input. The output models patterns such as when and what devices users prefer to use. Specifically, it uses Python libraries and cloud service APIs to perform iterative learning using clustering techniques and extract patterns.

[0522] Step 4:

[0523] The device uses its built-in camera and microphone to collect the user's voice and facial expressions. It acquires ambient sounds and visual data from the user's environment as input. It sends audio waveform data and image frames to the server as output. Specifically, the device compresses audio data appropriately and selects and transmits a portion of the image data from a continuous frame to reduce communication load.

[0524] Step 5:

[0525] The server uses an emotion recognition device to estimate emotional states from transmitted audio and facial expression data. It receives audio waveforms and image data transmitted by the terminal as input. As output, the emotional state (e.g., relaxed, stressed) is quantified and stored as analysis results. Specifically, it uses speech recognition APIs and image processing libraries to analyze voice tone and facial features, and reflects the estimated emotion in the database.

[0526] Step 6:

[0527] The server generates optimal operating methods and resource management for the user based on the obtained behavioral patterns and emotional states. It integrates behavioral models and emotional data as input. Outputs include suggestions such as adjusting lighting color temperature or creating music playlists. In terms of specific actions, it selects the most appropriate control signal from multiple candidates and sends the instruction to the terminal via a communication device.

[0528] Step 7:

[0529] The user receives a notification from the device and chooses whether to perform the suggested action. As input, they visually confirm the options displayed on the device screen. As output, they determine their selection through a simple interface operation, and the result is sent to the server. In terms of specific action, the selection is made via the touchscreen, and the selection data is re-introduced into the system as feedback.

[0530] Step 8:

[0531] The server analyzes feedback data and updates its analysis model to improve the accuracy of the system's suggestions. It analyzes user-selected data and subsequently collected sensor data as input. The output is a model update that makes the next suggestion more personalized. Specifically, the model is periodically retrained to reflect the latest data, ensuring the system is always up-to-date.

[0532] (Application Example 2)

[0533] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0534] To improve the customer experience in modern brick-and-mortar stores, personalized customer service is required. However, conventional technology struggles to perceive and reflect customer emotions and individual preferences in real time. Furthermore, the challenge lies in creating an environment optimized for each individual customer while increasing the energy efficiency of store operations.

[0535] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0536] In this invention, the server includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for suggesting optimal operating methods and energy management to the user, and an emotion estimation means for estimating the user's emotional state. This makes it possible to provide individually optimized customer service methods, product suggestions, and store environment adjustments based on the emotions and behavior of customers.

[0537] "Usage data" refers to information about user behavior and the environment acquired by detection devices.

[0538] A "detection device" refers to a device such as a sensor or camera used to acquire usage data.

[0539] "Information analysis methods" refer to technologies that analyze acquired data to estimate user behavior patterns and emotional states.

[0540] "Means of communication" refer to methods and devices for providing information and suggestions to users based on analysis results.

[0541] "Emotion estimation methods" are technologies that evaluate and infer a user's emotional state from voice and facial expression data.

[0542] A "user interface" refers to the screen display or device that allows a user to receive information and perform actions.

[0543] This invention is a system for improving the customer experience in physical stores. The system includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for providing optimal operations and suggestions, and an emotion estimation means for understanding the customer's emotional state.

[0544] The server aggregates data acquired through detection devices, and information analysis tools analyze this data. Specifically, smart glasses and in-store sensors record customer movements and facial expressions via cameras and microphones, and technology is used to estimate emotions from voice and facial features based on this data. Emotion analysis software, such as Microsoft Azure's Emotion API, is used to estimate the customer's level of relaxation and the product categories they are interested in from the analyzed data.

[0545] Based on the emotional state and behavioral patterns it obtains, the server instructs store staff via communication channels on the most appropriate customer service methods and product recommendations. These instructions are displayed in real time on the smart glasses worn by the staff. For example, if it is estimated that a customer is relaxing while holding a book, the staff can suggest calming music or recommend gentle reading materials as related products.

[0546] As a concrete example, when a customer enters a store's cafe and takes a seat, smart glasses inform the staff that "the customer is calm" based on their facial expression. Based on this information, the staff can offer the customer the most suitable cafe menu or a quiet environment for reading. In this way, it becomes possible to create an optimal environment tailored to the customer's emotional state.

[0547] Example prompt: "Please tell me how to estimate the customer's level of relaxation based on their facial expression data and adjust the music and product suggestions accordingly."

[0548] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0549] Step 1:

[0550] The server collects customer usage data through detection devices installed in physical stores, such as cameras and microphones. Customer facial expressions and voice data are captured as input. The server stores this data and formats it as basic data for subsequent processing.

[0551] Step 2:

[0552] The server uses emotion analysis software (e.g., Microsoft Azure Emotion API) to analyze the collected facial and voice data. Using the data formatted in the previous step as input, the server performs emotion analysis. In this analysis process, the data is decomposed into features, and the customer's emotional state (e.g., relaxed, excited, focused) is estimated based on the original data.

[0553] Step 3:

[0554] The server uses information analysis tools to generate a model based on the customer's emotional state and behavioral patterns, and calculates the optimal customer service method or suggested action. It uses the output of emotional analysis as input to infer the environment settings and product information that the customer is presumed to desire. At this stage, data analysis and machine learning models are applied.

[0555] Step 4:

[0556] The server transmits the generated suggestions to the store staff's smart glasses via a communication channel. The input is the optimized suggestions from the previous step. The server visualizes the data so that staff can view it in real time via the display and instructs them to take specific actions (e.g., adjust background music, recommend specific products).

[0557] Step 5:

[0558] Users interact with customers using suggested service methods and product information. Input is suggested information from the server. Based on this, users provide services tailored to customer needs and emotions, improving the customer experience.

[0559] Step 6:

[0560] The terminal acquires the results of the interaction with the customer as data again through a detection device and feeds it back to the server. The input consists of the user's customer service results and subsequent environmental data. The server uses this feedback information to improve the accuracy of the system by incorporating it into future analyses and suggestions.

[0561] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0562] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0563] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0564] [Fourth Embodiment]

[0565] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0566] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0567] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0568] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0569] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0570] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0571] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0572] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0573] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0574] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0575] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0576] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0577] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0578] This invention is constructed as a system equipped with sensor means, data analysis means, and information communication means. First, the sensor means built into the terminal acquires usage data of the home appliance in real time. This includes information such as usage time, operating mode, and ambient environmental conditions (e.g., room temperature and illuminance). This allows the terminal to transmit detailed usage data to the server.

[0579] Next, the server receives this data and analyzes it in detail using data analysis tools. By using machine learning algorithms, the server learns and models the user's lifestyle patterns. For example, it can recognize patterns in which users use specific home appliances at specific times of the day.

[0580] Furthermore, based on the analysis results obtained, the server generates suggestions for optimal operation methods and improvements to energy efficiency. These suggestions are transmitted to the terminal via information and communication means and notified to the user via the terminal's display or a smartphone app. For example, if the server detects a pattern of using the coffee maker at a certain time in the morning, it can suggest setting it to automatically turn on a little before that time.

[0581] As a result, users can reduce energy waste and improve convenience by performing the suggested actions. Furthermore, user selections and actions are recorded again via sensors and fed back into subsequent analysis. This allows the system to continuously learn from user behavior and further optimize its performance.

[0582] Thus, the system of the present invention is implemented as a smart home appliance solution that continuously evolves to match the user's lifestyle, achieving improvements in convenience and energy efficiency.

[0583] The following describes the processing flow.

[0584] Step 1:

[0585] The device collects real-time usage data of home appliances through sensors. This data includes usage time, device operating status, and ambient environmental conditions.

[0586] Step 2:

[0587] The device sends the collected data to the server at regular intervals. This data is stored in a database for use in subsequent analysis.

[0588] Step 3:

[0589] The server performs initial data cleansing on the received data. It removes noise and outliers and formats the data in a way that is suitable for analysis.

[0590] Step 4:

[0591] The server feeds cleansed data to machine learning algorithms to analyze and model the user's lifestyle patterns. This allows it to recognize usage trends at specific times and under specific conditions.

[0592] Step 5:

[0593] The server generates operation suggestions tailored to the user based on the analysis results. This includes automatic setting of operation schedules aimed at improving energy efficiency and suggestions for energy-saving modes.

[0594] Step 6:

[0595] The server sends the generated operation suggestions to the terminal via information and communication means. The terminal notifies the user of the suggestions through the user interface.

[0596] Step 7:

[0597] The user can either accept the suggestion or make their own adjustments. The results of this operation are again recorded by the device's sensors and fed back to the server.

[0598] Step 8:

[0599] The server uses the new feedback data to update and improve the user's model, thereby improving the accuracy of future suggestions.

[0600] (Example 1)

[0601] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0602] Modern home electronic devices are diverse, and their efficient use and energy conservation are essential. However, it is difficult for users to recognize their own behavioral patterns and manually configure devices appropriately, often resulting in wasted energy. To solve this, a system is needed that automatically learns user behavior and suggests optimized operation.

[0603] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0604] In this invention, the server includes detection means for collecting usage information, information processing means for analyzing the information collected from the detection means and modeling the user's behavioral tendencies, communication means for proposing efficient operating methods and energy-saving strategies to the user based on the model obtained by the information processing means, and proposal generation means for generating proposals using a generated AI model. As a result, the user can achieve efficient and energy-efficient use of the equipment without having to recognize their own behavioral patterns.

[0605] "Usage information" refers to information such as time, mode, and environmental conditions related to the use of home appliances and other devices.

[0606] "Detection means" refers to a system that acquires usage information using sensors or other detection devices.

[0607] "Information processing means" refers to technologies and devices used to analyze collected usage information and model user behavior trends and patterns.

[0608] "Communication methods" refer to network technologies and devices used to deliver suggestions to users regarding energy management and operation methods based on analysis results.

[0609] A "generative AI model" refers to an algorithm that uses artificial intelligence to learn from data and generate new suggestions.

[0610] "Suggestion generation method" refers to a system that uses a generation AI model to suggest the optimal operating method according to the user's behavior and usage situation.

[0611] This invention is a system for efficiently managing the usage status of home appliances using terminals installed in each home. The terminals are equipped with various sensors for detecting temperature, illuminance, motion, etc., and these sensors collect usage information in real time.

[0612] The terminal transfers the collected information to the server. The server is equipped with information processing capabilities and uses machine learning algorithms to analyze user behavior trends and usage patterns from the collected data. This analysis makes it possible to model patterns such as the frequent use of specific home appliances during specific time periods.

[0613] Furthermore, the server utilizes the generated AI model to create optimal operation suggestions based on the model obtained from the information processing means. These suggestions are transmitted to the terminal via communication means and presented to the user through the terminal's display or an application installed on the user's smartphone. For example, if a user has a habit of using a coffee maker at a specific time every morning, the server can generate a suggestion to automatically turn on the power at that time.

[0614] This allows users to improve the convenience of their daily lives while saving energy simply by performing the suggested actions. Furthermore, the actions performed by the user are fed back through the terminal and sent to the server. This feedback information is used for subsequent analysis, further enhancing the overall system and improving its ability to provide suggestions tailored to individual users.

[0615] An example of a prompt message might be, "Based on my morning routine, please provide suggestions for improving energy efficiency." This system creates an environment where users can manage their home appliances efficiently and smartly without any hassle.

[0616] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0617] Step 1:

[0618] The device uses its built-in sensors to collect information on the usage status of home appliances. The input data includes environmental information such as the appliance's usage time, mode, room temperature, and illuminance. The device periodically scans this information and outputs the data in streaming format. Specifically, the device acquires information at regular intervals and temporarily stores it in local memory.

[0619] Step 2:

[0620] The terminal sends the usage information it collects to the server. The input data is the usage information output from the terminal. The terminal encodes this data as a digital signal and transfers it to the server using a secure communication protocol. Specifically, the terminal performs batch transfers when a certain amount of data is reached.

[0621] Step 3:

[0622] The server analyzes the usage information it receives using information processing tools. The input data is usage information sent from the terminal. The server uses machine learning algorithms to analyze the data and derive and output user behavior patterns. For example, it may extract patterns in which specific home appliances are frequently used during certain time periods.

[0623] Step 4:

[0624] The server generates suggestions using a generative AI model based on the analysis results. The input data is the user's behavior patterns derived from the analysis. The server inputs this data into the generative AI model and outputs suggestions for optimal operating methods and energy-saving measures. Specifically, the server compares various candidate settings and selects the most effective one.

[0625] Step 5:

[0626] The server generates suggestions and sends them to the terminal via a communication method, notifying the user. The input data is the generated suggestions. The server generates a notification message and outputs it to the user's display or smartphone app via the terminal. Specifically, a notification is made through the user interface based on the message content.

[0627] Step 6:

[0628] The user operates the device to take action based on the suggestions. The input data is the suggestions received via the device. The user reviews the suggestions and performs specific actions through the device's UI. Specific actions include pressing buttons to change the settings of home appliances based on the suggestions.

[0629] Step 7:

[0630] The device collects user behavior data again via sensors and feeds it back to the server. The input data is the actual result of the user's actions. The device analyzes the obtained data and completes the feedback process by sending the results back to the server. Specifically, the device acquires new data from the sensors and sends it to the server in batches.

[0631] (Application Example 1)

[0632] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0633] In modern industry, optimizing machine operation and resource management are crucial challenges. However, currently, many industrial machines are not operated efficiently based on actual usage patterns, potentially leading to energy waste and increased operating costs. To solve this problem, a system is needed that monitors machine operating status in real time and proposes efficient operating methods based on that data.

[0634] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0635] In this invention, the server includes a detection means for acquiring usage data, a mathematical analysis means for analyzing the data acquired from the detection means and learning the user's behavior patterns, an information and communication means for proposing optimal operating methods and resource management to the user based on the results learned by the mathematical analysis means, and a function of the information and communication means for analyzing the operating status of industrial machinery and generating suggestions to improve efficiency. This makes it possible to improve the operating efficiency of industrial machinery and reduce unnecessary energy consumption.

[0636] "Usage data" refers to information indicating the operating status of industrial machinery, including, for example, operating hours, operating mode, and environmental conditions.

[0637] "Detection means" refers to a device or method for acquiring real-time usage data of industrial machinery.

[0638] "Mathematical analysis means" refers to methods and techniques for learning machine operation patterns from acquired usage data and deriving efficient operating methods.

[0639] "Information and communication means" refers to communication technology or devices used to transmit to the user, based on analysis results, suggestions for optimal operation methods and resource management.

[0640] "Behavioral patterns" refer to the repetitive and predictable ways in which industrial machinery is used or exhibits certain tendencies under specific conditions.

[0641] "Resource management" refers to management methods for efficiently operating industrial machinery while minimizing the consumption of energy and materials associated with its use.

[0642] The system for realizing this invention is configured to monitor and optimize the efficient operation of industrial machinery. The server acquires real-time usage data of the industrial machinery using detection means. This includes the machine's operating time, operating mode, and ambient environmental conditions.

[0643] The acquired data is sent to a server and analyzed using mathematical analysis tools. The server uses machine learning algorithms to learn machine usage patterns and model efficient operating methods.

[0644] Based on modeled usage patterns, the system sends suggestions for optimal operation methods and resource management to the terminal via information and communication means, notifying the user. For example, if the server recognizes that operating a machine during a specific time period is energy-efficient, it can suggest shifting the operation to that time period.

[0645] This system allows users to perform suggested operations and reduce energy consumption. The user's results are then fed back to the server via a detection mechanism, and the model is updated by a mathematical analysis mechanism. Through this process, the system continuously learns and generates even more optimized suggestions.

[0646] As a specific example, in one manufacturing plant, daytime temperature increases were leading to decreased machine efficiency. By introducing this system, the server detected patterns and suggested energy-efficient nighttime operations. This effectively reduced the plant's energy costs.

[0647] An example of a prompt to input into the generative AI model is: "Analyze the operating patterns of factory robots and test and report the optimal suggestions for reducing energy consumption in a generative lab environment."

[0648] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0649] Step 1:

[0650] The server acquires usage data from industrial machinery through detection means. This data includes information such as operating hours, operating mode, and environmental conditions. The acquired data is stored in a database and used as input for subsequent analysis processes.

[0651] Step 2:

[0652] The server inputs the acquired usage data into a mathematical analysis system and analyzes the data using machine learning algorithms. In this process, the server identifies machine usage patterns and generates a model to derive efficient operating methods. As a result, the characteristics of the main usage patterns are output.

[0653] Step 3:

[0654] Based on the generated model, the server sends suggestions for optimal operation methods and resource management to the terminal via information and communication means. For example, it may notify the terminal of energy-efficient operating hours. This allows the terminal to present specific improvement suggestions to the user.

[0655] Step 4:

[0656] The user executes the suggestions received through the terminal. The resulting usage data is then sent back to the server via the detection mechanism. This data is used as feedback and incorporated into the model update process.

[0657] Step 5:

[0658] The server analyzes the feedback data and updates the existing model using mathematical analysis tools. The updated model is used as the basis for future proposal generation. This allows the system to be continuously optimized and provide users with even more effective proposals.

[0659] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0660] This invention combines a system that learns the user's lifestyle patterns and proposes optimal operating methods and energy management with an emotion engine that recognizes the user's emotions. This system comprises sensor means, data analysis means, information communication means, and an emotion engine.

[0661] First, the sensor built into the device collects data on the usage of the home appliance. This data includes not only device usage information but also information about the usage environment. This data is sent to a server and stored in a database.

[0662] Next, the server uses data analysis tools to analyze the collected data and model the user's lifestyle patterns. In addition, an emotion engine estimates the user's emotional state from their voice and facial expressions. This estimation involves analyzing audio and video data collected, for example, via a camera or microphone, to obtain information such as whether the user is relaxed or stressed.

[0663] The server then takes into account both the user's lifestyle and emotional state to suggest the most suitable operating method. For example, if it estimates that the user is tired, it can suggest automatically adjusting the lighting to a softer glow and playing soothing music. This suggestion is sent to the terminal via information and communication, and the terminal notifies the user. The user can then review the suggestion and choose whether or not to implement it.

[0664] Furthermore, the selected action and the subsequent emotional state are fed back from the sensors and analyzed again on the server. This feedback helps to further refine the next suggestion, improving the system's accuracy and user satisfaction.

[0665] In this way, by learning and adapting to the user's lifestyle patterns and emotions in real time, it becomes possible to provide a more human-like environment. This invention is implemented as a next-generation smart home appliance system that simultaneously achieves user comfort and energy efficiency.

[0666] The following describes the processing flow.

[0667] Step 1:

[0668] The device collects real-time data on the usage of home appliances and environmental information through sensors. Furthermore, it simultaneously acquires data to detect the user's emotional state through voice recognition and video analysis.

[0669] Step 2:

[0670] The device transmits collected usage and emotional state data to the server. The server securely stores this data and prepares it for subsequent analysis.

[0671] Step 3:

[0672] The server uses data analysis tools to analyze the received data and model the user's lifestyle patterns. Simultaneously, it operates an emotion engine to evaluate the user's mental state based on the emotion data.

[0673] Step 4:

[0674] The server synthesizes the analysis results and generates optimal action suggestions tailored to the user's state and habits. For example, if the user is showing signs of fatigue, it may include specific instructions such as changing the lighting to a warmer color.

[0675] Step 5:

[0676] The server sends the generated operation suggestions to the terminal using information and communication means. The terminal notifies the user of the suggestions through the user interface. The notification is displayed as a display or audio alert.

[0677] Step 6:

[0678] The user can choose to perform an action based on the received suggestion or ignore the suggestion. The results of this choice are collected again through sensors.

[0679] Step 7:

[0680] The device then sends the user's actions and subsequent emotional state back to the server. The server uses this feedback data to update the model and improve the accuracy of future suggestions. This allows the system to continuously learn and better adapt to the user's needs.

[0681] (Example 2)

[0682] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0683] In recent years, there has been a growing demand for optimized operation efficiency and resource usage of devices in homes and offices. However, conventional systems have struggled to meet individual user needs because they rely solely on regular operating schedules without considering the user's emotional state. Furthermore, the lack of sufficient functionality for continuous system improvement through feedback has prevented them from maximizing user comfort.

[0684] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0685] In this invention, the server includes a detection device for acquiring information related to usage status, a data processing device for analyzing the information acquired from the detection device and learning the user's behavior patterns, an emotion recognition device for recognizing the user's emotional state in addition to the results learned by the data processing device, and a communication device for presenting the user with the optimal operating method and resource management based on the emotional state recognized by the emotion recognition device and the learning results. This enables optimized device operation tailored to the emotional state of each individual user and realizes continuous system improvement using feedback.

[0686] A "detection device" is a device used to acquire information related to usage conditions.

[0687] A "data processing device" is a device used to analyze acquired information and learn the user's behavioral patterns.

[0688] An "emotion recognition device" is a device used to recognize the emotional state of a user, and it has the function of estimating emotions from voice and facial expressions.

[0689] A "communication device" is a device used to present the user with the optimal operating method and resource management based on the analysis results and emotion recognition results.

[0690] This invention is a system for optimizing the operation of devices in the user's living environment and adapting to the user's emotional state. Specifically, this objective is achieved by combining a detection device, a data processing device, an emotion recognition device, and a communication device.

[0691] The terminal collects information about the usage status of home appliances and the environment through a detection device equipped with multiple sensors. This information includes the on / off status of lights, temperature settings, and volume levels. This data is transmitted to a server using wireless communication technology.

[0692] The server analyzes the information it receives using a data processing device and learns user behavior patterns using machine learning algorithms. This involves techniques such as time series analysis and clustering. For example, open-source libraries and commercial cloud-based AI services can be utilized.

[0693] The server further uses an emotion recognition device to estimate the emotional state from the voice and facial expression data collected by the terminal. A voice recognition API is used for voice analysis, and image processing software is used for facial expression analysis. This allows the server to obtain information such as whether the user is relaxed or stressed.

[0694] The optimized operating method is transmitted to the terminal via a communication device based on the analysis results and emotion recognition results. The terminal notifies the user of this information and displays a suggestion on the screen, for example, "To match your current mood, we will soften the lighting and play relaxing music." The user chooses whether to accept the suggestion, and the result of that choice is sent back to the server as feedback.

[0695] As a concrete example, if the server detects that a user is experiencing stress, it suggests changing the lighting to a warmer color and playing relaxing music. This suggestion enables the creation of an optimal environment based on the user's emotions.

[0696] An example of a prompt for a generating AI model could be: "Please describe a system that suggests the most relaxing appliance settings for a user based on appliance usage data and voice / facial expression data. What specific elements does the system analyze?"

[0697] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0698] Step 1:

[0699] The device uses multiple built-in sensors to collect data on the surrounding environment and equipment usage. Inputs include the on / off status of lights, air conditioner temperature settings, volume levels, room temperature, and humidity. This information is collected asynchronously and transmitted to the server wirelessly as output. Specifically, the sensors sense constantly changing environmental data in real time and convert it into digital signals.

[0700] Step 2:

[0701] The server stores the received usage and environmental data in a database. It receives various sensor data transmitted from terminals as input. As output, it organizes this data chronologically and stores it in the database in a format suitable for long-term analysis. Specifically, the server eliminates data duplication and inserts it into tables within the database.

[0702] Step 3:

[0703] The server analyzes stored data and applies machine learning algorithms to learn user behavior patterns. It takes raw usage data stored in a database as input. The output models patterns such as when and what devices users prefer to use. Specifically, it uses Python libraries and cloud service APIs to perform iterative learning using clustering techniques and extract patterns.

[0704] Step 4:

[0705] The device uses its built-in camera and microphone to collect the user's voice and facial expressions. It acquires ambient sounds and visual data from the user's environment as input. It sends audio waveform data and image frames to the server as output. Specifically, the device compresses audio data appropriately and selects and transmits a portion of the image data from a continuous frame to reduce communication load.

[0706] Step 5:

[0707] The server uses an emotion recognition device to estimate emotional states from transmitted audio and facial expression data. It receives audio waveforms and image data transmitted by the terminal as input. As output, the emotional state (e.g., relaxed, stressed) is quantified and stored as analysis results. Specifically, it uses speech recognition APIs and image processing libraries to analyze voice tone and facial features, and reflects the estimated emotion in the database.

[0708] Step 6:

[0709] The server generates optimal operating methods and resource management for the user based on the obtained behavioral patterns and emotional states. It integrates behavioral models and emotional data as input. Outputs include suggestions such as adjusting lighting color temperature or creating music playlists. In terms of specific actions, it selects the most appropriate control signal from multiple candidates and sends the instruction to the terminal via a communication device.

[0710] Step 7:

[0711] The user receives a notification from the device and chooses whether to perform the suggested action. As input, they visually confirm the options displayed on the device screen. As output, they determine their selection through a simple interface operation, and the result is sent to the server. In terms of specific action, the selection is made via the touchscreen, and the selection data is re-introduced into the system as feedback.

[0712] Step 8:

[0713] The server analyzes feedback data and updates its analysis model to improve the accuracy of the system's suggestions. It analyzes user-selected data and subsequently collected sensor data as input. The output is a model update that makes the next suggestion more personalized. Specifically, the model is periodically retrained to reflect the latest data, ensuring the system is always up-to-date.

[0714] (Application Example 2)

[0715] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0716] To improve the customer experience in modern brick-and-mortar stores, personalized customer service is required. However, conventional technology struggles to perceive and reflect customer emotions and individual preferences in real time. Furthermore, the challenge lies in creating an environment optimized for each individual customer while increasing the energy efficiency of store operations.

[0717] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0718] In this invention, the server includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for suggesting optimal operating methods and energy management to the user, and an emotion estimation means for estimating the user's emotional state. This makes it possible to provide individually optimized customer service methods, product suggestions, and store environment adjustments based on the emotions and behavior of customers.

[0719] "Usage data" refers to information about user behavior and the environment acquired by detection devices.

[0720] A "detection device" refers to a device such as a sensor or camera used to acquire usage data.

[0721] "Information analysis methods" refer to technologies that analyze acquired data to estimate user behavior patterns and emotional states.

[0722] "Means of communication" refer to methods and devices for providing information and suggestions to users based on analysis results.

[0723] "Emotion estimation methods" are technologies that evaluate and infer a user's emotional state from voice and facial expression data.

[0724] A "user interface" refers to the screen display or device that allows a user to receive information and perform actions.

[0725] This invention is a system for improving the customer experience in physical stores. The system includes a detection device for acquiring usage data, an information analysis means for learning user behavior patterns, a communication means for providing optimal operations and suggestions, and an emotion estimation means for understanding the customer's emotional state.

[0726] The server aggregates data acquired through detection devices, and information analysis tools analyze this data. Specifically, smart glasses and in-store sensors record customer movements and facial expressions via cameras and microphones, and technology is used to estimate emotions from voice and facial features based on this data. Emotion analysis software, such as Microsoft Azure's Emotion API, is used to estimate the customer's level of relaxation and the product categories they are interested in from the analyzed data.

[0727] Based on the emotional state and behavioral patterns it obtains, the server instructs store staff via communication channels on the most appropriate customer service methods and product recommendations. These instructions are displayed in real time on the smart glasses worn by the staff. For example, if it is estimated that a customer is relaxing while holding a book, the staff can suggest calming music or recommend gentle reading materials as related products.

[0728] As a concrete example, when a customer enters a store's cafe and takes a seat, smart glasses inform the staff that "the customer is calm" based on their facial expression. Based on this information, the staff can offer the customer the most suitable cafe menu or a quiet environment for reading. In this way, it becomes possible to create an optimal environment tailored to the customer's emotional state.

[0729] Example prompt: "Please tell me how to estimate the customer's level of relaxation based on their facial expression data and adjust the music and product suggestions accordingly."

[0730] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0731] Step 1:

[0732] The server collects customer usage data through detection devices installed in physical stores, such as cameras and microphones. Customer facial expressions and voice data are captured as input. The server stores this data and formats it as basic data for subsequent processing.

[0733] Step 2:

[0734] The server uses emotion analysis software (e.g., Microsoft Azure Emotion API) to analyze the collected facial and voice data. Using the data formatted in the previous step as input, the server performs emotion analysis. In this analysis process, the data is decomposed into features, and the customer's emotional state (e.g., relaxed, excited, focused) is estimated based on the original data.

[0735] Step 3:

[0736] The server uses information analysis tools to generate a model based on the customer's emotional state and behavioral patterns, and calculates the optimal customer service method or suggested action. It uses the output of emotional analysis as input to infer the environment settings and product information that the customer is presumed to desire. At this stage, data analysis and machine learning models are applied.

[0737] Step 4:

[0738] The server transmits the generated suggestions to the store staff's smart glasses via a communication channel. The input is the optimized suggestions from the previous step. The server visualizes the data so that staff can view it in real time via the display and instructs them to take specific actions (e.g., adjust background music, recommend specific products).

[0739] Step 5:

[0740] Users interact with customers using suggested service methods and product information. Input is suggested information from the server. Based on this, users provide services tailored to customer needs and emotions, improving the customer experience.

[0741] Step 6:

[0742] The terminal acquires the results of the interaction with the customer as data again through a detection device and feeds it back to the server. The input consists of the user's customer service results and subsequent environmental data. The server uses this feedback information to improve the accuracy of the system by incorporating it into future analyses and suggestions.

[0743] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0744] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0745] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0746] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0747] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0748] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0749] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0750] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0751] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0752] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0753] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0754] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0755] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0756] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0757] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0758] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0759] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0760] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0761] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0762] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0763] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0764] The following is further disclosed regarding the embodiments described above.

[0765] (Claim 1)

[0766] A sensor means for acquiring usage data,

[0767] A data analysis means for analyzing data acquired from the aforementioned sensor means and learning the user's lifestyle patterns,

[0768] Information and communication means for proposing optimal operating methods and energy management to the user based on the results learned by the aforementioned data analysis means,

[0769] A system that includes this.

[0770] (Claim 2)

[0771] The system according to claim 1, characterized in that the information communication means feeds back the execution result based on the operation proposed to the user through the sensor means, and the data analysis means updates the model using this feedback.

[0772] (Claim 3)

[0773] The system according to claim 1, characterized in that the information communication means notifies the user of the provided operation suggestion through a user interface.

[0774] "Example 1"

[0775] (Claim 1)

[0776] A detection means for collecting usage information,

[0777] Information processing means for analyzing information collected from the aforementioned detection means and modeling the user's behavioral tendencies,

[0778] Based on the model obtained by the aforementioned information processing means, a communication means for proposing efficient operating methods and energy-saving strategies to the user,

[0779] A proposal generation means for generating proposals using a generative AI model,

[0780] A system that includes this.

[0781] (Claim 2)

[0782] The system according to claim 1, characterized in that the communication means again acquires the operation history based on the proposal presented to the user through the detection means, and the information processing means modifies the model using this operation history.

[0783] (Claim 3)

[0784] The system according to claim 1, characterized in that the communication means provides operation suggestions to the user through a user interface.

[0785] "Application Example 1"

[0786] (Claim 1)

[0787] A detection means for acquiring usage data,

[0788] A mathematical analysis means for analyzing data obtained from the aforementioned detection means and learning the user's behavioral patterns,

[0789] Information and communication means for proposing optimal operating methods and resource management to the user based on the results learned by the mathematical analysis means,

[0790] The aforementioned information and communication means includes a function to analyze the operating status of industrial machinery and generate suggestions to help improve efficiency.

[0791] A system that includes this.

[0792] (Claim 2)

[0793] The system according to claim 1, characterized in that the information communication means feeds back the execution result based on the operation proposed to the user again through the detection means, and the mathematical analysis means updates the model using this feedback.

[0794] (Claim 3)

[0795] The system according to claim 1, characterized in that the information communication means notifies the user of the provided operation suggestion through a user interface.

[0796] "Example 2 of combining an emotion engine"

[0797] (Claim 1)

[0798] A detection device for acquiring information related to usage status,

[0799] A data processing device for analyzing information acquired from the aforementioned detection device and learning the user's behavior patterns,

[0800] In addition to the results learned by the aforementioned data processing device, an emotion recognition device for recognizing the user's emotional state is provided.

[0801] A communication device for presenting the user with the optimal operating method and resource management based on the emotional state recognized by the emotion recognition device and the learning results,

[0802] A system that includes this.

[0803] (Claim 2)

[0804] The system according to claim 1, characterized in that the communication device feeds back the execution result based on the action presented to the user again through the detection device, and the data processing device uses this feedback to improve the analysis model.

[0805] (Claim 3)

[0806] The system according to claim 1, characterized in that the communication device notifies the user of the generated operation proposal through a human-machine connection device.

[0807] "Application example 2 when combining with an emotional engine"

[0808] (Claim 1)

[0809] A detection device for acquiring usage data,

[0810] An information analysis means for analyzing data acquired from the aforementioned detection device and learning user behavior patterns,

[0811] A communication means for proposing the optimal operating method and energy management to the user based on the results learned by the aforementioned information analysis means,

[0812] An emotion estimation means for estimating the emotional state of the user,

[0813] A system that includes this.

[0814] (Claim 2)

[0815] The system according to claim 1, characterized in that the transmission means feeds back the execution result based on the operation proposed to the user through the detection device, the information analysis means updates the model using this feedback, and further makes a proposal taking into account the emotional state information from the emotion estimation means.

[0816] (Claim 3)

[0817] The system according to claim 1, characterized in that the communication means notifies the user of the provided operation suggestion through a user interface and adjusts the content of the interface display based on the user's emotional state. [Explanation of Symbols]

[0818] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A sensor means for acquiring usage data, A data analysis means for analyzing data acquired from the aforementioned sensor means and learning the user's lifestyle patterns, Information and communication means for proposing optimal operating methods and energy management to the user based on the results learned by the aforementioned data analysis means, A system that includes this.

2. The system according to claim 1, characterized in that the information communication means feeds back the execution result based on the operation proposed to the user through the sensor means, and the data analysis means updates the model using this feedback.

3. The system according to claim 1, characterized in that the information communication means notifies the user of the provided operation suggestion through a user interface.

Images