system

Abstract

The system according to this embodiment aims to automate and streamline household chores using smart home appliances. [Solution] The system according to the embodiment comprises a linking unit, a management unit, and an operation unit. The linking unit uses an AI agent to automate household chores in cooperation with smart home appliances. The management unit manages the household chore schedule of the smart home appliances linked by the linking unit and provides reminders. The operation unit allows the user to operate the smart home appliances by voice or text based on the household chore schedule managed by the management unit.

Description

Technical Field

[0004] ,

[0006] , , , ,

[0005] , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, the automation of housework using smart home appliances has not been sufficiently carried out, and there is room for improvement in the efficiency of housework.

[0005] The system according to the embodiment aims to automate and improve the efficiency of housework using smart home appliances.

Means for Solving the Problems

[0006] The system according to this embodiment comprises a linking unit, a management unit, and an operation unit. The linking unit uses an AI agent to automate household chores in cooperation with smart home appliances. The management unit manages the household chore schedules of the smart home appliances linked by the linking unit and provides reminders. The operation unit allows the user to operate the smart home appliances by voice or text based on the household chore schedule managed by the management unit. [Effects of the Invention]

[0007] The system according to this embodiment can automate and streamline household chores using smart home appliances. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between multiple computers. Examples of communication standards applicable to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The system according to an embodiment of the present invention is a system that automates daily household chores such as cleaning, cooking, laundry, feeding pets, and plant care by working in conjunction with smart home appliances equipped with an AI agent. In this system, the AI ​​agent works in conjunction with smart home appliances to automate daily household chores such as cleaning, cooking, laundry, feeding pets, and plant care. Next, the AI ​​agent manages the schedule of household chores and provides reminders. In addition, for cooking, it suggests various recipes and assists in creating necessary shopping lists. Furthermore, the user can operate the smart home appliances through a simple human interface such as voice or text. This system realizes the automation and efficiency of household chores, allowing the user to dedicate time to what they truly want to do and improving their quality of life. In addition, given the expected expansion of the smart home appliance and home robotics markets, this system also has great significance in the market. For example, the AI ​​agent works in conjunction with a cleaning robot to automate room cleaning. The AI ​​agent manages the cleaning schedule and provides reminders to the user. The user can operate the cleaning robot by voice or text. The AI ​​agent also works in conjunction with a cooking robot to automate cooking. The AI ​​agent manages cooking schedules and provides reminders to the user. The user can operate the cooking robot by voice or text. Furthermore, the AI ​​agent works in conjunction with the washing machine to automate laundry. The AI ​​agent manages laundry schedules and provides reminders to the user. The user can operate the washing machine by voice or text. In this way, the AI ​​agent can automate daily household chores such as cleaning, cooking, laundry, feeding pets, and plant care, reducing the burden on the user. As a result, systems equipped with AI agents can automate daily household chores and improve the user's quality of life.

[0029] The household automation system according to the embodiment comprises a linking unit, a management unit, and an operation unit. The linking unit automates household chores by linking an AI agent with smart home appliances. For example, the linking unit automates room cleaning by linking with a cleaning robot. The linking unit manages the cleaning schedule and reminds the user. The linking unit allows the user to operate the cleaning robot by voice or text. For example, the linking unit automates cooking by linking with a cooking robot. The linking unit manages the cooking schedule and reminds the user. The linking unit allows the user to operate the cooking robot by voice or text. For example, the linking unit automates laundry by linking with a washing machine. The linking unit manages the laundry schedule and reminds the user. The linking unit allows the user to operate the washing machine by voice or text. The management unit manages the household schedules of smart home appliances linked by the linking unit and provides reminders. For example, the management unit manages the cleaning schedule and reminds the user. The management unit manages the cooking schedule and reminds the user. The management unit manages the laundry schedule and sends reminders to the user. The management unit manages the cleaning schedule and sends reminders to the user, for example. The management unit manages the cooking schedule and sends reminders to the user. The management unit manages the laundry schedule and sends reminders to the user. The operation unit allows the user to operate smart home appliances by voice or text based on the household chore schedule managed by the management unit. The operation unit allows the user to operate a cleaning robot by voice. The operation unit allows the user to operate a cleaning robot by text. The operation unit allows the user to operate a cooking robot by voice. The operation unit allows the user to operate a cooking robot by text. The operation unit allows the user to operate a washing machine by voice. The operation unit allows the user to operate a washing machine by text. As a result, the household chore automation system according to the embodiment can automate household chores in cooperation with smart home appliances, and streamline schedule management and operation.

[0030] The integration unit automates household chores by having an AI agent work with smart home appliances. Specifically, the integration unit communicates with smart home appliances such as robotic vacuum cleaners, cooking robots, and washing machines, and controls the operation of each device. For example, when integrating with a robotic vacuum cleaner, the integration unit provides the AI ​​agent with room layout information and cleaning priorities to calculate the optimal cleaning route. The cleaning schedule is automatically set based on the user's lifestyle and past cleaning history, and the user is reminded. The user can operate the robotic vacuum cleaner through voice commands or text messages, for example, by giving instructions such as "clean the living room." When integrating with a cooking robot, the integration unit provides the AI ​​agent with recipe information and ingredient inventory status to automate the cooking process. The cooking schedule is also set based on the user's eating preferences and past cooking history, and the user is reminded. The user can give instructions to the cooking robot via voice or text, such as "make dinner." When integrating with a washing machine, the integration unit selects the optimal washing program according to the amount and type of laundry and manages the washing schedule. The laundry schedule is set based on the user's lifestyle and past laundry history, and the user is reminded accordingly. The user can give instructions to the washing machine via voice or text, such as "start washing." In this way, the integrated unit can automate household chores in conjunction with various smart home appliances, reducing the burden on the user. Furthermore, the integrated unit also has a function to monitor the status of each device and notify the user if an abnormality occurs. For example, if a cleaning robot gets stuck on an obstacle, a cooking robot fails to cut ingredients properly, or a washing machine detects a water leak, it will immediately send an alert to the user to prompt appropriate action. In this way, the integrated unit can not only automate household chores but also play a role in improving the safety and reliability of home appliances.

[0031] The Management Department manages and reminds users of the household chore schedules of smart home appliances linked by the Integration Department. Specifically, the Management Department centrally manages the operating schedules of each smart home appliance and sets the optimal schedule according to the user's lifestyle and preferences. For example, the cleaning schedule is automatically adjusted based on the time the user is out and the degree of dirtiness in the room. The Management Department reminds the user of the start and completion times of cleaning and suggests schedule changes as needed. Cooking schedules are also set based on the user's food preferences and past cooking history, and the user is reminded accordingly. For example, if the user prefers a particular dish on a specific day of the week, the cooking robot's schedule is automatically adjusted based on that information. Laundry schedules are also set based on the user's lifestyle and past laundry history, and the user is reminded accordingly. For example, if the user has a habit of doing laundry all at once on weekends, the washing machine's schedule is automatically adjusted based on that information. The Management Department centrally manages these schedules and reminds users at the appropriate time, thereby improving the efficiency of household chores. The Management Department also has a function to monitor the operating status of each smart home appliance and notify the user if an abnormality occurs. For example, if a cleaning robot fails to operate as scheduled, a cooking robot miscuts ingredients, or a washing machine detects a water leak, the system immediately sends an alert to the user, prompting appropriate action. This allows the management department to not only manage household chore schedules but also improve the safety and reliability of home appliances. Furthermore, the management department can collect user feedback and continuously improve the accuracy of schedules and the timing of reminders. For example, if a user is dissatisfied with the timing of reminders, the system adjusts the reminder timing based on that feedback. The management department can also reliably transmit information using multiple communication methods. For example, it can reliably deliver important information using not only smartphone notifications but also voice calls, SMS, and email. This allows the management department to provide users with information quickly and reliably, improving the efficiency and safety of household chores.

[0032] The control unit allows users to operate smart home appliances via voice or text based on a household chore schedule managed by the management unit. Specifically, the control unit uses voice recognition and natural language processing technologies to accurately understand user instructions and transmit them to the smart home appliances. For example, if a user gives the voice command "Clean the living room," the control unit analyzes the command and sends a command to the cleaning robot to start cleaning the living room. Similarly, if a user inputs "Make dinner" via text on their smartphone or tablet, the control unit analyzes the command and sends a command to the cooking robot to start preparing dinner. The control unit responds immediately to user instructions and controls the operation of smart home appliances to improve the efficiency of household chores. The control unit also records the user's instruction history and can make optimal suggestions based on past instructions. For example, if a user has a habit of cooking a specific dish on a specific day of the week in the past, the control unit can use that information to suggest, "Shall we make curry today?" Furthermore, the control unit can collect user feedback and continuously improve the accuracy of its voice recognition and natural language processing. For example, if a user misinterprets a particular instruction, the speech recognition model is retrained based on that feedback to improve recognition accuracy for subsequent uses. Furthermore, the control unit can reliably transmit information using multiple communication methods. For instance, it can reliably deliver important information using not only smartphone notifications but also voice calls, SMS, and email. This allows the control unit to provide information to the user quickly and reliably, improving the efficiency and safety of household chores. In addition, the control unit has features to encrypt and store voice and text data to protect user privacy and prevent unauthorized access by third parties. This allows the control unit to improve the efficiency and safety of household chores while protecting user privacy.

[0033] The integrated unit can automate everyday household chores such as cleaning, cooking, laundry, feeding pets, and plant care. For example, it can work with a cleaning robot to automate room cleaning. The unit manages the cleaning schedule and sends reminders to the user. The unit allows the user to control the cleaning robot by voice or text. For example, it can work with a cooking robot to automate cooking. The unit manages the cooking schedule and sends reminders to the user. The unit allows the user to control the cooking robot by voice or text. For example, it can work with a washing machine to automate laundry. The unit manages the laundry schedule and sends reminders to the user. The unit allows the user to control the washing machine by voice or text. By automating everyday household chores, the burden on the user can be reduced.

[0034] The management unit can manage household chore schedules and send reminders to users. For example, the management unit can manage cleaning schedules and send reminders to users. The management unit can manage cooking schedules and send reminders to users. The management unit can manage laundry schedules and send reminders to users. For example, the management unit can manage cleaning schedules and send reminders to users. The management unit can manage cooking schedules and send reminders to users. The management unit can manage laundry schedules and send reminders to users. This allows users to remember to do household chores through schedule management and reminders. Some or all of the above processes in the management unit may be performed using AI, for example, or not using AI. For example, the management unit can input household chore schedules into AI and have the AI ​​execute reminders.

[0035] The control unit allows users to operate smart home appliances using their voice or text. For example, the control unit allows users to operate a cleaning robot by voice. The control unit allows users to operate a cleaning robot by text. The control unit allows users to operate a cooking robot by voice. The control unit allows users to operate a cooking robot by text. The control unit allows users to operate a washing machine by voice. The control unit allows users to operate a washing machine by text. This makes it easy for users to operate smart home appliances using their voice or text. Some or all of the above-described processes in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input the user's voice into AI and have AI perform the operation of the smart home appliance.

[0036] The management department can suggest cooking recipes and assist in creating necessary shopping lists. For example, the management department suggests cooking recipes based on the user's preferences. The management department creates necessary shopping lists based on the suggested recipes. The management department provides the shopping lists to the user and assists with shopping. For example, the management department analyzes the user's past cooking history and suggests the most suitable recipes. The management department creates necessary shopping lists based on the suggested recipes. The management department provides the shopping lists to the user and assists with shopping. This reduces the effort required for the user to cook by suggesting cooking recipes and creating shopping lists. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input the user's preferences and past cooking history into the AI ​​and have the AI ​​perform recipe suggestion and shopping list creation.

[0037] The integration unit can perform household chores in conjunction with smart home appliances. For example, the integration unit can work with a cleaning robot to clean a room. The integration unit manages the cleaning schedule and sends reminders to the user. The integration unit allows the user to control the cleaning robot by voice or text. For example, the integration unit can work with a cooking robot to cook. The integration unit manages the cooking schedule and sends reminders to the user. The integration unit allows the user to control the cooking robot by voice or text. For example, the integration unit can work with a washing machine to do laundry. The integration unit manages the laundry schedule and sends reminders to the user. The integration unit allows the user to control the washing machine by voice or text. This improves the efficiency of household chores by performing them in conjunction with smart home appliances. Some or all of the above processes in the integration unit may be performed using AI, for example, or without AI. For example, the integration unit can input the operation of smart home appliances into AI and have AI perform household chores.

[0038] The collaborative unit can analyze the user's past household chore history when performing household chores and select the optimal method of execution. For example, the collaborative unit can analyze the frequency and method of cleaning performed by the user in the past and propose an optimal cleaning schedule. The collaborative unit can suggest new recipes based on recipes the user has made in the past. The collaborative unit can analyze the method of laundry performed by the user in the past and propose an optimal laundry method. In this way, the optimal method of execution of household chores can be selected by analyzing past household chore history. Some or all of the above processes in the collaborative unit may be performed using AI, for example, or without AI. For example, the collaborative unit can input the user's past household chore history data into AI and have the AI ​​select the optimal method of execution.

[0039] The collaborative unit can filter household chores based on the user's current lifestyle and areas of interest. For example, if the user is busy, the collaborative unit will prioritize suggesting chores that can be completed quickly. If the user is interested in health, the collaborative unit will suggest healthy cooking recipes. If the user is interested in pets, the collaborative unit will prioritize suggesting chores related to feeding and caring for pets. In this way, by filtering chores according to the user's lifestyle and areas of interest, more appropriate chores can be suggested. Some or all of the above processing in the collaborative unit may be performed using AI, for example, or not. For example, the collaborative unit can input data on the user's lifestyle and areas of interest into the AI ​​and have the AI ​​perform the chore filtering.

[0040] The collaborative unit can prioritize the execution of highly relevant household chores by considering the user's geographical location information when performing household chores. For example, if the user is at home, the collaborative unit will prioritize chores such as cleaning and cooking. If the user is out, the collaborative unit will prioritize chores such as feeding pets and caring for plants. If the user is in a specific location, the collaborative unit will prioritize chores related to that location. This improves the efficiency of household chores by prioritizing them based on the user's geographical location information. Some or all of the above processing in the collaborative unit may be performed using AI, for example, or without AI. For example, the collaborative unit can input the user's geographical location information into AI and have the AI ​​perform highly relevant household chores.

[0041] The integration unit can analyze the user's social media activity while performing household chores and execute related chores. For example, if the user posts about cooking on social media, the integration unit will prioritize cooking chores. If the user posts about pets on social media, the integration unit will prioritize feeding and caring for pets. If the user posts about plants on social media, the integration unit will prioritize caring for plants. This allows the system to provide household chores tailored to the user's interests by performing chores based on the user's social media activity. Some or all of the above processing in the integration unit may be performed using AI, for example, or without AI. For example, the integration unit can input the user's social media activity data into AI and have the AI ​​perform related household chores.

[0042] The management department can optimize the current schedule by referring to past schedule data when managing household chore schedules. For example, the management department can analyze the schedule of household chores that the user has performed in the past and propose an optimal schedule. The management department can optimize the current schedule by referring to the frequency of household chores that the user has performed in the past. The management department can optimize the current schedule by referring to the time of day that the user has performed household chores in the past. In this way, the current schedule can be optimized by referring to past schedule data. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input past schedule data into AI and have AI perform the optimization of the current schedule.

[0043] The management department can apply different management methods to each category of household chore when managing household chore schedules. For example, the management department can manage cleaning schedules with different frequencies for each room. The management department can manage cooking schedules taking into account the expiration dates of ingredients. The management department can manage laundry schedules according to the type of clothing and how often it is used. By applying different management methods to each category of household chore, the management of household chore schedules becomes more efficient. Some or all of the above processes in the management department may be performed using AI, for example, or not. For example, the management department can input data for each category of household chore into the AI ​​and have the AI ​​apply different management methods.

[0044] The management department can analyze schedule changes based on the submission timing of household chores when managing household chore schedules. For example, if household chores are submitted early, the management department will execute them earlier. If household chores are submitted late, the management department will execute them later. The management department will adjust the schedule priorities according to the submission timing of household chores. In this way, schedule optimization can be achieved by analyzing schedule changes based on the submission timing of household chores. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input household chore submission timing data into AI and have the AI ​​perform the analysis of schedule changes.

[0045] The management department can analyze household chore schedules by referring to relevant market data. For example, the management department can propose an optimal schedule based on the relevant market data. The management department can adjust schedule priorities by referring to the relevant market data. The management department can analyze the relevant market data and propose changes to the schedule. In this way, the schedule can be optimized by referring to the relevant market data. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input the relevant market data for household chores into AI and have the AI ​​perform the schedule analysis.

[0046] The control unit can adjust the level of detail of an operation based on its importance during operation. For example, the control unit provides detailed instructions for high-importance chores and simplified instructions for low-importance chores. The control unit adjusts how the instructions are displayed according to their importance. This makes it easier for the user to operate the system by adjusting the level of detail according to the importance of the chore. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input chore importance data into the AI ​​and have the AI ​​adjust the level of detail of the operations.

[0047] The control unit can apply different operation algorithms depending on the category of household chore during operation. For example, for cleaning, the control unit can apply a different algorithm for each room. For cooking, the control unit can apply a different algorithm for each recipe. For laundry, the control unit can apply a different algorithm for each type of clothing. This allows the control unit to apply the optimal operation algorithm according to the category of household chore. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input household chore category data into the AI ​​and have the AI ​​execute the application of different operation algorithms.

[0048] The control unit can determine the priority of operations based on the submission timing of household chores during operation. For example, the control unit will give a higher priority to household chores that are submitted earlier, and a lower priority to household chores that are submitted later. The control unit adjusts the priority of operations according to the submission timing. This allows household chores to be performed efficiently by determining the priority of operations based on the submission timing. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input household chore submission timing data into AI and have the AI ​​perform the determination of operation priorities.

[0049] The control unit can adjust the order of operations based on the relevance of household chores during operation. For example, the control unit prioritizes the order of operations for highly relevant chores. For less relevant chores, the control unit postpones the order of operations. The control unit adjusts the order of operations according to the relevance of the chores. This allows for efficient performance of household chores by adjusting the order of operations based on their relevance. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input data on the relevance of chores into the AI ​​and have the AI ​​perform the adjustment of the order of operations.

[0050] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0051] The integration unit not only automates household chores by linking with smart home appliances, but can also acquire user health data and suggest chores tailored to their health condition. For example, if the user's health data indicates accumulated fatigue, it will prioritize relaxing chores and postpone strenuous tasks such as cleaning and cooking. Furthermore, if the user's health data reveals a lack of exercise, it can suggest chores that incorporate light exercise. It can also support a healthy lifestyle by suggesting nutritionally balanced recipes based on the user's health data. In this way, by suggesting chores tailored to the user's health condition, it can support the maintenance of the user's health.

[0052] In addition to automating household chores, the integration system can also suggest chores based on the user's hobbies and preferences. For example, if a user enjoys gardening, it will prioritize suggesting plant care chores. If a user enjoys cooking, it can suggest new recipes and cooking methods to expand their enjoyment of cooking. Furthermore, if a user has pets, it can prioritize suggesting pet feeding and care chores to enrich their time with their pets. In this way, by suggesting chores that match the user's hobbies and preferences, they can enjoy doing household chores.

[0053] In addition to managing household chore schedules, the management department can also optimize schedules according to the user's lifestyle. For example, if a user is a night owl, household chores that can be done at night will be prioritized in the schedule. Similarly, if a user is an early riser, household chores that can be done in the morning will be prioritized in the schedule. Furthermore, the timing of household chores can be adjusted according to the user's lifestyle, and a manageable schedule can be proposed. By optimizing the schedule according to the user's lifestyle, the burden of household chores can be reduced.

[0054] The control unit can also provide guidance to users when operating smart home appliances. For example, if a user is operating a robotic vacuum cleaner for the first time, it can guide them through the operation procedure step by step. It can also guide users through the operation procedure based on recipes when operating a cooking robot. Furthermore, it can guide users through the optimal operation procedure according to the type of laundry when operating a washing machine. This allows users to operate smart home appliances without confusion, improving the efficiency of household chores.

[0055] In addition to managing household chore schedules, the management department can also monitor users' progress in real time and adjust schedules as needed. For example, if a user finishes a chore earlier than planned, the schedule can be adjusted to move the next chore forward. Conversely, if a user is behind schedule, the schedule can be adjusted to postpone the next chore. Furthermore, if a user forgets a chore, a reminder can be sent to encourage them to complete it. This improves the efficiency of household chores by adjusting schedules according to the user's progress.

[0056] The following briefly describes the processing flow for example form 1.

[0057] Step 1: The integration unit uses an AI agent to automate household chores by coordinating with smart home appliances. For example, it will work with cleaning robots, cooking robots, washing machines, etc., to automate each of these tasks. Step 2: The management department manages the household chore schedules of smart home appliances linked by the integration department and sends reminders to users. For example, it manages the schedules for cleaning, cooking, and laundry and sends reminders for each chore. Step 3: The control unit allows users to operate smart home appliances via voice or text based on the household schedule managed by the management unit. For example, users can control robotic vacuums, cooking robots, and washing machines via voice or text.

[0058] (Example of form 2) The system according to an embodiment of the present invention is a system that automates daily household chores such as cleaning, cooking, laundry, feeding pets, and plant care by working in conjunction with smart home appliances equipped with an AI agent. In this system, the AI ​​agent works in conjunction with smart home appliances to automate daily household chores such as cleaning, cooking, laundry, feeding pets, and plant care. Next, the AI ​​agent manages the schedule of household chores and provides reminders. In addition, for cooking, it suggests various recipes and assists in creating necessary shopping lists. Furthermore, the user can operate the smart home appliances through a simple human interface such as voice or text. This system realizes the automation and efficiency of household chores, allowing the user to dedicate time to what they truly want to do and improving their quality of life. In addition, given the expected expansion of the smart home appliance and home robotics markets, this system also has great significance in the market. For example, the AI ​​agent works in conjunction with a cleaning robot to automate room cleaning. The AI ​​agent manages the cleaning schedule and provides reminders to the user. The user can operate the cleaning robot by voice or text. The AI ​​agent also works in conjunction with a cooking robot to automate cooking. The AI ​​agent manages cooking schedules and provides reminders to the user. The user can operate the cooking robot by voice or text. Furthermore, the AI ​​agent works in conjunction with the washing machine to automate laundry. The AI ​​agent manages laundry schedules and provides reminders to the user. The user can operate the washing machine by voice or text. In this way, the AI ​​agent can automate daily household chores such as cleaning, cooking, laundry, feeding pets, and plant care, reducing the burden on the user. As a result, systems equipped with AI agents can automate daily household chores and improve the user's quality of life.

[0059] The household automation system according to the embodiment comprises a linking unit, a management unit, and an operation unit. The linking unit automates household chores by linking an AI agent with smart home appliances. For example, the linking unit automates room cleaning by linking with a cleaning robot. The linking unit manages the cleaning schedule and reminds the user. The linking unit allows the user to operate the cleaning robot by voice or text. For example, the linking unit automates cooking by linking with a cooking robot. The linking unit manages the cooking schedule and reminds the user. The linking unit allows the user to operate the cooking robot by voice or text. For example, the linking unit automates laundry by linking with a washing machine. The linking unit manages the laundry schedule and reminds the user. The linking unit allows the user to operate the washing machine by voice or text. The management unit manages the household schedules of smart home appliances linked by the linking unit and provides reminders. For example, the management unit manages the cleaning schedule and reminds the user. The management unit manages the cooking schedule and reminds the user. The management unit manages the laundry schedule and sends reminders to the user. The management unit manages the cleaning schedule and sends reminders to the user, for example. The management unit manages the cooking schedule and sends reminders to the user. The management unit manages the laundry schedule and sends reminders to the user. The operation unit allows the user to operate smart home appliances by voice or text based on the household chore schedule managed by the management unit. The operation unit allows the user to operate a cleaning robot by voice. The operation unit allows the user to operate a cleaning robot by text. The operation unit allows the user to operate a cooking robot by voice. The operation unit allows the user to operate a cooking robot by text. The operation unit allows the user to operate a washing machine by voice. The operation unit allows the user to operate a washing machine by text. As a result, the household chore automation system according to the embodiment can automate household chores in cooperation with smart home appliances, and streamline schedule management and operation.

[0060] The integration unit automates household chores by having an AI agent work with smart home appliances. Specifically, the integration unit communicates with smart home appliances such as robotic vacuum cleaners, cooking robots, and washing machines, and controls the operation of each device. For example, when integrating with a robotic vacuum cleaner, the integration unit provides the AI ​​agent with room layout information and cleaning priorities to calculate the optimal cleaning route. The cleaning schedule is automatically set based on the user's lifestyle and past cleaning history, and the user is reminded. The user can operate the robotic vacuum cleaner through voice commands or text messages, for example, by giving instructions such as "clean the living room." When integrating with a cooking robot, the integration unit provides the AI ​​agent with recipe information and ingredient inventory status to automate the cooking process. The cooking schedule is also set based on the user's eating preferences and past cooking history, and the user is reminded. The user can give instructions to the cooking robot via voice or text, such as "make dinner." When integrating with a washing machine, the integration unit selects the optimal washing program according to the amount and type of laundry and manages the washing schedule. The laundry schedule is set based on the user's lifestyle and past laundry history, and the user is reminded accordingly. The user can give instructions to the washing machine via voice or text, such as "start washing." In this way, the integrated unit can automate household chores in conjunction with various smart home appliances, reducing the burden on the user. Furthermore, the integrated unit also has a function to monitor the status of each device and notify the user if an abnormality occurs. For example, if a cleaning robot gets stuck on an obstacle, a cooking robot fails to cut ingredients properly, or a washing machine detects a water leak, it will immediately send an alert to the user to prompt appropriate action. In this way, the integrated unit can not only automate household chores but also play a role in improving the safety and reliability of home appliances.

[0061] The Management Department manages and reminds users of the household chore schedules of smart home appliances linked by the Integration Department. Specifically, the Management Department centrally manages the operating schedules of each smart home appliance and sets the optimal schedule according to the user's lifestyle and preferences. For example, the cleaning schedule is automatically adjusted based on the time the user is out and the degree of dirtiness in the room. The Management Department reminds the user of the start and completion times of cleaning and suggests schedule changes as needed. Cooking schedules are also set based on the user's food preferences and past cooking history, and the user is reminded accordingly. For example, if the user prefers a particular dish on a specific day of the week, the cooking robot's schedule is automatically adjusted based on that information. Laundry schedules are also set based on the user's lifestyle and past laundry history, and the user is reminded accordingly. For example, if the user has a habit of doing laundry all at once on weekends, the washing machine's schedule is automatically adjusted based on that information. The Management Department centrally manages these schedules and reminds users at the appropriate time, thereby improving the efficiency of household chores. The Management Department also has a function to monitor the operating status of each smart home appliance and notify the user if an abnormality occurs. For example, if a cleaning robot fails to operate as scheduled, a cooking robot miscuts ingredients, or a washing machine detects a water leak, the system immediately sends an alert to the user, prompting appropriate action. This allows the management department to not only manage household chore schedules but also improve the safety and reliability of home appliances. Furthermore, the management department can collect user feedback and continuously improve the accuracy of schedules and the timing of reminders. For example, if a user is dissatisfied with the timing of reminders, the system adjusts the reminder timing based on that feedback. The management department can also reliably transmit information using multiple communication methods. For example, it can reliably deliver important information using not only smartphone notifications but also voice calls, SMS, and email. This allows the management department to provide users with information quickly and reliably, improving the efficiency and safety of household chores.

[0062] The control unit allows users to operate smart home appliances via voice or text based on a household chore schedule managed by the management unit. Specifically, the control unit uses voice recognition and natural language processing technologies to accurately understand user instructions and transmit them to the smart home appliances. For example, if a user gives the voice command "Clean the living room," the control unit analyzes the command and sends a command to the cleaning robot to start cleaning the living room. Similarly, if a user inputs "Make dinner" via text on their smartphone or tablet, the control unit analyzes the command and sends a command to the cooking robot to start preparing dinner. The control unit responds immediately to user instructions and controls the operation of smart home appliances to improve the efficiency of household chores. The control unit also records the user's instruction history and can make optimal suggestions based on past instructions. For example, if a user has a habit of cooking a specific dish on a specific day of the week in the past, the control unit can use that information to suggest, "Shall we make curry today?" Furthermore, the control unit can collect user feedback and continuously improve the accuracy of its voice recognition and natural language processing. For example, if a user misinterprets a particular instruction, the speech recognition model is retrained based on that feedback to improve recognition accuracy for subsequent uses. Furthermore, the control unit can reliably transmit information using multiple communication methods. For instance, it can reliably deliver important information using not only smartphone notifications but also voice calls, SMS, and email. This allows the control unit to provide information to the user quickly and reliably, improving the efficiency and safety of household chores. In addition, the control unit has features to encrypt and store voice and text data to protect user privacy and prevent unauthorized access by third parties. This allows the control unit to improve the efficiency and safety of household chores while protecting user privacy.

[0063] The integrated unit can automate everyday household chores such as cleaning, cooking, laundry, feeding pets, and plant care. For example, it can work with a cleaning robot to automate room cleaning. The unit manages the cleaning schedule and sends reminders to the user. The unit allows the user to control the cleaning robot by voice or text. For example, it can work with a cooking robot to automate cooking. The unit manages the cooking schedule and sends reminders to the user. The unit allows the user to control the cooking robot by voice or text. For example, it can work with a washing machine to automate laundry. The unit manages the laundry schedule and sends reminders to the user. The unit allows the user to control the washing machine by voice or text. By automating everyday household chores, the burden on the user can be reduced.

[0064] The management unit can manage household chore schedules and send reminders to users. For example, the management unit can manage cleaning schedules and send reminders to users. The management unit can manage cooking schedules and send reminders to users. The management unit can manage laundry schedules and send reminders to users. For example, the management unit can manage cleaning schedules and send reminders to users. The management unit can manage cooking schedules and send reminders to users. The management unit can manage laundry schedules and send reminders to users. This allows users to remember to do household chores through schedule management and reminders. Some or all of the above processes in the management unit may be performed using AI, for example, or not using AI. For example, the management unit can input household chore schedules into AI and have the AI ​​execute reminders.

[0065] The control unit allows users to operate smart home appliances using their voice or text. For example, the control unit allows users to operate a cleaning robot by voice. The control unit allows users to operate a cleaning robot by text. The control unit allows users to operate a cooking robot by voice. The control unit allows users to operate a cooking robot by text. The control unit allows users to operate a washing machine by voice. The control unit allows users to operate a washing machine by text. This makes it easy for users to operate smart home appliances using their voice or text. Some or all of the above-described processes in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input the user's voice into AI and have AI perform the operation of the smart home appliance.

[0066] The management department can suggest cooking recipes and assist in creating necessary shopping lists. For example, the management department suggests cooking recipes based on the user's preferences. The management department creates necessary shopping lists based on the suggested recipes. The management department provides the shopping lists to the user and assists with shopping. For example, the management department analyzes the user's past cooking history and suggests the most suitable recipes. The management department creates necessary shopping lists based on the suggested recipes. The management department provides the shopping lists to the user and assists with shopping. This reduces the effort required for the user to cook by suggesting cooking recipes and creating shopping lists. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input the user's preferences and past cooking history into the AI ​​and have the AI ​​perform recipe suggestion and shopping list creation.

[0067] The integration unit can perform household chores in conjunction with smart home appliances. For example, the integration unit can work with a cleaning robot to clean a room. The integration unit manages the cleaning schedule and sends reminders to the user. The integration unit allows the user to control the cleaning robot by voice or text. For example, the integration unit can work with a cooking robot to cook. The integration unit manages the cooking schedule and sends reminders to the user. The integration unit allows the user to control the cooking robot by voice or text. For example, the integration unit can work with a washing machine to do laundry. The integration unit manages the laundry schedule and sends reminders to the user. The integration unit allows the user to control the washing machine by voice or text. This improves the efficiency of household chores by performing them in conjunction with smart home appliances. Some or all of the above processes in the integration unit may be performed using AI, for example, or without AI. For example, the integration unit can input the operation of smart home appliances into AI and have AI perform household chores.

[0068] The collaborative unit can estimate the user's emotions and adjust the priority of household chores based on the estimated emotions. For example, if the user is feeling stressed, the collaborative unit will prioritize cleaning to create a relaxing environment. If the user is tired, the collaborative unit will postpone more burdensome chores such as cooking and laundry. If the user is feeling energetic, the collaborative unit will prioritize smaller chores such as feeding pets and caring for plants. In this way, the user's burden can be reduced by adjusting the priority of chores according to their emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the collaborative unit may be performed using AI, for example, or not using AI. For example, the collaborative unit can input user emotion data into AI and have the AI ​​adjust the priority of household chores.

[0069] The collaborative unit can analyze the user's past household chore history when performing household chores and select the optimal method of execution. For example, the collaborative unit can analyze the frequency and method of cleaning performed by the user in the past and propose an optimal cleaning schedule. The collaborative unit can suggest new recipes based on recipes the user has made in the past. The collaborative unit can analyze the method of laundry performed by the user in the past and propose an optimal laundry method. In this way, the optimal method of execution of household chores can be selected by analyzing past household chore history. Some or all of the above processes in the collaborative unit may be performed using AI, for example, or without AI. For example, the collaborative unit can input the user's past household chore history data into AI and have the AI ​​select the optimal method of execution.

[0070] The collaborative unit can filter household chores based on the user's current lifestyle and areas of interest. For example, if the user is busy, the collaborative unit will prioritize suggesting chores that can be completed quickly. If the user is interested in health, the collaborative unit will suggest healthy cooking recipes. If the user is interested in pets, the collaborative unit will prioritize suggesting chores related to feeding and caring for pets. In this way, by filtering chores according to the user's lifestyle and areas of interest, more appropriate chores can be suggested. Some or all of the above processing in the collaborative unit may be performed using AI, for example, or not. For example, the collaborative unit can input data on the user's lifestyle and areas of interest into the AI ​​and have the AI ​​perform the chore filtering.

[0071] The integration unit can estimate the user's emotions and adjust the timing of household chores based on the estimated emotions. For example, if the user is relaxed, the integration unit will prioritize relaxation time by postponing household chores. If the user is in a hurry, the integration unit will adjust the timing of household chores to be performed quickly. If the user is tired, the integration unit will postpone household chores until the next day. In this way, the user's burden can be reduced by adjusting the timing of household chores according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the integration unit may be performed using AI, for example, or not using AI. For example, the integration unit can input user emotion data into AI and have the AI ​​adjust the timing of household chores.

[0072] The collaborative unit can prioritize the execution of highly relevant household chores by considering the user's geographical location information when performing household chores. For example, if the user is at home, the collaborative unit will prioritize chores such as cleaning and cooking. If the user is out, the collaborative unit will prioritize chores such as feeding pets and caring for plants. If the user is in a specific location, the collaborative unit will prioritize chores related to that location. This improves the efficiency of household chores by prioritizing them based on the user's geographical location information. Some or all of the above processing in the collaborative unit may be performed using AI, for example, or without AI. For example, the collaborative unit can input the user's geographical location information into AI and have the AI ​​perform highly relevant household chores.

[0073] The integration unit can analyze the user's social media activity while performing household chores and execute related chores. For example, if the user posts about cooking on social media, the integration unit will prioritize cooking chores. If the user posts about pets on social media, the integration unit will prioritize feeding and caring for pets. If the user posts about plants on social media, the integration unit will prioritize caring for plants. This allows the system to provide household chores tailored to the user's interests by performing chores based on the user's social media activity. Some or all of the above processing in the integration unit may be performed using AI, for example, or without AI. For example, the integration unit can input the user's social media activity data into AI and have the AI ​​perform related household chores.

[0074] The management unit can estimate the user's emotions and adjust how the household schedule is displayed based on the estimated emotions. For example, if the user is stressed, the management unit provides a simple and highly visible schedule display. If the user is relaxed, the management unit provides a schedule display with detailed information. If the user is in a hurry, the management unit provides a schedule display that gets straight to the point. By adjusting how the household schedule is displayed according to the user's emotions, it becomes easier for the user to understand the schedule. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the management unit may be performed using AI, for example, or not using AI. For example, the management unit can input user emotion data into AI and have the AI ​​adjust how the household schedule is displayed.

[0075] The management department can optimize the current schedule by referring to past schedule data when managing household chore schedules. For example, the management department can analyze the schedule of household chores that the user has performed in the past and propose an optimal schedule. The management department can optimize the current schedule by referring to the frequency of household chores that the user has performed in the past. The management department can optimize the current schedule by referring to the time of day that the user has performed household chores in the past. In this way, the current schedule can be optimized by referring to past schedule data. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input past schedule data into AI and have AI perform the optimization of the current schedule.

[0076] The management department can apply different management methods to each category of household chore when managing household chore schedules. For example, the management department can manage cleaning schedules with different frequencies for each room. The management department can manage cooking schedules taking into account the expiration dates of ingredients. The management department can manage laundry schedules according to the type of clothing and how often it is used. By applying different management methods to each category of household chore, the management of household chore schedules becomes more efficient. Some or all of the above processes in the management department may be performed using AI, for example, or not. For example, the management department can input data for each category of household chore into the AI ​​and have the AI ​​apply different management methods.

[0077] The management unit can estimate the user's emotions and determine the priority of household chores based on those estimated emotions. For example, if the user is stressed, the management unit will prioritize relaxing chores. If the user is tired, the management unit will prioritize less burdensome chores. If the user is energetic, the management unit will prioritize time-consuming chores. This reduces the user's burden by determining the priority of household chores according to their emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the management unit may be performed using AI, or not using AI. For example, the management unit can input user emotion data into an AI and have the AI ​​determine the priority of household chores.

[0078] The management department can analyze schedule changes based on the submission timing of household chores when managing household chore schedules. For example, if household chores are submitted early, the management department will execute them earlier. If household chores are submitted late, the management department will execute them later. The management department will adjust the schedule priorities according to the submission timing of household chores. In this way, schedule optimization can be achieved by analyzing schedule changes based on the submission timing of household chores. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input household chore submission timing data into AI and have the AI ​​perform the analysis of schedule changes.

[0079] The management department can analyze household chore schedules by referring to relevant market data. For example, the management department can propose an optimal schedule based on the relevant market data. The management department can adjust schedule priorities by referring to the relevant market data. The management department can analyze the relevant market data and propose changes to the schedule. In this way, the schedule can be optimized by referring to the relevant market data. Some or all of the above processes in the management department may be performed using AI, for example, or not using AI. For example, the management department can input the relevant market data for household chores into AI and have the AI ​​perform the schedule analysis.

[0080] The control unit can estimate the user's emotions and adjust the way it presents the controls based on those emotions. For example, if the user is nervous, the control unit provides a simple and easily visible control method. If the user is relaxed, it provides a control method that includes detailed information. If the user is in a hurry, it provides a control method that gets straight to the point. By adjusting the way it presents the controls according to the user's emotions, it makes the controls easier for the user to use. Emotion estimation is achieved using an emotion estimation function, for example, with an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the control unit may be performed using AI, or not using AI. For example, the control unit can input user emotion data into an AI and have the AI ​​adjust the way it presents the controls.

[0081] The control unit can adjust the level of detail of an operation based on its importance during operation. For example, the control unit provides detailed instructions for high-importance chores and simplified instructions for low-importance chores. The control unit adjusts how the instructions are displayed according to their importance. This makes it easier for the user to operate the system by adjusting the level of detail according to the importance of the chore. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input chore importance data into the AI ​​and have the AI ​​adjust the level of detail of the operations.

[0082] The control unit can apply different operation algorithms depending on the category of household chore during operation. For example, for cleaning, the control unit can apply a different algorithm for each room. For cooking, the control unit can apply a different algorithm for each recipe. For laundry, the control unit can apply a different algorithm for each type of clothing. This allows the control unit to apply the optimal operation algorithm according to the category of household chore. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input household chore category data into the AI ​​and have the AI ​​execute the application of different operation algorithms.

[0083] The control unit can estimate the user's emotions and adjust the length of the operation based on the estimated emotions. For example, if the user is in a hurry, the control unit shortens the length of the operation. If the user is relaxed, the control unit lengthens the length of the operation. If the user is tired, the control unit adjusts the length of the operation. This reduces the user's burden by adjusting the length of the operation according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input user emotion data into the AI ​​and have the AI ​​adjust the length of the operation.

[0084] The control unit can determine the priority of operations based on the submission timing of household chores during operation. For example, the control unit will give a higher priority to household chores that are submitted earlier, and a lower priority to household chores that are submitted later. The control unit adjusts the priority of operations according to the submission timing. This allows household chores to be performed efficiently by determining the priority of operations based on the submission timing. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input household chore submission timing data into AI and have the AI ​​perform the determination of operation priorities.

[0085] The control unit can adjust the order of operations based on the relevance of household chores during operation. For example, the control unit prioritizes the order of operations for highly relevant chores. For less relevant chores, the control unit postpones the order of operations. The control unit adjusts the order of operations according to the relevance of the chores. This allows for efficient performance of household chores by adjusting the order of operations based on their relevance. Some or all of the above processing in the control unit may be performed using AI, for example, or without AI. For example, the control unit can input data on the relevance of chores into the AI ​​and have the AI ​​perform the adjustment of the order of operations.

[0086] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0087] The integration unit not only automates household chores by linking with smart home appliances, but can also acquire user health data and suggest chores tailored to their health condition. For example, if the user's health data indicates accumulated fatigue, it will prioritize relaxing chores and postpone strenuous tasks such as cleaning and cooking. Furthermore, if the user's health data reveals a lack of exercise, it can suggest chores that incorporate light exercise. It can also support a healthy lifestyle by suggesting nutritionally balanced recipes based on the user's health data. In this way, by suggesting chores tailored to the user's health condition, it can support the maintenance of the user's health.

[0088] In addition to automating household chores, the integration system can also suggest chores based on the user's hobbies and preferences. For example, if a user enjoys gardening, it will prioritize suggesting plant care chores. If a user enjoys cooking, it can suggest new recipes and cooking methods to expand their enjoyment of cooking. Furthermore, if a user has pets, it can prioritize suggesting pet feeding and care chores to enrich their time with their pets. In this way, by suggesting chores that match the user's hobbies and preferences, they can enjoy doing household chores.

[0089] In addition to managing household chore schedules, the management department can also optimize schedules according to the user's lifestyle. For example, if a user is a night owl, household chores that can be done at night will be prioritized in the schedule. Similarly, if a user is an early riser, household chores that can be done in the morning will be prioritized in the schedule. Furthermore, the timing of household chores can be adjusted according to the user's lifestyle, and a manageable schedule can be proposed. By optimizing the schedule according to the user's lifestyle, the burden of household chores can be reduced.

[0090] The control unit can also provide guidance to users when operating smart home appliances. For example, if a user is operating a robotic vacuum cleaner for the first time, it can guide them through the operation procedure step by step. It can also guide users through the operation procedure based on recipes when operating a cooking robot. Furthermore, it can guide users through the optimal operation procedure according to the type of laundry when operating a washing machine. This allows users to operate smart home appliances without confusion, improving the efficiency of household chores.

[0091] In addition to managing household chore schedules, the management department can also monitor users' progress in real time and adjust schedules as needed. For example, if a user finishes a chore earlier than planned, the schedule can be adjusted to move the next chore forward. Conversely, if a user is behind schedule, the schedule can be adjusted to postpone the next chore. Furthermore, if a user forgets a chore, a reminder can be sent to encourage them to complete it. This improves the efficiency of household chores by adjusting schedules according to the user's progress.

[0092] The integration unit can estimate the user's emotions and adjust how household chores are performed based on those emotions. For example, if the user is feeling stressed, it can play relaxing music while cleaning. If the user is tired, it can adjust the chore to be completed in a shorter time. Furthermore, if the user is feeling energetic, it can incorporate elements to make the chore more enjoyable. In this way, the burden of household chores can be reduced by adjusting how chores are performed according to the user's emotions.

[0093] The management department can estimate the user's emotions and flexibly adjust the household chore schedule based on those emotions. For example, if a user has a sudden change of plans, the system automatically adjusts the chore schedule and postpones it. If the user wants to relax, the chore schedule is eased to allow for relaxation time. Furthermore, if the user is feeling energetic, the schedule can be packed with chores to complete them efficiently. In this way, the burden of household chores can be reduced by flexibly adjusting the schedule according to the user's emotions.

[0094] The control unit can estimate the user's emotions and customize the interface based on those emotions. For example, if the user is tense, it can provide a simple and intuitive interface. If the user is relaxed, it can provide an interface with more detailed information. Furthermore, if the user is in a hurry, it can provide an interface with simplified operating procedures. By customizing the control interface according to the user's emotions, the burden of operation can be reduced.

[0095] The management department can estimate the user's emotions and adjust the notification method for household chore schedules based on those emotions. For example, if the user is stressed, notifications will be sent with gentle sounds or lights. If the user is relaxed, notifications will be less frequent. Furthermore, if the user is in a hurry, important household chore notifications can be prioritized. This allows for smoother management of household chore schedules by adjusting notification methods according to the user's emotions.

[0096] The integration unit can estimate the user's emotions and adjust the timing of household chores based on the estimated emotions. For example, if the user is relaxed, it will prioritize relaxation time by postponing household chores. If the user is in a hurry, the integration unit will adjust the timing of household chores to be performed quickly. If the user is tired, the integration unit will postpone household chores until the next day. In this way, the user's burden can be reduced by adjusting the timing of household chores according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the integration unit may be performed using AI, for example, or not using AI. For example, the integration unit can input user emotion data into AI and have the AI ​​adjust the timing of household chores.

[0097] The following briefly describes the processing flow for example form 2.

[0098] Step 1: The integration unit uses an AI agent to automate household chores by coordinating with smart home appliances. For example, it will work with cleaning robots, cooking robots, washing machines, etc., to automate each of these tasks. Step 2: The management department manages the household chore schedules of smart home appliances linked by the integration department and sends reminders to users. For example, it manages the schedules for cleaning, cooking, and laundry and sends reminders for each chore. Step 3: The control unit allows users to operate smart home appliances via voice or text based on the household schedule managed by the management unit. For example, users can control robotic vacuums, cooking robots, and washing machines via voice or text.

[0099] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0100] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0101] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0102] Each of the multiple elements, including the aforementioned coordination unit, management unit, and operation unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the coordination unit is implemented by the control unit 46A of the smart device 14, which uses the smart device 14's camera 42 and microphone 38B to detect the status of household chores and manages the progress of the chores via the control unit 46A. The management unit is implemented by the specific processing unit 290 of the data processing unit 12, which manages the schedule of household chores and sends reminders to the user. The operation unit is implemented by the control unit 46A of the smart device 14, which allows the user to operate smart home appliances by voice or text. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0103] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0104] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0105] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0106] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0107] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0108] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0109] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0110] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0111] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0112] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0113] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0114] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0115] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0116] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0117] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0118] Each of the multiple elements, including the aforementioned coordination unit, management unit, and operation unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the coordination unit is implemented by the control unit 46A of the smart glasses 214, which detects the status of household chores using the smart glasses 214's camera 42 and microphone 238, and manages the progress of the household chores via the control unit 46A. The management unit is implemented by the specific processing unit 290 of the data processing unit 12, which manages the schedule of household chores and sends reminders to the user. The operation unit is implemented by the control unit 46A of the smart glasses 214, which allows the user to operate smart home appliances by voice or text. The correspondence between each unit and the device or control unit is not limited to the example described above, and various modifications are possible.

[0119] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0120] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0121] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0122] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0123] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0124] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0125] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0126] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0127] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0128] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0129] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0130] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0131] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0132] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0133] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0134] Each of the multiple elements, including the aforementioned coordination unit, management unit, and operation unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the coordination unit is implemented by the control unit 46A of the headset terminal 314, which uses the headset terminal 314's camera 42 and microphone 238 to detect the status of household chores and manages the progress of the chores via the control unit 46A. The management unit is implemented by the specific processing unit 290 of the data processing unit 12, which manages the schedule of household chores and sends reminders to the user. The operation unit is implemented by the control unit 46A of the headset terminal 314, which allows the user to operate smart home appliances by voice or text. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0135] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0136] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0137] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0138] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0139] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0140] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0141] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0142] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0143] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0144] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0145] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0146] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0147] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0148] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0149] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0150] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0151] Each of the multiple elements, including the aforementioned coordination unit, management unit, and operation unit, is implemented in at least one of the robot 414 and the data processing unit 12. For example, the coordination unit is implemented by the control unit 46A of the robot 414, which uses the robot 414's camera 42 and microphone 238 to detect the status of household chores and manages the progress of the chores via the control unit 46A. The management unit is implemented by the specific processing unit 290 of the data processing unit 12, which manages the schedule of household chores and sends reminders to the user. The operation unit is implemented by the control unit 46A of the robot 414, which allows the user to operate smart home appliances by voice or text. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0152] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0153] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0154] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0155] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0156] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0157] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0158] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0159] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0160] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0161] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0162] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0163] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0164] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0165] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0166] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0167] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0168] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0169] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0170] (Note 1) The AI ​​agent works in conjunction with smart home appliances to automate household chores, The aforementioned collaboration unit manages and reminds the household chore schedules of smart home appliances that are linked by the aforementioned collaboration unit, and The system includes an operating unit that allows the user to operate smart home appliances by voice or text based on a household chore schedule managed by the aforementioned management unit. A system characterized by the following features. (Note 2) The aforementioned linkage unit is, Automate everyday household chores such as cleaning, cooking, laundry, feeding pets, and plant care. The system described in Appendix 1, characterized by the features described herein. (Note 3) The aforementioned management department, Manage household chore schedules and send reminders to users. The system described in Appendix 1, characterized by the features described herein. (Note 4) The aforementioned operating unit is Users can control smart home appliances using their voice or text. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned management department, We provide cooking recipe suggestions and help create shopping lists. The system described in Appendix 1, characterized by the features described herein. (Note 6) The aforementioned linkage unit is, Perform household chores in conjunction with smart home appliances. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned linkage unit is, It estimates the user's emotions and adjusts the priority of household chores based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 8) The aforementioned linkage unit is, When performing household chores, the system analyzes the user's past chore history and selects the optimal method of execution. The system described in Appendix 1, characterized by the features described herein. (Note 9) The aforementioned linkage unit is, When performing household chores, filtering is performed based on the user's current lifestyle and areas of interest. The system described in Appendix 1, characterized by the features described herein. (Note 10) The aforementioned linkage unit is, It estimates the user's emotions and adjusts the timing of household chores based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 11) The aforementioned linkage unit is, When performing household chores, the system prioritizes the most relevant chores by considering the user's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 12) The aforementioned linkage unit is, When performing household chores, the system analyzes the user's social media activity and performs related chores. The system described in Appendix 1, characterized by the features described herein. (Note 13) The aforementioned management department, The system estimates the user's emotions and adjusts how the household chore schedule is displayed based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned management department, When managing household chore schedules, refer to past schedule data to optimize the current schedule. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned management department, When managing household chore schedules, apply different management methods to each category of chore. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned management department, It estimates the user's emotions and determines the priority of household chore schedules based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 17) The aforementioned management department, When managing household chore schedules, analyze schedule changes based on when chores are submitted. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned management department analyzes the schedule by referring to market data related to household chores when managing the household chore schedule. The system described in Appendix 1, characterized by the features described herein. (Note 19) The aforementioned operating unit estimates the user's emotions and adjusts the method of expressing the operation based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 20) The aforementioned control unit adjusts the level of detail of the operation based on the importance of the household chore during operation. The system described in Appendix 1, characterized by the features described herein. (Note 21) The aforementioned control unit applies different operating algorithms depending on the category of household chore during operation. The system described in Appendix 1, characterized by the features described herein. (Note 22) The aforementioned operating unit estimates the user's emotions and adjusts the length of the operation based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned control unit determines the priority of operations based on the timing of household chore submissions during operation. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned control unit adjusts the order of operations based on the relevance of household chores during operation. The system described in Appendix 1, characterized by the features described herein. [Explanation of symbols]

[0171] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. The AI ​​agent works in conjunction with smart home appliances to automate household chores, The aforementioned collaboration unit manages and reminds the household chore schedules of smart home appliances that are linked by the aforementioned collaboration unit, and The system includes an operating unit that allows the user to operate smart home appliances by voice or text based on a household chore schedule managed by the aforementioned management unit. A system characterized by the following features.

2. The aforementioned linkage unit is, Automate everyday household chores such as cleaning, cooking, laundry, feeding pets, and plant care. The system according to feature 1.

3. The aforementioned management department, Manage household chore schedules and send reminders to users. The system according to feature 1.

4. The aforementioned operating unit is Users can control smart home appliances using their voice or text. The system according to feature 1.

5. The aforementioned management department, We provide cooking recipe suggestions and help create shopping lists. The system according to feature 1.

6. The aforementioned linkage unit is, Perform household chores in conjunction with smart home appliances. The system according to feature 1.

7. The aforementioned linkage unit is, It estimates the user's emotions and adjusts the priority of household chores based on those estimated emotions. The system according to feature 1.

8. The aforementioned linkage unit is, When performing household chores, the system analyzes the user's past chore history and selects the optimal method of execution. The system according to feature 1.

Images