System

Abstract

An object of a system according to an embodiment is to intuitively operate a cleaning robot through communication with a user and grasp a state of the cleaning robot.SOLUTION: A system according to an embodiment includes a cleaning robot, a communication unit, a control unit, and a state tracking unit. The communication unit performs communication with a user. The control unit controls the cleaning robot based on the instruction of the user received by the communication unit. The state tracking unit monitors a state of the cleaning robot.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies have had the problem that it is not intuitive to operate a cleaning robot or understand its status, making it difficult for users to use.

[0005] The system according to the embodiment aims to enable a user to intuitively operate a cleaning robot and understand its status through communication with the user. [Means for solving the problem]

[0006] A system according to an embodiment includes a cleaning robot, a communication unit, a control unit, and a status tracking unit. The communication unit communicates with a user. The control unit controls the cleaning robot based on user instructions received by the communication unit. The status tracking unit monitors the status of the cleaning robot. [Effects of the Invention]

[0007] The system according to the embodiment allows the user to intuitively operate the cleaning robot and understand its status through communication with the user. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The cleaning robot system according to an embodiment of the present invention is a system that is equipped with a generative AI, communicates with a user, controls the cleaning robot, and tracks its status. This enables the cleaning robot system to control the cleaning robot based on user instructions and monitor its status, thereby enabling efficient cleaning.

[0029] A cleaning robot system according to an embodiment includes a generation AI, a communication unit, a control unit, and a state tracking unit. The generation AI understands user instructions. For example, if the user instructs the generation AI to "clean the living room," the generation AI understands the instruction and instructs the cleaning robot to clean the living room. Furthermore, if the user asks, "What's the remaining battery life of the cleaning robot?", the generation AI can check the status of the cleaning robot and inform the user of the remaining battery life. The communication unit communicates with the user via the generation AI. For example, the communication unit receives the user's voice instructions and transmits them to the generation AI. Furthermore, the communication unit can communicate responses generated by the generation AI to the user. The control unit controls the cleaning robot based on the user's instructions received by the communication unit. For example, if the user instructs the control unit to "clean the kitchen," the control unit instructs the cleaning robot to clean the kitchen. Furthermore, if the user instructs the control unit to "clean under the sofa," the control unit can instruct the cleaning robot to clean under the sofa. The state tracking unit monitors the status of the cleaning robot. For example, the status tracking unit sends a notification to the user through the generation AI when the cleaning robot has completed cleaning or when the battery is low. The status tracking unit can also monitor the location information and operating status of the cleaning robot in real time and report them to the user as needed. As a result, the cleaning robot system according to the embodiment controls the cleaning robot based on user instructions and monitors its status, enabling efficient cleaning. For example, the user can control the cleaning robot using a smartphone while away from home and keep track of the status of the cleaning robot. Furthermore, the advice and information provided by the generation AI can help maximize the use of the cleaning robot.

[0030] The communication unit can learn the user's past instruction history and make suggestions based on the user's preferences and habits. For example, the generation AI in the communication unit learns the user's past instruction history and prioritizes suggestions for areas that the user frequently instructs to be cleaned. For example, if the user frequently instructs the AI ​​to "clean the kitchen," the AI ​​may suggest, "Shall I clean the kitchen today?" The communication unit also learns the user's cleaning schedule and automatically makes suggestions for the time periods when the user frequently instructs the AI ​​to clean. For example, if the user instructs the AI ​​to "clean the living room before going to bed" every night, the AI ​​may suggest, "Is it about time to clean the living room?" The communication unit also learns the user's cleaning frequency and suggests areas that the user has not instructed to be cleaned for a certain period of time. For example, if the user has not instructed the AI ​​to "clean the bedroom" for more than a week, the AI ​​may suggest, "How about cleaning the bedroom?" This allows the AI ​​to provide more personalized services by making suggestions based on the user's preferences and habits.

[0031] The communication unit allows the generation AI to communicate with multiple users simultaneously and process each instruction appropriately. For example, the communication unit allows the generation AI to simultaneously receive instructions from multiple users and process each instruction appropriately. For example, if user A says, "Clean the living room," and user B says, "Clean the kitchen," the generation AI will respond, "I'll clean the living room and then the kitchen, in that order." The communication unit also allows the generation AI to process instructions from multiple users based on priority. For example, if user A says, "Clean the living room now," and user B says, "Clean the kitchen later," the generation AI will respond, "I'll clean the living room first, and then the kitchen." The communication unit also allows the generation AI to process instructions from multiple users simultaneously and assign multiple tasks to the cleaning robot. For example, if user A says, "Clean the living room," and user B says, "Clean the kitchen," the generation AI will respond, "I'll clean the living room and the kitchen at the same time." This allows the generation AI to process instructions from multiple users simultaneously, enabling more efficient operation.

[0032] The control unit can acquire the user's location information and suggest an optimal cleaning route. For example, the control unit allows the generation AI to acquire the location information of the user's smartphone and suggest an optimal cleaning route before the user returns home. For example, the control unit may suggest, "Shall I clean the living room and kitchen?" before the user returns home. The control unit also allows the generation AI to suggest, based on the user's location information, that the user start cleaning when they approach their home. For example, when the user approaches their home, the control unit may suggest, "Shall I clean the living room now?". The control unit also allows the generation AI to suggest, based on the user's location information, that the user start cleaning when they leave their home. For example, when the user leaves their home, the control unit may suggest, "Shall I clean the kitchen now?". This enables efficient cleaning by suggesting an optimal cleaning route based on the user's location information.

[0033] The control unit can learn the user's schedule and suggest the optimal cleaning time. For example, the generation AI in the control unit works with the user's calendar app to learn the user's schedule and suggest the optimal cleaning time. For example, the control unit may suggest, "Shall we clean the living room now?" when the user is in a meeting. The control unit also learns the user's schedule and suggests cleaning when the user is not at home. For example, the control unit may suggest, "Shall we clean the kitchen now?" when the user is out. The control unit also learns the user's schedule and suggests cleaning when the user is relaxing. For example, the control unit may suggest, "Shall we clean the bedroom now?" when the user is watching a movie. This enables efficient cleaning by suggesting the optimal cleaning time based on the user's schedule.

[0034] The control unit can work with the user's smart home devices to clean in coordination with other home appliances. For example, the control unit's generation AI works with smart home devices to coordinate other home appliances when the cleaning robot starts cleaning. For example, the control unit turns off the smart lights when the cleaning robot starts cleaning the living room. The control unit's generation AI also works with smart home devices to play music on a smart speaker when the cleaning robot starts cleaning. For example, the control unit plays relaxing music when the cleaning robot cleans the kitchen. The control unit's generation AI also works with smart home devices to adjust the temperature with a smart thermostat when the cleaning robot starts cleaning. For example, the control unit sets a comfortable temperature when the cleaning robot cleans the bedroom. This enables more efficient cleaning by working with smart home devices.

[0035] The control unit can accept instructions from the user not only through voice commands, but also through text messages and gestures. For example, if a user sends a text message from their smartphone saying, "Clean the living room," the generation AI understands the instruction and instructs the cleaning robot to clean the living room. The control unit also allows the generation AI to accept instructions not only through voice commands, but also through gestures. For example, if a user points at an area they want cleaned, the generation AI recognizes the gesture and instructs the cleaning robot to clean that area. The control unit also creates a system where the generation AI can accept instructions through voice commands, text messages, and gestures. For example, if a user says, "Clean the kitchen," sends a text saying, "Clean the bedroom," or gestures, pointing at the living room, the generation AI understands each instruction and gives the cleaning robot the appropriate instruction. This allows for a wider variety of operations by accepting instructions not only through voice commands, but also through text messages and gestures.

[0036] The control unit can learn the layout of the user's home and automatically generate an optimal cleaning route. For example, the control unit has a generation AI learn the layout of the user's home and automatically generate an optimal cleaning route. For example, it proposes a route that efficiently cleans from the living room to the kitchen and bedroom. The control unit also has a generation AI learn the layout of the user's home and automatically generate an optimal cleaning route that takes into account the arrangement of furniture. For example, it proposes a route that efficiently cleans under sofas and tables. The control unit also has a generation AI learn the layout of the user's home and automatically generate an optimal cleaning route based on the user's cleaning priorities. For example, it proposes a route that cleans the living room first, followed by the kitchen and bedrooms. This enables efficient cleaning by automatically generating an optimal cleaning route based on the layout of the home.

[0037] The control unit can adjust the cleaning frequency of a specific area based on a user's instructions. The control unit, for example, causes the generation AI to adjust the cleaning frequency of a specific area based on a user's instructions. For example, if a user instructs the generation AI to "clean the living room every day," the generation AI sets the cleaning frequency of the living room to every day. The control unit also causes the generation AI to adjust the cleaning frequency of a specific area based on a user's instructions. For example, if a user instructs the generation AI to "clean the kitchen once a week," the generation AI sets the cleaning frequency of the kitchen to once a week. The control unit also causes the generation AI to adjust the cleaning frequency of a specific area based on a user's instructions. For example, if a user instructs the generation AI to "clean the bedroom once a month," the generation AI sets the cleaning frequency of the bedroom to once a month. This allows for efficient cleaning by adjusting the cleaning frequency based on the user's instructions.

[0038] The control unit can detect the movements of the user's pet and clean specific areas while the pet is away. For example, the control unit uses a camera to detect the movements of the user's pet through the generation AI, and cleans the living room while the pet is away. For example, when the pet moves to another room, the control unit suggests, "I'll clean the living room." The control unit can also use a camera to detect the movements of the user's pet and clean the kitchen while the pet is away. For example, the control unit suggests, "I'll clean the kitchen" while the pet is in the living room. The control unit can also use a camera to detect the movements of the user's pet and clean the bedroom while the pet is away. For example, the control unit suggests, "I'll clean the bedroom" while the pet is out. This enables efficient cleaning by cleaning specific areas while the pet is away.

[0039] The control unit learns the layout of the user's furniture and is able to efficiently clean under the furniture and in the gaps between the furniture. For example, the control unit allows the generation AI to learn the layout of the user's furniture and propose a route that efficiently cleans under the sofa and in the gaps between the furniture. For example, it automatically generates a route that focuses on cleaning under the sofa. The control unit also allows the generation AI to learn the layout of the user's furniture and propose a route that efficiently cleans under the table and in the gaps between the furniture. For example, it automatically generates a route that focuses on cleaning under the table. The control unit also allows the generation AI to learn the layout of the user's furniture and propose a route that efficiently cleans under the bed and in the gaps between the furniture. For example, it automatically generates a route that focuses on cleaning under the bed. This enables more thorough cleaning by efficiently cleaning under the furniture and in the gaps between the furniture.

[0040] The state tracking unit can analyze the sensor data of the cleaning robot in real time and notify the user if an abnormality is detected. For example, the generation AI analyzes the sensor data of the cleaning robot in real time and notifies the user if an abnormality is detected. For example, if the cleaning robot hits an obstacle, the state tracking unit notifies the user, saying, "The cleaning robot has hit an obstacle." The generation AI also analyzes the sensor data of the cleaning robot in real time and notifies the user if an abnormality is detected. For example, if the battery of the cleaning robot is low, the state tracking unit notifies the user, saying, "The battery is low." The generation AI also analyzes the sensor data of the cleaning robot in real time and notifies the user if an abnormality is detected. For example, if the brush of the cleaning robot is clogged, the state tracking unit notifies the user, saying, "The brush is clogged." This allows for a quick response by notifying the user if an abnormality is detected.

[0041] The status tracking unit can automatically generate a maintenance schedule for the cleaning robot and propose it to the user. For example, the generation AI in the status tracking unit analyzes the usage status of the cleaning robot, automatically generates an optimal maintenance schedule, and proposes it to the user. For example, it notifies the user that "the next maintenance is one week later." The status tracking unit also analyzes the sensor data of the cleaning robot and automatically detects when maintenance is necessary and proposes it to the user. For example, it notifies the user that "it's time to replace the brush." ​​The status tracking unit also learns the usage history of the cleaning robot, automatically generates an optimal maintenance schedule, and proposes it to the user. For example, it notifies the user that "it's time to clean the filter." This automatically generated maintenance schedule enables efficient maintenance.

[0042] The status tracking unit can predict when to replace the cleaning robot's consumables (such as filters and brushes) and notify the user. For example, the generation AI in the status tracking unit analyzes the cleaning robot's usage data, predicts when to replace the filter, and notifies the user. For example, it notifies the user that "It's almost time to replace the filter." The status tracking unit also analyzes the cleaning robot's sensor data, predicts when to replace the brush, and notifies the user. For example, it notifies the user that "It's almost time to replace the brush." ​​The status tracking unit also learns the cleaning robot's usage history, predicts when to replace consumables, and notifies the user. For example, it notifies the user that "Please replace the filter and brush at the next maintenance." This enables efficient maintenance by predicting when to replace consumables.

[0043] The state tracking unit can analyze the cleaning robot's past operation history and suggest the optimal operation pattern. For example, the state tracking unit allows the generation AI to analyze the cleaning robot's past operation history and suggest the optimal operation pattern. For example, the state tracking unit notifies the user, "Based on past data, we suggest a new pattern to make cleaning the living room more efficient." The state tracking unit also allows the generation AI to analyze the cleaning robot's operation history and suggest the optimal operation pattern to improve cleaning efficiency. For example, the state tracking unit suggests, "Would you like to try a new operation pattern to make cleaning the kitchen more efficient?" The state tracking unit also allows the generation AI to analyze the cleaning robot's past operation history and suggest the optimal operation pattern based on the user's cleaning habits. For example, the state tracking unit notifies the user, "We suggest a new operation pattern to make cleaning the bedroom more efficient." This enables efficient cleaning by analyzing past operation history.

[0044] The generative AI can learn the user's cleaning habits and suggest efficient cleaning methods. For example, the generative AI can learn the user's cleaning habits and suggest efficient cleaning methods. For example, it might suggest, "Why not try rearranging the furniture to make cleaning the living room more efficient?" The generative AI can also learn the user's cleaning habits and suggest efficient cleaning methods. For example, it might suggest, "Why not try changing the order of cleaning to make cleaning the kitchen more efficient?" The generative AI can also learn the user's cleaning habits and suggest efficient cleaning methods. For example, it might suggest, "Why not try adjusting the frequency of cleaning to make cleaning the bedroom more efficient?" In this way, the generative AI can learn the user's cleaning habits and suggest efficient cleaning methods.

[0045] The generative AI can suggest appropriate cleaning methods based on the user's home environment (e.g., whether or not they have pets or allergies). The generative AI can, for example, learn the user's home environment and suggest appropriate cleaning methods based on whether or not they have pets. For example, it might suggest, "Why not try using a specific brush to efficiently clean up pet hair?" The generative AI can also learn the user's home environment and suggest appropriate cleaning methods based on allergies. For example, it might suggest, "Why not try using a HEPA filter to reduce allergies?" The generative AI can also learn the user's home environment and suggest appropriate cleaning methods based on whether or not they have pets or allergies. For example, it might suggest, "Why not try using a specific cleaning mode to efficiently clean up pet hair and allergens?" This enables efficient cleaning by suggesting appropriate cleaning methods based on the home environment.

[0046] The generation AI can analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, the generation AI can analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, it can suggest, "It's almost time to replace the brush. Here's how to replace it." The generation AI can also analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, it can suggest, "It's almost time to clean the filter. Here's how to clean it." The generation AI can also analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, it can suggest, "It's almost time to replace the battery. Here's how to replace it." In this way, the generation AI can suggest the optimal maintenance method by analyzing the usage history.

[0047] The generating AI can incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, the generating AI can incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, it can propose, "Everyone in the family wants to clean the living room. Clean the living room as a priority." The generating AI can also incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, it can propose, "Everyone in the family wants to clean the kitchen. Clean the kitchen as a priority." The generating AI can also incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, it can propose, "Everyone in the family wants to clean the bedroom. Clean the bedroom as a priority." In this way, the generating AI can incorporate the opinions of all members of the family and propose a cleaning method that will satisfy everyone.

[0048] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0049] The control unit can monitor the user's health condition and suggest an appropriate cleaning mode. For example, if the user has allergies, the generating AI will suggest, "Would you like to clean in allergy prevention mode?" If the user has a cold, the generating AI will suggest, "Would you like to clean in cold prevention mode?" Furthermore, if the user feels they are not getting enough exercise, the generating AI will suggest, "Would you like to help clean while exercising?" This can support a healthier living environment by providing cleaning modes that suit the user's health condition.

[0050] The control unit can detect the movements of the user's pet and clean specific areas while the pet is away. For example, the generation AI can use a camera to detect the movements of the user's pet and clean the living room while the pet is away. For example, when the pet moves to another room, it can suggest, "I'll clean the living room." The generation AI can also use a camera to detect the movements of the user's pet and clean the kitchen while the pet is away. For example, it can suggest, "I'll clean the kitchen" while the pet is in the living room. The generation AI can also use a camera to detect the movements of the user's pet and clean the bedroom while the pet is away. For example, it can suggest, "I'll clean the bedroom" while the pet is out. This allows for efficient cleaning by cleaning specific areas while the pet is away.

[0051] The control unit can learn the user's schedule and suggest the optimal cleaning time. For example, the generation AI can work with the user's calendar app to learn the user's schedule and suggest the optimal cleaning time. For example, if the user is in a meeting, it can suggest, "Shall we clean the living room now?". The generation AI can also learn the user's schedule and suggest cleaning times when the user is not at home. For example, it can suggest, "Shall we clean the kitchen now?" when the user is out. The generation AI can also learn the user's schedule and suggest cleaning times when the user is relaxing. For example, it can suggest, "Shall we clean the bedroom now?" when the user is watching a movie. This allows for efficient cleaning by suggesting the optimal cleaning time based on the user's schedule.

[0052] The control unit learns the layout of the user's furniture and is able to efficiently clean under furniture and in gaps. For example, the generation AI learns the layout of the user's furniture and suggests a route that efficiently cleans under sofas and in gaps. For example, it automatically generates a route that focuses on cleaning under sofas. The generation AI also learns the layout of the user's furniture and suggests a route that efficiently cleans under tables and in gaps. For example, it automatically generates a route that focuses on cleaning under tables. The generation AI also learns the layout of the user's furniture and suggests a route that efficiently cleans under beds and in gaps. For example, it automatically generates a route that focuses on cleaning under beds. This enables more thorough cleaning by efficiently cleaning under furniture and in gaps.

[0053] The control unit can learn the layout of the user's home and automatically generate the optimal cleaning route. For example, the generation AI learns the layout of the user's home and automatically generates the optimal cleaning route. For example, it proposes a route that efficiently cleans from the living room to the kitchen and bedroom. The generation AI also learns the layout of the user's home and automatically generates the optimal cleaning route that takes into account the arrangement of furniture. For example, it proposes a route that efficiently cleans under sofas and tables. Furthermore, the generation AI learns the layout of the user's home and automatically generates the optimal cleaning route based on the user's cleaning priorities. For example, it proposes a route that cleans the living room first, followed by the kitchen and bedroom. This enables efficient cleaning by automatically generating the optimal cleaning route based on the layout of the home.

[0054] The control unit can adjust the cleaning frequency of specific areas based on user instructions. For example, the generation AI adjusts the cleaning frequency of specific areas based on user instructions. For example, if the user instructs, "Clean the living room every day," the generation AI sets the cleaning frequency of the living room to every day. Also, if the user instructs, "Clean the kitchen once a week," the generation AI sets the cleaning frequency of the kitchen to once a week. Furthermore, if the user instructs, "Clean the bedroom once a month," the generation AI sets the cleaning frequency of the bedroom to once a month. This allows for efficient cleaning by adjusting the cleaning frequency based on user instructions.

[0055] The processing flow of the first embodiment will be briefly explained below.

[0056] Step 1: The communication unit communicates with the user through the generation AI. For example, it receives the user's voice instructions and conveys them to the generation AI. It can also convey the responses generated by the generation AI to the user. Step 2: The control unit controls the cleaning robot based on the user's instructions received by the communication unit. For example, if the user instructs the cleaning robot to "clean the kitchen," the control unit instructs the cleaning robot to clean the kitchen. Also, if the user instructs the cleaning robot to "clean under the sofa," the control unit instructs the cleaning robot to clean under the sofa. Step 3: The status tracking unit monitors the status of the cleaning robot. For example, when the cleaning robot completes cleaning or when the battery is low, it sends a notification to the user through the generation AI. It can also monitor the location and operating status of the cleaning robot in real time and report it to the user as needed.

[0057] (Example 2) The cleaning robot system according to an embodiment of the present invention is a system that is equipped with a generative AI, communicates with a user, controls the cleaning robot, and tracks its status. This enables the cleaning robot system to control the cleaning robot based on user instructions and monitor its status, thereby enabling efficient cleaning.

[0058] A cleaning robot system according to an embodiment includes a generation AI, a communication unit, a control unit, and a state tracking unit. The generation AI understands user instructions. For example, if the user instructs the generation AI to "clean the living room," the generation AI understands the instruction and instructs the cleaning robot to clean the living room. Furthermore, if the user asks, "What's the remaining battery life of the cleaning robot?", the generation AI can check the status of the cleaning robot and inform the user of the remaining battery life. The communication unit communicates with the user via the generation AI. For example, the communication unit receives the user's voice instructions and transmits them to the generation AI. Furthermore, the communication unit can communicate responses generated by the generation AI to the user. The control unit controls the cleaning robot based on the user's instructions received by the communication unit. For example, if the user instructs the control unit to "clean the kitchen," the control unit instructs the cleaning robot to clean the kitchen. Furthermore, if the user instructs the control unit to "clean under the sofa," the control unit can instruct the cleaning robot to clean under the sofa. The state tracking unit monitors the status of the cleaning robot. For example, the status tracking unit sends a notification to the user through the generation AI when the cleaning robot has completed cleaning or when the battery is low. The status tracking unit can also monitor the location information and operating status of the cleaning robot in real time and report them to the user as needed. As a result, the cleaning robot system according to the embodiment controls the cleaning robot based on user instructions and monitors its status, enabling efficient cleaning. For example, the user can control the cleaning robot using a smartphone while away from home and keep track of the status of the cleaning robot. Furthermore, the advice and information provided by the generation AI can help maximize the use of the cleaning robot.

[0059] The communication unit can analyze the user's tone of voice and phrasing to generate responses appropriate to the user's emotional state. For example, the generation AI analyzes the user's tone of voice and generates a calming response if the user is angry. For example, if the user says angrily, "Why haven't you finished cleaning yet?", the generation AI generates a response such as, "Sorry, I'll check it right away." The communication unit also analyzes the user's phrasing to generate a casual response if the user is relaxed. For example, if the user says in a relaxed tone, "Can you clean the living room for me?", the generation AI generates a response such as, "Sure, I'll do it right away." The communication unit also analyzes the user's tone of voice and phrasing comprehensively to generate a response including encouraging words if the user is feeling stressed. For example, if the user says, "I'm tired today, so please clean up," the generation AI generates a response such as, "Thank you for your hard work. Leave the cleaning of the living room to me." This enables more natural communication by generating responses appropriate to the user's emotional state.

[0060] The communication unit can learn the user's past instruction history and make suggestions based on the user's preferences and habits. For example, the generation AI in the communication unit learns the user's past instruction history and prioritizes suggestions for areas that the user frequently instructs to be cleaned. For example, if the user frequently instructs the AI ​​to "clean the kitchen," the AI ​​may suggest, "Shall I clean the kitchen today?" The communication unit also learns the user's cleaning schedule and automatically makes suggestions for the time periods when the user frequently instructs the AI ​​to clean. For example, if the user instructs the AI ​​to "clean the living room before going to bed" every night, the AI ​​may suggest, "Is it about time to clean the living room?" The communication unit also learns the user's cleaning frequency and suggests areas that the user has not instructed to be cleaned for a certain period of time. For example, if the user has not instructed the AI ​​to "clean the bedroom" for more than a week, the AI ​​may suggest, "How about cleaning the bedroom?" This allows the AI ​​to provide more personalized services by making suggestions based on the user's preferences and habits.

[0061] The communication unit can use the emotion estimation function to generate a relaxing response when the user is feeling stressed. For example, the generation AI analyzes the user's tone of voice and vocabulary and generates a relaxing response when the user is feeling stressed. For example, if the user says, "I'm really tired today," the generation AI generates a response such as, "Thank you for your hard work. Leave it to me to clean the living room." The communication unit also analyzes the user's emotional state and suggests relaxing music when the user is feeling stressed. For example, if the user says, "I'm irritable today," the generation AI suggests, "Shall I play some relaxing music?" The communication unit also analyzes the user's emotional state and suggests a relaxing activity when the user is feeling stressed. For example, if the user says, "I don't want to do anything today," the generation AI suggests, "Would you like to watch a relaxing movie?" This allows the generation AI to generate a relaxing response when the user is feeling stressed, thereby reducing the user's psychological burden.

[0062] The communication unit can recognize the user's gestures and facial expressions using a camera and generate responses based on them. For example, the generation AI uses a camera to recognize the user's gestures and instructs the AI ​​to clean the area the user points to. For example, if the user points to a corner of the living room, the generation AI responds, "I'll clean that area." The communication unit also uses a camera to recognize the user's facial expressions and generates a casual response if the user is smiling. For example, if the user says, "Please clean," with a smile, the generation AI responds, "Of course, I'll do it right away." The communication unit also uses a camera to comprehensively recognize the user's gestures and facial expressions and generates a response that includes encouraging words if the user appears tired. For example, if the user says, "Please clean," with a tired expression, the generation AI responds, "Thank you for your hard work. Leave it to me to clean the living room." This enables more intuitive operation by recognizing the user's gestures and facial expressions.

[0063] The communication unit allows the generation AI to communicate with multiple users simultaneously and process each instruction appropriately. For example, the communication unit allows the generation AI to simultaneously receive instructions from multiple users and process each instruction appropriately. For example, if user A says, "Clean the living room," and user B says, "Clean the kitchen," the generation AI will respond, "I'll clean the living room and then the kitchen, in that order." The communication unit also allows the generation AI to process instructions from multiple users based on priority. For example, if user A says, "Clean the living room now," and user B says, "Clean the kitchen later," the generation AI will respond, "I'll clean the living room first, and then the kitchen." The communication unit also allows the generation AI to process instructions from multiple users simultaneously and assign multiple tasks to the cleaning robot. For example, if user A says, "Clean the living room," and user B says, "Clean the kitchen," the generation AI will respond, "I'll clean the living room and the kitchen at the same time." This allows the generation AI to process instructions from multiple users simultaneously, enabling more efficient operation.

[0064] The communication unit uses the emotion estimation function to enable the cleaning robot to offer jokes and light-hearted conversation when the user is enjoying themselves. For example, the generation AI analyzes the user's emotional state and offers jokes when the user is enjoying themselves. For example, if the user says "Please clean up" with a smile, the generation AI might joke, "Of course, but maybe the cleaning robot needs a break too." The communication unit also analyzes the user's emotional state and offers light-hearted conversation when the user is enjoying themselves. For example, if the user says "Please clean up" with a happy tone, the generation AI might suggest, "Shall we talk about the latest movie while we clean the living room?" The communication unit also analyzes the user's emotional state and, when the user is enjoying themselves, humorously reports the cleaning robot's progress. For example, if the user says "Please clean up" with a happy tone, the generation AI might report, "The cleaning of the living room is going well. It looks like you're dancing." This allows the user to enjoy the experience by offering jokes and light-hearted conversation when they are enjoying themselves.

[0065] The control unit can acquire the user's location information and suggest an optimal cleaning route. For example, the control unit allows the generation AI to acquire the location information of the user's smartphone and suggest an optimal cleaning route before the user returns home. For example, the control unit may suggest, "Shall I clean the living room and kitchen?" before the user returns home. The control unit also allows the generation AI to suggest, based on the user's location information, that the user start cleaning when they approach their home. For example, when the user approaches their home, the control unit may suggest, "Shall I clean the living room now?". The control unit also allows the generation AI to suggest, based on the user's location information, that the user start cleaning when they leave their home. For example, when the user leaves their home, the control unit may suggest, "Shall I clean the kitchen now?". This enables efficient cleaning by suggesting an optimal cleaning route based on the user's location information.

[0066] The control unit can learn the user's schedule and suggest the optimal cleaning time. For example, the generation AI in the control unit works with the user's calendar app to learn the user's schedule and suggest the optimal cleaning time. For example, the control unit may suggest, "Shall we clean the living room now?" when the user is in a meeting. The control unit also learns the user's schedule and suggests cleaning when the user is not at home. For example, the control unit may suggest, "Shall we clean the kitchen now?" when the user is out. The control unit also learns the user's schedule and suggests cleaning when the user is relaxing. For example, the control unit may suggest, "Shall we clean the bedroom now?" when the user is watching a movie. This enables efficient cleaning by suggesting the optimal cleaning time based on the user's schedule.

[0067] The control unit can use the emotion estimation function to suggest a quick cleaning mode when the user is in a hurry. For example, the control unit has the generation AI analyze the user's tone of voice and language and suggest a quick cleaning mode when the user is in a hurry. For example, if the user says, "Clean quickly," the generation AI suggests, "Clean the living room in quick mode." The control unit also has the generation AI analyze the user's emotional state and suggest increasing the speed of the cleaning robot when the user is in a hurry. For example, if the user says, "Clean quickly," the generation AI suggests, "Increase the speed of the cleaning robot to clean the living room." The control unit also has the generation AI analyze the user's emotional state and suggest quickly cleaning only important areas when the user is in a hurry. For example, if the user says, "Clean the living room quickly," the generation AI suggests, "Quickly clean only important areas of the living room." This enables efficient cleaning by suggesting a quick cleaning mode when the user is in a hurry.

[0068] The control unit can work with the user's smart home devices to clean in coordination with other home appliances. For example, the control unit's generation AI works with smart home devices to coordinate other home appliances when the cleaning robot starts cleaning. For example, the control unit turns off the smart lights when the cleaning robot starts cleaning the living room. The control unit's generation AI also works with smart home devices to play music on a smart speaker when the cleaning robot starts cleaning. For example, the control unit plays relaxing music when the cleaning robot cleans the kitchen. The control unit's generation AI also works with smart home devices to adjust the temperature with a smart thermostat when the cleaning robot starts cleaning. For example, the control unit sets a comfortable temperature when the cleaning robot cleans the bedroom. This enables more efficient cleaning by working with smart home devices.

[0069] The control unit can accept instructions from the user not only through voice commands, but also through text messages and gestures. For example, if a user sends a text message from their smartphone saying, "Clean the living room," the generation AI understands the instruction and instructs the cleaning robot to clean the living room. The control unit also allows the generation AI to accept instructions not only through voice commands, but also through gestures. For example, if a user points at an area they want cleaned, the generation AI recognizes the gesture and instructs the cleaning robot to clean that area. The control unit also creates a system where the generation AI can accept instructions through voice commands, text messages, and gestures. For example, if a user says, "Clean the kitchen," sends a text saying, "Clean the bedroom," or gestures, pointing at the living room, the generation AI understands each instruction and gives the cleaning robot the appropriate instruction. This allows for a wider variety of operations by accepting instructions not only through voice commands, but also through text messages and gestures.

[0070] The control unit can use the emotion estimation function to cause the cleaning robot to operate in quiet mode when the user is relaxed. For example, the control unit causes the generation AI to analyze the user's emotional state and operate the cleaning robot in quiet mode when the user is relaxed. For example, if it is determined that the user is relaxed, the generation AI suggests, "Clean the living room in quiet mode." The control unit also causes the generation AI to analyze the user's emotional state and set the cleaning robot to minimize its operating noise when the user is relaxed. For example, if it is determined that the user is relaxed, the generation AI suggests, "Clean the kitchen in quiet mode." The control unit also causes the generation AI to analyze the user's emotional state and adjust the operating speed of the cleaning robot to operate in quiet mode when the user is relaxed. For example, if it is determined that the user is relaxed, the generation AI suggests, "Clean the bedroom in quiet mode." This allows the user to operate in quiet mode when relaxed, providing a comfortable environment.

[0071] The control unit can learn the layout of the user's home and automatically generate an optimal cleaning route. For example, the control unit has a generation AI learn the layout of the user's home and automatically generate an optimal cleaning route. For example, it proposes a route that efficiently cleans from the living room to the kitchen and bedroom. The control unit also has a generation AI learn the layout of the user's home and automatically generate an optimal cleaning route that takes into account the arrangement of furniture. For example, it proposes a route that efficiently cleans under sofas and tables. The control unit also has a generation AI learn the layout of the user's home and automatically generate an optimal cleaning route based on the user's cleaning priorities. For example, it proposes a route that cleans the living room first, followed by the kitchen and bedrooms. This enables efficient cleaning by automatically generating an optimal cleaning route based on the layout of the home.

[0072] The control unit can adjust the cleaning frequency of a specific area based on a user's instructions. The control unit, for example, causes the generation AI to adjust the cleaning frequency of a specific area based on a user's instructions. For example, if a user instructs the generation AI to "clean the living room every day," the generation AI sets the cleaning frequency of the living room to every day. The control unit also causes the generation AI to adjust the cleaning frequency of a specific area based on a user's instructions. For example, if a user instructs the generation AI to "clean the kitchen once a week," the generation AI sets the cleaning frequency of the kitchen to once a week. The control unit also causes the generation AI to adjust the cleaning frequency of a specific area based on a user's instructions. For example, if a user instructs the generation AI to "clean the bedroom once a month," the generation AI sets the cleaning frequency of the bedroom to once a month. This allows for efficient cleaning by adjusting the cleaning frequency based on the user's instructions.

[0073] The control unit can use the emotion estimation function to prioritize cleaning areas that the user is particularly concerned about. For example, the control unit causes the generation AI to analyze the user's emotional state and prioritize cleaning areas that the user is particularly concerned about. For example, if it is determined that the user is particularly concerned about the living room, the generation AI suggests, "I will prioritize cleaning the living room." The control unit also causes the generation AI to analyze the user's emotional state and prioritize cleaning areas that the user is particularly concerned about. For example, if it is determined that the user is particularly concerned about the kitchen, the generation AI suggests, "I will prioritize cleaning the kitchen." The control unit also causes the generation AI to analyze the user's emotional state and prioritize cleaning areas that the user is particularly concerned about. For example, if it is determined that the user is particularly concerned about the bedroom, the generation AI suggests, "I will prioritize cleaning the bedroom." This allows the user's satisfaction to be improved by prioritizing cleaning areas that the user is particularly concerned about.

[0074] The control unit can detect the movements of the user's pet and clean specific areas while the pet is away. For example, the control unit uses a camera to detect the movements of the user's pet through the generation AI, and cleans the living room while the pet is away. For example, when the pet moves to another room, the control unit suggests, "I'll clean the living room." The control unit can also use a camera to detect the movements of the user's pet and clean the kitchen while the pet is away. For example, the control unit suggests, "I'll clean the kitchen" while the pet is in the living room. The control unit can also use a camera to detect the movements of the user's pet and clean the bedroom while the pet is away. For example, the control unit suggests, "I'll clean the bedroom" while the pet is out. This enables efficient cleaning by cleaning specific areas while the pet is away.

[0075] The control unit learns the layout of the user's furniture and is able to efficiently clean under the furniture and in the gaps between the furniture. For example, the control unit allows the generation AI to learn the layout of the user's furniture and propose a route that efficiently cleans under the sofa and in the gaps between the furniture. For example, it automatically generates a route that focuses on cleaning under the sofa. The control unit also allows the generation AI to learn the layout of the user's furniture and propose a route that efficiently cleans under the table and in the gaps between the furniture. For example, it automatically generates a route that focuses on cleaning under the table. The control unit also allows the generation AI to learn the layout of the user's furniture and propose a route that efficiently cleans under the bed and in the gaps between the furniture. For example, it automatically generates a route that focuses on cleaning under the bed. This enables more thorough cleaning by efficiently cleaning under the furniture and in the gaps between the furniture.

[0076] The control unit uses the emotion estimation function to enable the cleaning robot to automatically suggest the next area to clean if the user is satisfied. For example, the control unit allows the generation AI to analyze the user's emotional state and suggest the next area if the user is satisfied. For example, if it is determined that the user is satisfied with cleaning the living room, the generation AI suggests, "Shall we clean the kitchen next?" The control unit also allows the generation AI to analyze the user's emotional state and suggest the next area if the user is satisfied. For example, if it is determined that the user is satisfied with cleaning the kitchen, the generation AI suggests, "Shall we clean the bedroom next?" The control unit also allows the generation AI to analyze the user's emotional state and suggest the next area if the user is satisfied. For example, if it is determined that the user is satisfied with cleaning the bedroom, the generation AI suggests, "Shall we clean the living room next?" This enables efficient cleaning by suggesting the next area if the user is satisfied.

[0077] The state tracking unit can analyze the sensor data of the cleaning robot in real time and notify the user if an abnormality is detected. For example, the generation AI analyzes the sensor data of the cleaning robot in real time and notifies the user if an abnormality is detected. For example, if the cleaning robot hits an obstacle, the state tracking unit notifies the user, saying, "The cleaning robot has hit an obstacle." The generation AI also analyzes the sensor data of the cleaning robot in real time and notifies the user if an abnormality is detected. For example, if the battery of the cleaning robot is low, the state tracking unit notifies the user, saying, "The battery is low." The generation AI also analyzes the sensor data of the cleaning robot in real time and notifies the user if an abnormality is detected. For example, if the brush of the cleaning robot is clogged, the state tracking unit notifies the user, saying, "The brush is clogged." This allows for a quick response by notifying the user if an abnormality is detected.

[0078] The status tracking unit can automatically generate a maintenance schedule for the cleaning robot and propose it to the user. For example, the generation AI in the status tracking unit analyzes the usage status of the cleaning robot, automatically generates an optimal maintenance schedule, and proposes it to the user. For example, it notifies the user that "the next maintenance is one week later." The status tracking unit also analyzes the sensor data of the cleaning robot and automatically detects when maintenance is necessary and proposes it to the user. For example, it notifies the user that "it's time to replace the brush." ​​The status tracking unit also learns the usage history of the cleaning robot, automatically generates an optimal maintenance schedule, and proposes it to the user. For example, it notifies the user that "it's time to clean the filter." This automatically generated maintenance schedule enables efficient maintenance.

[0079] The state tracking unit can use the emotion estimation function to provide a detailed status report when the user is feeling anxious. For example, the generation AI analyzes the user's emotional state and provides a detailed status report when the user is feeling anxious. For example, if the user anxiously asks, "Is the cleaning robot okay?", the generation AI reports in detail, "The current battery level is 80%, and cleaning the living room is in progress." The state tracking unit also analyzes the user's emotional state and provides a detailed status report when the user is feeling anxious. For example, if the user anxiously asks, "What's the status of the cleaning robot?", the generation AI reports in detail, "The brush condition is good, and the next maintenance is in three days." The state tracking unit also analyzes the user's emotional state and provides a detailed status report when the user is feeling anxious. For example, if the user anxiously asks, "Where is the cleaning robot?", the generation AI reports in detail, "It is currently cleaning the living room, and there are 10 minutes of cleaning time left." This allows the user to feel more at ease by providing a detailed status report when they are feeling anxious.

[0080] The status tracking unit can predict when to replace the cleaning robot's consumables (such as filters and brushes) and notify the user. For example, the generation AI in the status tracking unit analyzes the cleaning robot's usage data, predicts when to replace the filter, and notifies the user. For example, it notifies the user that "It's almost time to replace the filter." The status tracking unit also analyzes the cleaning robot's sensor data, predicts when to replace the brush, and notifies the user. For example, it notifies the user that "It's almost time to replace the brush." ​​The status tracking unit also learns the cleaning robot's usage history, predicts when to replace consumables, and notifies the user. For example, it notifies the user that "Please replace the filter and brush at the next maintenance." This enables efficient maintenance by predicting when to replace consumables.

[0081] The state tracking unit can analyze the cleaning robot's past operation history and suggest the optimal operation pattern. For example, the state tracking unit allows the generation AI to analyze the cleaning robot's past operation history and suggest the optimal operation pattern. For example, the state tracking unit notifies the user, "Based on past data, we suggest a new pattern to make cleaning the living room more efficient." The state tracking unit also allows the generation AI to analyze the cleaning robot's operation history and suggest the optimal operation pattern to improve cleaning efficiency. For example, the state tracking unit suggests, "Would you like to try a new operation pattern to make cleaning the kitchen more efficient?" The state tracking unit also allows the generation AI to analyze the cleaning robot's past operation history and suggest the optimal operation pattern based on the user's cleaning habits. For example, the state tracking unit notifies the user, "We suggest a new operation pattern to make cleaning the bedroom more efficient." This enables efficient cleaning by analyzing past operation history.

[0082] The state tracking unit can use the emotion estimation function to provide a concise status report when the user feels relieved. For example, the generation AI analyzes the user's emotional state and provides a concise status report when the user feels relieved. For example, if it is determined that the user feels relieved, the state tracking unit briefly reports, "The cleaning robot is operating normally." The generation AI also analyzes the user's emotional state and provides a concise status report when the user feels relieved. For example, if it is determined that the user feels relieved, the state tracking unit briefly reports, "The battery is fully charged." The generation AI also analyzes the user's emotional state and provides a concise status report when the user feels relieved. For example, if it is determined that the user feels relieved, the state tracking unit briefly reports, "The cleaning robot is operating normally." This reduces the burden on the user by providing a concise status report when the user feels relieved.

[0083] The generative AI can learn the user's cleaning habits and suggest efficient cleaning methods. For example, the generative AI can learn the user's cleaning habits and suggest efficient cleaning methods. For example, it might suggest, "Why not try rearranging the furniture to make cleaning the living room more efficient?" The generative AI can also learn the user's cleaning habits and suggest efficient cleaning methods. For example, it might suggest, "Why not try changing the order of cleaning to make cleaning the kitchen more efficient?" The generative AI can also learn the user's cleaning habits and suggest efficient cleaning methods. For example, it might suggest, "Why not try adjusting the frequency of cleaning to make cleaning the bedroom more efficient?" In this way, the generative AI can learn the user's cleaning habits and suggest efficient cleaning methods.

[0084] The generative AI can suggest appropriate cleaning methods based on the user's home environment (e.g., whether or not they have pets or allergies). The generative AI can, for example, learn the user's home environment and suggest appropriate cleaning methods based on whether or not they have pets. For example, it might suggest, "Why not try using a specific brush to efficiently clean up pet hair?" The generative AI can also learn the user's home environment and suggest appropriate cleaning methods based on allergies. For example, it might suggest, "Why not try using a HEPA filter to reduce allergies?" The generative AI can also learn the user's home environment and suggest appropriate cleaning methods based on whether or not they have pets or allergies. For example, it might suggest, "Why not try using a specific cleaning mode to efficiently clean up pet hair and allergens?" This enables efficient cleaning by suggesting appropriate cleaning methods based on the home environment.

[0085] The generative AI can use its emotion estimation function to provide specific solutions when the user is in trouble. For example, the generative AI can analyze the user's emotional state and provide specific solutions when the user is in trouble. For example, if it determines that the user is in trouble, it might suggest, "If cleaning the living room isn't going well, try replacing the brush." ​​The generative AI can also analyze the user's emotional state and provide specific solutions when the user is in trouble. For example, if it determines that the user is in trouble, it might suggest, "If cleaning the kitchen isn't going well, try changing the cleaning mode." The generative AI can also analyze the user's emotional state and provide specific solutions when the user is in trouble. For example, if it determines that the user is in trouble, it might suggest, "If cleaning the bedroom isn't going well, try cleaning the filter." This allows the system to help the user solve their problems by providing specific solutions when they are in trouble.

[0086] The generation AI can analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, the generation AI can analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, it can suggest, "It's almost time to replace the brush. Here's how to replace it." The generation AI can also analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, it can suggest, "It's almost time to clean the filter. Here's how to clean it." The generation AI can also analyze the user's usage history of the cleaning robot and suggest the optimal maintenance method. For example, it can suggest, "It's almost time to replace the battery. Here's how to replace it." In this way, the generation AI can suggest the optimal maintenance method by analyzing the usage history.

[0087] The generating AI can incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, the generating AI can incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, it can propose, "Everyone in the family wants to clean the living room. Clean the living room as a priority." The generating AI can also incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, it can propose, "Everyone in the family wants to clean the kitchen. Clean the kitchen as a priority." The generating AI can also incorporate the opinions of all members of the user's family and propose a cleaning method that will satisfy everyone. For example, it can propose, "Everyone in the family wants to clean the bedroom. Clean the bedroom as a priority." In this way, the generating AI can incorporate the opinions of all members of the family and propose a cleaning method that will satisfy everyone.

[0088] The generation AI can use its emotion estimation function to make suggestions for further efficiency improvements if the user is satisfied. For example, the generation AI analyzes the user's emotional state and makes suggestions for further efficiency improvements if the user is satisfied. For example, if it determines that the user is satisfied, it may suggest, "Why not rearrange the furniture to make cleaning your living room more efficient?" The generation AI also analyzes the user's emotional state and makes suggestions for further efficiency improvements if the user is satisfied. For example, if it determines that the user is satisfied, it may suggest, "Why not change the order of cleaning tasks to make cleaning your kitchen more efficient?" The generation AI also analyzes the user's emotional state and makes suggestions for further efficiency improvements if the user is satisfied. For example, if it determines that the user is satisfied, it may suggest, "Why not adjust the frequency of cleaning tasks to make cleaning your bedroom more efficient?" In this way, by making suggestions for further efficiency improvements if the user is satisfied, it is possible to further improve cleaning efficiency.

[0089] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0090] The control unit can monitor the user's health condition and suggest an appropriate cleaning mode. For example, if the user has allergies, the generating AI will suggest, "Would you like to clean in allergy prevention mode?" If the user has a cold, the generating AI will suggest, "Would you like to clean in cold prevention mode?" Furthermore, if the user feels they are not getting enough exercise, the generating AI will suggest, "Would you like to help clean while exercising?" This can support a healthier living environment by providing cleaning modes that suit the user's health condition.

[0091] The communication unit estimates the user's emotional state, and when the user is relaxed, the cleaning robot can operate in quiet mode. For example, if the user is determined to be relaxed, the generation AI suggests, "Clean the living room in quiet mode." Also, if the user is determined to be relaxed, the generation AI suggests, "Clean the kitchen in quiet mode." Furthermore, if the user is determined to be relaxed, the generation AI suggests, "Clean the bedroom in quiet mode." This allows the robot to operate in quiet mode when the user is relaxed, providing a comfortable environment.

[0092] The control unit can detect the movements of the user's pet and clean specific areas while the pet is away. For example, the generation AI can use a camera to detect the movements of the user's pet and clean the living room while the pet is away. For example, when the pet moves to another room, it can suggest, "I'll clean the living room." The generation AI can also use a camera to detect the movements of the user's pet and clean the kitchen while the pet is away. For example, it can suggest, "I'll clean the kitchen" while the pet is in the living room. The generation AI can also use a camera to detect the movements of the user's pet and clean the bedroom while the pet is away. For example, it can suggest, "I'll clean the bedroom" while the pet is out. This allows for efficient cleaning by cleaning specific areas while the pet is away.

[0093] The communication unit estimates the user's emotional state, allowing the cleaning robot to offer jokes and light conversation when the user is enjoying themselves. For example, if the user says "Please clean up" with a smile, the generation AI will joke, "Of course, but maybe the cleaning robot needs a break too." If the user says "Please clean up" in a cheerful tone, the generation AI will suggest, "Shall we talk about the latest movie while we clean the living room?" If the user says "Please clean up" in a cheerful tone, the generation AI will report, "The cleaning of the living room is going well. It looks like you're dancing." This allows the robot to offer jokes and light conversation when the user is enjoying themselves, providing a more enjoyable experience.

[0094] The control unit can learn the user's schedule and suggest the optimal cleaning time. For example, the generation AI can work with the user's calendar app to learn the user's schedule and suggest the optimal cleaning time. For example, if the user is in a meeting, it can suggest, "Shall we clean the living room now?". The generation AI can also learn the user's schedule and suggest cleaning times when the user is not at home. For example, it can suggest, "Shall we clean the kitchen now?" when the user is out. The generation AI can also learn the user's schedule and suggest cleaning times when the user is relaxing. For example, it can suggest, "Shall we clean the bedroom now?" when the user is watching a movie. This allows for efficient cleaning by suggesting the optimal cleaning time based on the user's schedule.

[0095] The control unit learns the layout of the user's furniture and is able to efficiently clean under furniture and in gaps. For example, the generation AI learns the layout of the user's furniture and suggests a route that efficiently cleans under sofas and in gaps. For example, it automatically generates a route that focuses on cleaning under sofas. The generation AI also learns the layout of the user's furniture and suggests a route that efficiently cleans under tables and in gaps. For example, it automatically generates a route that focuses on cleaning under tables. The generation AI also learns the layout of the user's furniture and suggests a route that efficiently cleans under beds and in gaps. For example, it automatically generates a route that focuses on cleaning under beds. This enables more thorough cleaning by efficiently cleaning under furniture and in gaps.

[0096] The communication unit can estimate the user's emotional state and generate a response that will relax the user if they are feeling stressed. For example, if the user says, "I'm really tired today," the generation AI will generate a response such as, "Thank you for your hard work. Leave it to me to clean the living room." If the user says, "I'm feeling irritated today," the generation AI will suggest, "Shall I play some relaxing music?" If the user says, "I don't feel like doing anything today," the generation AI will suggest, "Would you like to watch a relaxing movie?" This allows the system to generate a relaxing response when the user is feeling stressed, thereby reducing the user's psychological burden.

[0097] The control unit can learn the layout of the user's home and automatically generate the optimal cleaning route. For example, the generation AI learns the layout of the user's home and automatically generates the optimal cleaning route. For example, it proposes a route that efficiently cleans from the living room to the kitchen and bedroom. The generation AI also learns the layout of the user's home and automatically generates the optimal cleaning route that takes into account the arrangement of furniture. For example, it proposes a route that efficiently cleans under sofas and tables. Furthermore, the generation AI learns the layout of the user's home and automatically generates the optimal cleaning route based on the user's cleaning priorities. For example, it proposes a route that cleans the living room first, followed by the kitchen and bedroom. This enables efficient cleaning by automatically generating the optimal cleaning route based on the layout of the home.

[0098] The control unit can adjust the cleaning frequency of specific areas based on user instructions. For example, the generation AI adjusts the cleaning frequency of specific areas based on user instructions. For example, if the user instructs, "Clean the living room every day," the generation AI sets the cleaning frequency of the living room to every day. Also, if the user instructs, "Clean the kitchen once a week," the generation AI sets the cleaning frequency of the kitchen to once a week. Furthermore, if the user instructs, "Clean the bedroom once a month," the generation AI sets the cleaning frequency of the bedroom to once a month. This allows for efficient cleaning by adjusting the cleaning frequency based on user instructions.

[0099] The control unit can estimate the user's emotional state and suggest a quick cleaning mode if the user is in a hurry. For example, if the user says, "Clean quickly," the generation AI will suggest, "Clean the living room in quick mode." Also, if the user says, "Clean quickly," the generation AI will suggest, "Increase the speed of the cleaning robot and clean the living room." Furthermore, if the user says, "Clean the living room quickly," the generation AI will suggest, "Quickly clean only the important areas of the living room." This allows for efficient cleaning by suggesting a quick cleaning mode when the user is in a hurry.

[0100] The processing flow of the second embodiment will be briefly explained below.

[0101] Step 1: The communication unit communicates with the user through the generation AI. For example, it receives the user's voice instructions and conveys them to the generation AI. It can also convey the responses generated by the generation AI to the user. Step 2: The control unit controls the cleaning robot based on the user's instructions received by the communication unit. For example, if the user instructs the cleaning robot to "clean the kitchen," the control unit instructs the cleaning robot to clean the kitchen. Also, if the user instructs the cleaning robot to "clean under the sofa," the control unit instructs the cleaning robot to clean under the sofa. Step 3: The status tracking unit monitors the status of the cleaning robot. For example, when the cleaning robot completes cleaning or when the battery is low, it sends a notification to the user through the generation AI. It can also monitor the location and operating status of the cleaning robot in real time and report it to the user as needed.

[0102] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0103] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0104] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0105] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0106] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0107] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0108] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0109] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0110] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0111] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0112] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0113] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0114] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0115] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0116] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0117] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0118] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AI other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0119] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0120] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0121] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0122] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0123] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0124] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0125] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0126] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0127] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0128] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0129] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0130] In the headset type terminal 314, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0131] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0132] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0133] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AI other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0134] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0135] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0136] 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0137] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0138] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0139] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0140] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0141] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0142] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0143] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0144] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0145] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate a user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, but is not limited to these examples. Furthermore, the estimation and prediction of emotion also includes, for example, emotion analysis.

[0146] In the robot 414, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. The robot 414 also has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0147] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0148] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0149] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AI other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0150] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0151] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0152] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0153] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0154] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0155] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0156] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0157] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0158] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0159] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0160] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0161] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0162] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0163] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0164] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0165] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0166] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0167] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0168] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0169] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. A cleaning robot equipped with generative AI, The generated AI is a communication unit that communicates with users; a control unit that controls the cleaning robot based on the instruction of the user received by the communication unit; a status tracking unit that monitors the status of the cleaning robot A system characterized by:

2. The communication unit Analyzing the user's tone of voice and phrasing to generate a response that corresponds to the user's emotional state 2. The system of claim 1.

3. The communication unit Learns the user's past instruction history and makes suggestions based on the user's preferences and habits 2. The system of claim 1.

4. The communication unit Generate a relaxing response if the user is feeling stressed 2. The system of claim 1.

5. The communication unit A camera recognizes the user's gestures and facial expressions and generates a response based on them.

2. The system of claim 1.

6. The communication unit Communicate with multiple users simultaneously and process each user's instructions appropriately 2. The system of claim 1.

Images