System

Abstract

A system is provided.SOLUTION: A system, comprising: means for receiving a request from a user; means for generating a cleaning schedule based on the received request; means for controlling a cleaning robot according to the cleaning schedule; means for monitoring a status of the cleaning robot; means for notifying the user of status information; and means for collecting feedback from the user and reflecting the feedback in a next cleaning plan.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Existing cleaning robots often fail to meet user expectations, for example, cleaning the wrong spot or getting stuck on objects. This requires users to spend time reconfiguring the robot to address these issues, which can be frustrating. Furthermore, insufficient communication with the user makes it difficult to fully utilize the cleaning robot's functions. This invention aims to solve these problems and provide a user-friendly and efficient cleaning experience. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including the following means: a means for receiving requests from a user, a means for generating a cleaning schedule based on the received requests, a means for controlling the cleaning robot according to the cleaning schedule, a means for monitoring the status of the cleaning robot, a means for notifying the user of the status information, and a means for collecting feedback from the user and reflecting the feedback in the next cleaning plan. This system allows the user to easily operate the cleaning robot and improves the efficiency and reliability of the robot's operation. Furthermore, by analyzing the user's requests via voice or text, monitoring the robot's operation in real time, and sending an alert to the user if an abnormality occurs, the system allows the user to use the cleaning robot with greater peace of mind.

[0006] A "user" is a person who operates a cleaning robot and provides requests and feedback.

[0007] A "request" refers to cleaning instructions or schedule requests that a user inputs to a cleaning robot.

[0008] A "cleaning schedule" refers to a plan for a cleaning robot to perform cleaning tasks at a specific time or date.

[0009] A "generative AI model" is an algorithm that analyzes user requests and past cleaning data to propose optimal cleaning plans and schedules.

[0010] A "cleaning plan" refers to a plan created by a generative AI model that specifies which specific areas a cleaning robot will clean and how.

[0011] A "cleaning robot" is a mechanical device that cleans automatically and operates based on user instructions and generative AI models.

[0012] "Means" refers to components that perform specific functions or roles within a system.

[0013] The "server" is a computer system that receives user requests, generates cleaning schedules and cleaning plans based on those requests using a generative AI model, and communicates with the cleaning robot and the user.

[0014] A "terminal" is a device operated by a user, including a smartphone, a PC, etc.

[0015] "Status" refers to information indicating the current state and operating status of the cleaning robot.

[0016] "Feedback" refers to the evaluations and opinions that users provide regarding the results of cleaning and the behavior of the cleaning robot.

[0017] An "alert" refers to a warning message that is sent to the user when the cleaning robot detects an abnormality or problem. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system is designed to enable users to operate the cleaning robot from a smartphone or PC terminal to clean efficiently and effectively, and utilizes generative AI models to provide optimal cleaning schedules and plans based on user requests.

[0040] Program processing explanation

[0041] Receiving a user request

[0042] Device: The user launches the application and enters a request via voice or text, such as "Clean the living room at 10 AM on Saturday."

[0043] Terminal: This request is sent to the server.

[0044] Generate a schedule

[0045] Server: The server parses the received request and passes it to the generative AI model.

[0046] Server: The generative AI model takes into account the request and past cleaning history to generate an optimal schedule.

[0047] Server: The generated schedule is notified to the user and they are asked to confirm it.

[0048] Create and review cleaning plans

[0049] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then approves the final schedule.

[0050] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[0051] Server: Sends the completed cleaning plan to the cleaning robot.

[0052] Controlling the cleaning robot

[0053] Cleaning robot: The cleaning robot will automatically start working at the specified time.

[0054] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, efficiently cleaning the target area.

[0055] Status monitoring and user notification

[0056] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[0057] Server: If an abnormality is detected, it will send an alert to the user. If the process is normal, it will send a notification that the cleaning is complete.

[0058] Gathering user feedback

[0059] On-device: After cleaning is complete, the user can check the results in the app and provide feedback if necessary.

[0060] Server: Feedback is collected and passed to the generative AI model, which then influences the creation of the next cleaning plan.

[0061] Specific examples

[0062] For example, if a user makes a request to "clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[0063] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0067] Step 2:

[0068] Terminal: The input request is sent to the server as text data.

[0069] Step 3:

[0070] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[0071] Step 4:

[0072] Server: The generative AI model takes into account the request content and past cleaning history data to generate the optimal cleaning schedule.

[0073] Step 5:

[0074] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[0075] Step 6:

[0076] Terminal: The user can check the notified schedule and make any necessary changes. Once the changes are complete, the final schedule is confirmed.

[0077] Step 7:

[0078] Server: Based on the finalized schedule, the generative AI model creates a specific cleaning plan and sends it to the cleaning robot.

[0079] Step 8:

[0080] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[0081] Step 9:

[0082] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[0083] Step 10:

[0084] Server: The server monitors the status of the cleaning robot in real time, checking cleaning progress and battery level.

[0085] Step 11:

[0086] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[0087] Step 12:

[0088] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[0089] Step 13:

[0090] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[0091] Step 14:

[0092] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[0093] Step 15:

[0094] Server: Collects user feedback, provides it to the generative AI model, and reflects it in creating the next cleaning plan.

[0095] As described above, this system efficiently formulates cleaning plans based on user requests and optimally controls the cleaning robot, providing users with a convenient and reliable cleaning experience.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Manually setting cleaning schedules and operating cleaning robots is time-consuming and difficult for users to clean effectively and efficiently. Furthermore, if an abnormality occurs during cleaning, it is difficult to immediately respond, and feedback on the cleaning results is not reflected in the next cleaning plan.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for receiving requests from a user, means for analyzing the received requests, means for notifying the user's terminal of a schedule generated based on the analysis results, means for the user to modify and approve the schedule, means for controlling the cleaning robot according to the approved cleaning schedule, means for monitoring the status of the cleaning robot in real time, means for notifying the user of status information and abnormalities, and means for collecting feedback from the user and reflecting it in the next cleaning plan. This reduces the user's effort and enables effective and efficient cleaning. Furthermore, by being able to respond quickly to abnormalities when they occur and reflecting feedback in the next cleaning plan, the quality of cleaning can be improved.

[0101] "User" refers to an individual or organization that uses this system to operate a cleaning robot.

[0102] A "request" refers to a cleaning instruction or request that a user makes to the system via a smartphone or other device.

[0103] "Parsing" refers to the process of understanding the received request and converting it into an appropriate data format.

[0104] "Generative AI model" refers to an artificial intelligence model that generates optimal cleaning schedules and plans based on past data and current instructions.

[0105] A "schedule" refers to a timetable for a cleaning robot to clean at a specific time and place.

[0106] "Terminal" refers to a device such as a smartphone or PC that a user uses to access and operate the system.

[0107] A "cleaning robot" is a machine that moves automatically and cleans designated areas.

[0108] "Status" refers to real-time information such as the cleaning robot's current operating status and remaining battery level.

[0109] "Notification" refers to the act of sending information from a server to a user's terminal or the information itself.

[0110] "Feedback" refers to the evaluations and opinions provided by users regarding the cleaning results of the cleaning robot.

[0111] "Real-time" refers to a situation in which information processing and communication occur almost simultaneously, and data is reflected without delay.

[0112] This invention is a system that allows users to operate a cleaning robot from a smartphone or personal computer (PC) to clean efficiently and effectively. The system utilizes a generative AI model to provide an optimal cleaning schedule and cleaning plan based on the user's requests.

[0113] The user launches a dedicated application on their smartphone or PC and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday." The device receives this request and sends it to the server, which interprets and analyzes the request and retrieves the necessary data (for example, past cleaning history and room layout information) from a database.

[0114] The server passes the interpreted and analyzed requests to the generative AI model, which generates an optimal cleaning schedule. The generated schedule is then sent to the user's device. The user can check the schedule on their device and make any necessary changes. Once the changes are complete and the user approves them, the final cleaning plan is generated.

[0115] This final cleaning plan is sent from the server to the cleaning robot. The cleaning robot operates automatically at the designated time and cleans while monitoring its surroundings in real time using obstacle sensors and cameras. The server monitors the status of the cleaning robot in real time, checking the battery condition and cleaning progress. If an abnormality is detected, the server immediately sends an alert to the user's device, and if the cleaning is progressing normally, it sends a notification that the cleaning is complete.

[0116] After the cleaning is completed, the user can review the results on the app and provide feedback, which is then sent to the server via the device, where it is passed to the generative AI model to be used to plan the next cleaning.

[0117] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[0118] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[0119] Prompt Sentence Examples

[0120] Say, "Clean the living room at 10am on Saturday."

[0121] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0122] Step 1:

[0123] User: The user launches a dedicated smartphone app and inputs a request via voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0124] Operation:

[0125] The user speaks into the smartphone's microphone or enters a request via text.

[0126] Input: "Clean the living room on Saturday at 10 AM"

[0127] Output: Request as text data

[0128] Step 2:

[0129] Terminal: Send this text data to the server.

[0130] Operation:

[0131] The app converts the speech into text and generates text data.

[0132] Input: A voice or text request from the user

[0133] Output: HTTPS request to the server

[0134] Step 3:

[0135] Server: Interprets and analyzes the received request.

[0136] Operation:

[0137] The server analyzes the received text data and extracts the necessary information.

[0138] Input: Request as text data

[0139] Output: Structured data as a result of analysis

[0140] Step 4:

[0141] Server: Based on the analysis results, retrieves past cleaning history and house layout information from the database.

[0142] Operation:

[0143] The server queries the database to retrieve the required data.

[0144] Input: Structured data of analysis results

[0145] Output: Past cleaning history and house layout information

[0146] Step 5:

[0147] Server: Passes the acquired data and analysis results to the AI ​​model to generate an optimal cleaning schedule.

[0148] Operation:

[0149] A generative AI model calculates the schedule based on past data and requests.

[0150] Input: Past cleaning history, house layout information, structured data of analysis results

[0151] Output: Optimal cleaning schedule

[0152] Step 6:

[0153] Server: Notifies the user of the generated schedule.

[0154] Operation:

[0155] The server sends the schedule in JSON format to the user's device.

[0156] Enter: your optimal cleaning schedule

[0157] Output: Notification to user terminal

[0158] Step 7:

[0159] Terminal: The user checks the schedule and adjusts it if necessary.

[0160] Operation:

[0161] The app displays the schedule to the user and provides an interface for modification.

[0162] Input: Schedule as notification to user device

[0163] Output: Revised or approved schedule

[0164] Step 8:

[0165] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[0166] Operation:

[0167] A generative AI model generates a detailed cleaning plan based on the approved schedule.

[0168] Input: Approved Schedule

[0169] Output: A detailed cleaning plan

[0170] Step 9:

[0171] Server: Sends the completed cleaning plan to the cleaning robot.

[0172] Operation:

[0173] The server transmits the cleaning plan to the cleaning robot via communication.

[0174] Input: Specific cleaning plan

[0175] Output: Instructions for the cleaning robot

[0176] Step 10:

[0177] Cleaning robot: Automatically starts cleaning at the specified time.

[0178] Operation:

[0179] The cleaning robot will start moving according to the schedule and perform cleaning.

[0180] Input: Instructions for the cleaning robot

[0181] Output: Cleaning run

[0182] Step 11:

[0183] Server: Monitors the status of the cleaning robot in real time.

[0184] Operation:

[0185] The server receives data from the cleaning robot and checks its battery status and progress.

[0186] Input: Status data from the cleaning robot

[0187] Output: Real-time monitoring information

[0188] Step 12:

[0189] Server: If an abnormality is detected, an alert is sent immediately to the user's device.

[0190] Operation:

[0191] If an abnormality is detected, an alert message is sent to the user terminal.

[0192] Input: Anomaly detection status data

[0193] Output: Alert notification to user terminal

[0194] Step 13:

[0195] Server: If cleaning is completed successfully, send a cleaning completion notification to the user.

[0196] Operation:

[0197] The server sends a "cleaning completed" notification to the user terminal.

[0198] Input: Cleaning completion status data

[0199] Output: Completion notification to user terminal

[0200] Step 14:

[0201] On the device: Users can view the cleaning results and provide feedback on the app.

[0202] Operation:

[0203] The user enters a rating and comment on the app's feedback screen.

[0204] Input: Check cleaning results and enter feedback

[0205] Output: Feedback data

[0206] Step 15:

[0207] Server: Collects feedback and passes it to the generative AI model to help plan the next cleaning.

[0208] Operation:

[0209] The server analyzes the feedback data and applies the feedback to the generative AI model.

[0210] Input: Feedback data

[0211] Output: Improved cleaning plan

[0212] (Application example 1)

[0213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] Conventional cleaning robot management systems require users to input requests and check and modify schedules, which is time-consuming. Even when voice requests are accepted, subsequent processing is limited, making intuitive operation through visual devices difficult. Furthermore, the status of the cleaning robot and notifications in the event of an abnormality are often insufficient, resulting in a loss of user convenience.

[0215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0216] In this invention, the server includes means for receiving requests from a user, means for generating a cleaning schedule based on the received requests, means for controlling the cleaning robot according to the cleaning schedule, means for monitoring the status of the cleaning robot, means for notifying the user of the status information, means for collecting feedback from the user and reflecting it in the next cleaning plan, means for visually confirming and modifying the cleaning schedule through the user's visual device, and means for converting voice requests from the user into text and processing the requests. This allows the user to easily make requests through voice input and intuitively confirm and modify the cleaning schedule using the visual device in real time. Furthermore, since the status of the cleaning robot is monitored in real time and the user is immediately notified in the event of an abnormality, the cleaning robot can be managed more efficiently and effectively.

[0217] The "means for receiving a request from a user" refers to a device or software for receiving an operation instruction given by the user to the cleaning robot.

[0218] "Means for generating a cleaning schedule based on received requests" refers to an algorithm or program that analyzes user instructions and determines appropriate cleaning timings in accordance with those instructions.

[0219] The "means for controlling the cleaning robot in accordance with the cleaning schedule" is a control system for operating the cleaning robot in accordance with the generated schedule.

[0220] The "means for monitoring the status of the cleaning robot" is a monitoring system for obtaining real-time information such as the operating status and battery condition of the cleaning robot.

[0221] The "means for notifying the user of status information" is a notification system for informing the user of the current status or abnormalities of the cleaning robot.

[0222] The "means of collecting feedback from users and reflecting it in the next cleaning plan" is a data collection and analysis system that receives users' evaluations and requests and uses them in the next cleaning plan.

[0223] "Means for visually reviewing and modifying the cleaning schedule through a user's visual device" refers to an interface and software that allows a user to review and, if necessary, modify their cleaning plan using a visual device such as smart glasses or a head-mounted display.

[0224] The "means for converting voice requests from the user into text and processing the requests" is a voice recognition system that recognizes user instructions input by voice as text and processes them as control instructions for the cleaning robot.

[0225] This invention is a system that efficiently manages a cleaning robot using user voice instructions. By wearing a visual device such as smart glasses and inputting voice requests, the user can create and modify cleaning schedules and monitor the status of the cleaning robot in real time. Specific embodiments of this system are described below.

[0226] Hardware and software used

[0227] Smart glasses (e.g. Google Glass, Vuzix Blade)

[0228] Google Cloud Speech-to-Text API: A solution for converting voice input to text

[0229] Firebase Functions: Cloud functions for sending and processing requests

[0230] OpenAI GPT-4 model: Generative AI that analyzes requests and generates schedules and cleaning plans

[0231] iRobot API: An interface for controlling the cleaning robot

[0232] Firebase Realtime Database: A database for monitoring the status of the cleaning robot in real time.

[0233] Processing flow

[0234] Receiving and converting user requests

[0235] A user uses the smart glasses to input voice commands, such as "Clean the living room at 10 AM on Saturday," which are converted into text by the Google Cloud Speech-to-Text API through the smart glasses' microphone and sent to the server via Firebase Functions.

[0236] Generate a schedule

[0237] The server analyzes the received request and passes it to the OpenAI GPT-4 model. This generative AI model takes into account the request and past cleaning history to generate an optimal cleaning schedule. The generated schedule is visually displayed on the user's smart glasses.

[0238] Check and modify the schedule

[0239] The user can review the proposed schedule through the smart glasses and make any necessary changes, either by touchpad or voice input. Once the user finally approves the schedule, it is sent to the server and the cleaning plan is finalized.

[0240] Controlling the cleaning robot

[0241] The cleaning plan is sent to the cleaning robot via the iRobot API, and the cleaning robot starts cleaning at the designated time, checking its surroundings using obstacle sensors and cameras.

[0242] Status monitoring and notification

[0243] The status of the cleaning robot is monitored in real time using Firebase Realtime Database. If an abnormality is detected, the server sends an alert to the user's smart glasses, and if normal progress is made, a notification is sent when cleaning is complete.

[0244] Gathering and implementing feedback

[0245] After cleaning is complete, the user can check the results through the smart glasses and provide feedback, which is then sent back to the server and collected to be used to create the next cleaning plan.

[0246] Specific examples

[0247] For example, if a user provides a voice request such as "Clean the living room at 10 AM on Saturday," the following occurs:

[0248] 1. Voice command: "Clean the living room at 10 AM on Saturday."

[0249] 2. Text Transformation: "Clean the living room at 10 AM on Saturday."

[0250] 3. Send request: Send to server

[0251] 4. Analysis and generation: Generate schedules and cleaning plans using OpenAI GPT-4 models

[0252] 5. Notification and confirmation: The schedule is notified to the smart glasses and displayed as "Clean the living room at 10:00 AM on Saturday."

[0253] 6. Send cleaning plan: After approval, send it to the cleaning robot via iRobot API

[0254] 7. Real-time monitoring: The status of the cleaning robot is monitored in real time, and users are notified if an abnormality occurs.

[0255] This allows users to easily make requests through voice input and intuitively check and modify cleaning schedules in real time using visual devices.

[0256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0257] Step 1:

[0258] The user speaks their request

[0259] Users input voice requests through the smart glasses, such as "Clean the living room at 10 a.m. on Saturday," and this voice input is captured by the smart glasses' microphone.

[0260] Input: Voice request (e.g., "Clean the living room at 10 AM on Saturday")

[0261] Output: Audio data

[0262] Step 2:

[0263] Transcription of voice requests

[0264] The smart glasses send the captured voice data to the Google Cloud Speech-to-Text API, which analyzes the voice data and converts it into corresponding text data.

[0265] Input: Audio data

[0266] Output: Text data (e.g. "Clean the living room at 10 AM on Saturday")

[0267] Step 3:

[0268] Sending a text request to the server

[0269] The smart glasses then send the converted text data to the server via Firebase Functions, which includes details such as the date, time, and location specified by the user.

[0270] Input: Text data (e.g., "Clean the living room at 10 AM on Saturday")

[0271] Output: Request data sent to the server

[0272] Step 4:

[0273] Schedule generation using generative AI models

[0274] The server analyzes the received request data and passes it to the OpenAI GPT-4 model, which then generates an optimal cleaning schedule based on the request and the user's cleaning history.

[0275] Input: Request data (e.g., "Clean the living room at 10 AM on Saturday"), past cleaning history

[0276] Output: Optimal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[0277] Step 5:

[0278] Schedule notification and confirmation

[0279] The generated cleaning schedule is sent to the user's smart glasses via Firebase Functions, where the user can view the visual display of the schedule and modify it using the touchpad or voice input.

[0280] Input: Your ideal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[0281] Output: Schedule, confirmation or correction data sent to the user's smart glasses

[0282] Step 6:

[0283] Send revised schedule and approve

[0284] Once the user confirms and modifies the schedule and finally approves it, the modified schedule is sent back to the server, which then receives the schedule and finalizes the cleaning plan.

[0285] Input: Check or correct data

[0286] Output: Approved cleaning schedule

[0287] Step 7:

[0288] Sending cleaning plans to cleaning robots

[0289] The server sends the determined cleaning plan to the cleaning robot via the iRobot API, and the cleaning robot automatically starts working at the specified time.

[0290] Input: Approved cleaning schedule

[0291] Output: The cleaning plan sent to the cleaning robot

[0292] Step 8:

[0293] Real-time monitoring of cleaning robot status

[0294] The status of the cleaning robot is monitored in real time using the Firebase Realtime Database, where progress and battery status are continuously checked.

[0295] Input: Vacuum robot status information

[0296] Output: Real-time status data

[0297] Step 9:

[0298] Status information notification and abnormality detection

[0299] If an abnormality is detected, the server will send a real-time alert to the user's smart glasses. Under normal circumstances, a cleaning completion notification will be sent.

[0300] Input: Real-time status data, anomaly detection data

[0301] Output: Alert notified to user or cleaning completion notification

[0302] Step 10:

[0303] Collecting and incorporating user feedback

[0304] After the cleaning is completed, the user can check the cleaning results through the smart glasses and provide feedback, which will be sent to the server and reflected in the generative AI model's next plan.

[0305] Input: Feedback data

[0306] Output: Feedback collected, data reflected in next cleaning plan

[0307] This allows users to easily operate the cleaning robot through voice input and manage it in real time using a visual device.

[0308] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0309] The present invention is a system that recognizes a user's emotions and adjusts the cleaning schedule and cleaning plan based on those emotions to provide a more suitable cleaning experience for the user. This system is equipped with an emotion engine that recognizes the user's emotions, analyzes the user's emotional state, and reflects the analysis results in the generation of the next cleaning plan and notification content.

[0310] Program processing explanation

[0311] Receiving a user request

[0312] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0313] Terminal: This request is sent to the server.

[0314] Generate a schedule

[0315] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[0316] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[0317] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[0318] Analysis and adjustment by emotion engine

[0319] Device: The user indicates their emotions through the camera on their smartphone or PC, either through voice or facial expression. For example, the user may have a dissatisfied expression or an angry tone of voice.

[0320] Server: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state. If they express dissatisfaction, it is assumed that there is a problem with the cleaning schedule or content.

[0321] Server: Based on the user's emotional state, the generative AI model readjusts the cleaning schedule and plan to best reflect it.

[0322] Create and review cleaning plans

[0323] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then confirms the final schedule.

[0324] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[0325] Server: Sends the completed cleaning plan to the cleaning robot.

[0326] Controlling the cleaning robot

[0327] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[0328] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[0329] Status monitoring and user notification

[0330] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[0331] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[0332] Cleaning completion and notification

[0333] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[0334] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[0335] Gathering user feedback

[0336] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[0337] Server: User feedback is passed to the emotion engine, which analyzes the user's emotional state again. The analysis results are reflected in the generation of the next cleaning plan and notification content.

[0338] Specific examples

[0339] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process goes like this: The user enters the request, which is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is then notified to the user, who can review it and make any necessary adjustments. The emotion engine analyzes the user's emotions and readjusts the schedule and plans accordingly.

[0340] The final cleaning plan is sent to the cleaning robot, which starts cleaning at the specified time. The cleaning robot is monitored in real time, and if any problems occur, an alert is sent to the user. After the cleaning is completed, the user can check the results and provide feedback. This feedback is reflected in the generation of the next cleaning plan, and the emotion engine provides a more personalized cleaning experience for the user.

[0341] As a result, the system can take the user's emotions into account and provide a more flexible and personalized cleaning experience.

[0342] The processing flow will be explained below.

[0343] Step 1:

[0344] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0345] Step 2:

[0346] Terminal: The input request is sent to the server as text data.

[0347] Step 3:

[0348] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[0349] Step 4:

[0350] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[0351] Step 5:

[0352] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[0353] Step 6:

[0354] On the device, the user checks the schedule and makes any necessary changes. Once the changes are complete, the user confirms the final schedule.

[0355] Step 7:

[0356] Server: Based on the final schedule, the generative AI model creates a specific cleaning plan.

[0357] Step 8:

[0358] Server: The emotion engine analyzes the voice and facial expression data provided by the user through the application to identify the user's emotional state.

[0359] Step 9:

[0360] Server: The emotion engine makes necessary adjustments to the cleaning schedule and plan based on the results of emotion analysis. For example, if the user expresses dissatisfaction, the cleaning plan will be reevaluated to improve efficiency.

[0361] Step 10:

[0362] Server: Sends the completed cleaning plan to the cleaning robot.

[0363] Step 11:

[0364] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[0365] Step 12:

[0366] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, allowing it to clean efficiently.

[0367] Step 13:

[0368] Server: Monitors the status of the cleaning robot in real time, checking cleaning progress and battery condition.

[0369] Step 14:

[0370] Server: If an anomaly is detected, for example if the cleaning robot gets stuck on an obstacle, the server immediately sends an alert to the user.

[0371] Step 15:

[0372] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[0373] Step 16:

[0374] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[0375] Step 17:

[0376] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[0377] Step 18:

[0378] Server: The user feedback is passed to the emotion engine, which analyzes the user's emotional state again. For example, if the user is satisfied, the same plan will be offered next time.

[0379] Step 19:

[0380] Server: The results of the user's emotion analysis are reflected in the generation of the next cleaning plan, and the emotion engine adjusts it to provide a cleaning experience that is even more suited to the user.

[0381] As described above, this system efficiently formulates cleaning plans based on user requests and uses an emotion engine to analyze and reflect the user's emotional state, providing a flexible and personalized cleaning experience.

[0382] Example 2

[0383] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0384] Existing cleaning robots and related systems have difficulty providing optimal cleaning plans for users because they are unable to fully reflect user emotions and feedback. Furthermore, they lack systems for quickly responding to abnormalities that occur during the cleaning robot's operation. This can lead to an unsatisfactory user experience.

[0385] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0386] In this invention, the server includes a means for receiving requests from a user, a means for analyzing the received requests using a generative AI model to generate a cleaning schedule, and a means for recognizing the user's emotions and adjusting the cleaning schedule based on the user's emotional state. This allows for an optimal cleaning schedule that reflects the user's requests and emotions, enabling a more personalized cleaning experience for the user. It also enables rapid response through real-time monitoring and anomaly detection.

[0387] The "means for receiving requests from users" refers to a mechanism that allows the system to receive requests in voice or text format sent by users using smartphones or personal computer terminals.

[0388] "Means for analyzing using a generative AI model and generating a cleaning schedule" refers to a function that uses a generative AI model to analyze received user requests and automatically create an optimal cleaning schedule based on them.

[0389] "Means for recognizing the user's emotions and adjusting the cleaning schedule based on their emotional state" refers to a system that uses sensors such as cameras and microphones to analyze the user's emotions from their facial expressions and voice, and then optimizes the existing cleaning schedule based on the results of that analysis.

[0390] "Means for controlling the cleaning robot according to the adjusted cleaning schedule" means hardware and software mechanisms for directing and managing the operation of the cleaning robot according to the optimized cleaning schedule.

[0391] "Means for monitoring the status of the cleaning robot in real time" refers to a function that acquires and monitors the operating status, battery status, sensor information, etc. of the cleaning robot in real time.

[0392] The "means for notifying the user of status information" is a mechanism for notifying the user of the current cleaning status, whether or not there are any abnormalities, etc., based on status information monitored in real time.

[0393] "Means for collecting feedback from users and reflecting it in the next cleaning plan along with the results of the analysis of their emotional state" is a function that collects feedback provided by users and reflects it, including the results of the emotion analysis by the emotion engine, in optimizing the next cleaning schedule and cleaning plan.

[0394] This invention relates to a system that recognizes a user's emotions and adjusts cleaning schedules and plans based on them. The system receives requests from the user and generates an optimal cleaning schedule using a generative AI model. It then analyzes the user's emotional state using an emotion engine and readjusts the cleaning schedule based on the analysis results. It also has a function to monitor the operation of the cleaning robot and notify the user of the status in real time.

[0395] A user logs in to the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text. For example, a user may input a request such as "Clean the living room at 10:00 AM on Saturday." This request is then sent from the terminal to the server.

[0396] The server receives the request and analyzes it using a generative AI model. At this time, the voice data is converted into text and input as a prompt to the generative AI model. For example, a prompt such as, "Based on the user's request, please schedule cleaning of the living room for 10:00 AM on Saturday. Also, please consider past cleaning history data and propose the optimal plan" is given to the generative AI model.

[0397] The generative AI model generates an optimal cleaning schedule based on the request and past cleaning history data, and the server notifies the user's device. The user can review this schedule and make any necessary changes. Once changes are complete, the final schedule is confirmed.

[0398] The device captures the user's facial expressions and voice through a camera and microphone. The emotion engine on the server analyzes this data to identify the user's emotional state. It identifies emotions such as dissatisfaction or joy and adjusts the cleaning schedule and plan accordingly.

[0399] The determined cleaning schedule is sent from the server to the cleaning robot. The cleaning robot starts working at the designated time and begins cleaning the living room. The cleaning robot uses obstacle sensors and cameras to clean efficiently.

[0400] The server monitors the status of the cleaning robot in real time, checking its operation and battery condition. If an abnormality occurs, an alert is sent to the user immediately. When cleaning is complete, the server generates a "Cleaning Completed" notification and sends it to the user's device. The user can check this notification and view detailed cleaning results through the application.

[0401] Users can provide feedback on their cleaning results and input it within the application. This feedback is passed to the emotion engine on the server, which again analyzes the user's emotional state. The analysis results are reflected in the generation of the next cleaning plan and notifications, providing a more personalized cleaning experience.

[0402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0403] Step 1:

[0404] Input: A user logs into the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0405] Specific operation: After the user inputs a request, the device sends the request to the server. In the case of voice input, the device converts the voice data into text data.

[0406] Output: The input request is sent to the server in text format.

[0407] Step 2:

[0408] Input: The server receives the request from the device.

[0409] What happens: The server parses this request and prepares it to be input as a prompt to the generative AI model.

[0410] Data processing and calculation: The voice data is converted into text, and a prompt sentence is generated in a format suitable for the generative AI model.

[0411] Output: The generative AI model is given a prompt such as, "Based on the user's request, please schedule the living room cleaning for 10 AM on Saturday. Also, please take into account past cleaning history data and suggest the optimal plan."

[0412] Step 3:

[0413] Input: The prompt passed to the generative AI model from the server.

[0414] Specific operation: The generative AI model analyzes the request content based on the prompt text and generates an optimal cleaning schedule.

[0415] Data processing and calculation: The request details are compared with past cleaning history data to generate the optimal cleaning schedule.

[0416] Output: The generated optimal cleaning schedule is returned to the server.

[0417] Step 4:

[0418] Input: Optimal cleaning schedule from a generative AI model.

[0419] Specific operation: The server notifies the user's device of the generated cleaning schedule.

[0420] Output: A cleaning schedule notification is sent to the user's device.

[0421] Step 5:

[0422] Input: Cleaning schedule notification sent to the user's device.

[0423] What happens next: The user reviews the proposed schedule, makes any necessary adjustments, and then confirms the final schedule.

[0424] Output: The modified or confirmed schedule is sent to the server.

[0425] Step 6:

[0426] Input: Confirmed cleaning schedule.

[0427] Specific operation: The server sends the determined cleaning schedule to the cleaning robot.

[0428] Output: The cleaning schedule sent to the cleaning robot.

[0429] Step 7:

[0430] Input: The cleaning schedule sent to the cleaning robot.

[0431] Specific operation: The cleaning robot will automatically start working at the specified time and clean the living room.

[0432] Output: The operating status of the cleaning robot is sent to the server in real time.

[0433] Step 8:

[0434] Input: The operating status of the cleaning robot.

[0435] Specific operation: The server monitors the status of the cleaning robot in real time and sends an alert to the user if an abnormality occurs.

[0436] Output: The user is notified of the operation status of the cleaning robot and the monitoring results of the battery status.

[0437] Step 9:

[0438] Input: Notification that the robot vacuum cleaner has completed cleaning.

[0439] Specific operation: The server generates a notification that "cleaning is complete" and sends it to the user's device.

[0440] Output: A cleaning completion notification is sent to the user's device.

[0441] Step 10:

[0442] Input: Notification after cleaning is completed and cleaning result.

[0443] What it does: The user reviews the cleaning results and provides feedback within the app.

[0444] Output: Feedback is sent to the server and reflected in the next cleaning plan.

[0445] Step 11:

[0446] Input: User feedback.

[0447] Specific operation: The server's emotion engine analyzes the feedback and the user's emotional state and reflects it in the next cleaning plan.

[0448] Output: Emotional state analysis and feedback will be incorporated into the next cleaning plan.

[0449] (Application example 2)

[0450] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0451] Conventional work device control systems generate work schedules and control devices based on user requests, but do not consider the user's emotional state. As a result, they are unable to provide effective work plans when the user is dissatisfied or when a response tailored to a specific situation is required. Especially in brick-and-mortar stores, understanding the customer's emotional state leads to improved service, so responses that reflect emotions are required.

[0452] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing and analyzing the user's emotions, means for adjusting the work schedule and work plan based on the analyzed emotional information, and means for collecting feedback from the user and reflecting it in the next work plan. This makes it possible to provide a more effective and personalized work plan that corresponds to the user's emotional state.

[0453] "User" means any person or entity that uses the System.

[0454] A "request" refers to a specific request for work or service sent by a user.

[0455] "Work Schedule" refers to the timeline and sequence of work to be created based on received requests.

[0456] "Working device" refers to hardware or robotic systems that perform work based on a specified schedule.

[0457] "Status" indicates status information such as the current working state of the working device, progress, and whether or not there is a problem.

[0458] "Emotion" refers to a user's mental state or mood, and is generally analyzed from facial expressions, tone of voice, etc.

[0459] "Analysis" refers to the process of deriving specific information or results from input data.

[0460] "Adjusting work schedules and plans" refers to modifying and optimizing existing schedules and plans based on analyzed data.

[0461] "Feedback" refers to the evaluations and impressions collected from users after completing a task.

[0462] 1. System Overview

[0463] The system for implementing this invention captures the user's emotions and adjusts the behavior of a customer service robot based on those emotions. The user sends a request using a PC or smartphone, and the customer service robot takes appropriate action in response to the request. This system analyzes the user's facial expressions and voice to provide optimal service in real time.

[0464] 2. Hardware and Software Configuration

[0465] The server implements this system using the following hardware and software:

[0466] Customer service robot: A robotic device that handles customer service in brick-and-mortar stores

[0467] Camera: A device for capturing the user's facial expressions

[0468] Microphone: A device for capturing the user's voice.

[0469] OpenCV: Image processing library, used for face detection and facial expression analysis

[0470] PyTorch: A machine learning library for running emotion recognition models

[0471] Google Cloud Speech-to-Text: Speech recognition API, converts user speech into text

[0472] paho.mqtt: MQTT communication module, sending emotion data in real time

[0473] 3. Processing Flow

[0474] The server receives requests from users and generates and adjusts work schedules based on the requests by using the following means:

[0475] First, the system captures the user's facial expressions using a camera and analyzes their emotions using OpenCV and PyTorch. It also collects the user's voice using a microphone and converts it into text using Google Cloud Speech-to-Text. This data is then sent to a server, where an optimal work schedule is generated using a generative AI model.

[0476] 4. Generative AI Model and Prompts

[0477] Based on the generated emotion data, the generative AI model builds optimized work schedules and plans.

[0478] As an example of a prompt sentence, the following sentence is fed into the generative AI model:

[0479] User emotions detected: [negative, positive]

[0480] Based on this prompt, the server generates the optimal response based on the user's emotions and reflects it in the customer service robot.

[0481] 5. Specific Examples

[0482] For example, if a user "looks busy" in a physical store, a customer service robot can be instructed to respond quickly by saying, "Let me know if there's anything I can help you with." This is achieved by a generative AI model that analyzes the user's emotions from the prompt text and generates an appropriate response.

[0483] In this way, the system can grasp the user's emotional state in real time and provide appropriate services, thereby significantly improving the user experience.

[0484] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0485] Step 1:

[0486] The terminal receives a request from the user. The input is a request (voice or text) sent by the user using a smartphone or PC. This request is sent to the server.

[0487] Step 2:

[0488] The server analyzes the received request. As input, it receives the user's textual request, which is passed to the generative AI model. Here, the audio data is converted to text using Google Cloud Speech-to-Text. As output, the request content is analyzed and the data required for further processing is generated.

[0489] Step 3:

[0490] The server uses a generative AI model to generate an optimal work schedule, taking into account the request content and past data. The inputs are the analyzed request data and past work history data. This data is then input into the generative AI model to process the data and generate an optimal schedule. The output is the generated work schedule.

[0491] Step 4:

[0492] The server sends the generated work schedule to the terminal. The input is the generated work schedule. The output is a confirmation notification sent to the user's terminal. The user checks this and makes any necessary corrections.

[0493] Step 5:

[0494] The device uses a camera and microphone to recognize and analyze the user's emotions. The input is the user's facial expressions and voice. The facial expression data captured by the camera is analyzed using OpenCV and PyTorch, and the voice data is converted to text using Google Cloud Speech-to-Text. The output is the analyzed emotion data.

[0495] Step 6:

[0496] The server adjusts the work schedule and work plan using a generative AI model based on the analyzed emotion data. The inputs are the analyzed emotion data and the generated work schedule. For data calculations, the emotion data is input into the generative AI model as a prompt sentence, and the work schedule is adjusted. An example of a prompt sentence is "User emotions detected: [negative, positive]". The output is an adjusted work schedule.

[0497] Step 7:

[0498] The server creates a final work plan based on the adjusted work schedule and sends it to the terminal. The input is the adjusted work schedule. The output is the final work plan sent to the user's terminal.

[0499] Step 8:

[0500] The terminal sends the final work plan to the customer service robot and has it execute it. The input is the final work plan sent from the server. The output is that the customer service robot starts the specified work.

[0501] Step 9:

[0502] The server monitors the status of the customer service robot in real time and sends an alert to the user if an abnormality is detected. The input is the status information of the customer service robot. The output is an alert sent to the user's device if an abnormality is detected.

[0503] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0504] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0505] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0506] [Second embodiment]

[0507] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0508] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0509] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0510] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0511] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0512] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0513] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0514] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0515] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0516] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0517] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0518] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0519] The system is designed to enable users to operate the cleaning robot from a smartphone or PC terminal to clean efficiently and effectively, and utilizes generative AI models to provide optimal cleaning schedules and plans based on user requests.

[0520] Program processing explanation

[0521] Receiving a user request

[0522] Device: The user launches the application and enters a request via voice or text, such as "Clean the living room at 10 AM on Saturday."

[0523] Terminal: This request is sent to the server.

[0524] Generate a schedule

[0525] Server: The server parses the received request and passes it to the generative AI model.

[0526] Server: The generative AI model takes into account the request and past cleaning history to generate an optimal schedule.

[0527] Server: The generated schedule is notified to the user and they are asked to confirm it.

[0528] Create and review cleaning plans

[0529] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then approves the final schedule.

[0530] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[0531] Server: Sends the completed cleaning plan to the cleaning robot.

[0532] Controlling the cleaning robot

[0533] Cleaning robot: The cleaning robot will automatically start working at the specified time.

[0534] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, efficiently cleaning the target area.

[0535] Status monitoring and user notification

[0536] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[0537] Server: If an abnormality is detected, it will send an alert to the user. If the process is normal, it will send a notification that the cleaning is complete.

[0538] Gathering user feedback

[0539] On-device: After cleaning is complete, the user can check the results in the app and provide feedback if necessary.

[0540] Server: Feedback is collected and passed to the generative AI model, which then influences the creation of the next cleaning plan.

[0541] Specific examples

[0542] For example, if a user makes a request to "clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[0543] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[0544] The processing flow will be explained below.

[0545] Step 1:

[0546] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0547] Step 2:

[0548] Terminal: The input request is sent to the server as text data.

[0549] Step 3:

[0550] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[0551] Step 4:

[0552] Server: The generative AI model takes into account the request content and past cleaning history data to generate the optimal cleaning schedule.

[0553] Step 5:

[0554] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[0555] Step 6:

[0556] Terminal: The user can check the notified schedule and make any necessary changes. Once the changes are complete, the final schedule is confirmed.

[0557] Step 7:

[0558] Server: Based on the finalized schedule, the generative AI model creates a specific cleaning plan and sends it to the cleaning robot.

[0559] Step 8:

[0560] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[0561] Step 9:

[0562] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[0563] Step 10:

[0564] Server: The server monitors the status of the cleaning robot in real time, checking cleaning progress and battery level.

[0565] Step 11:

[0566] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[0567] Step 12:

[0568] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[0569] Step 13:

[0570] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[0571] Step 14:

[0572] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[0573] Step 15:

[0574] Server: Collects user feedback, provides it to the generative AI model, and reflects it in creating the next cleaning plan.

[0575] As described above, this system efficiently formulates cleaning plans based on user requests and optimally controls the cleaning robot, providing users with a convenient and reliable cleaning experience.

[0576] Example 1

[0577] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] Manually setting cleaning schedules and operating cleaning robots is time-consuming and difficult for users to clean effectively and efficiently. Furthermore, if an abnormality occurs during cleaning, it is difficult to immediately respond, and feedback on the cleaning results is not reflected in the next cleaning plan.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0580] In this invention, the server includes means for receiving requests from a user, means for analyzing the received requests, means for notifying the user's terminal of a schedule generated based on the analysis results, means for the user to modify and approve the schedule, means for controlling the cleaning robot according to the approved cleaning schedule, means for monitoring the status of the cleaning robot in real time, means for notifying the user of status information and abnormalities, and means for collecting feedback from the user and reflecting it in the next cleaning plan. This reduces the user's effort and enables effective and efficient cleaning. Furthermore, by being able to respond quickly to abnormalities when they occur and reflecting feedback in the next cleaning plan, the quality of cleaning can be improved.

[0581] "User" refers to an individual or organization that uses this system to operate a cleaning robot.

[0582] A "request" refers to a cleaning instruction or request that a user makes to the system via a smartphone or other device.

[0583] "Parsing" refers to the process of understanding the received request and converting it into an appropriate data format.

[0584] "Generative AI model" refers to an artificial intelligence model that generates optimal cleaning schedules and plans based on past data and current instructions.

[0585] A "schedule" refers to a timetable for a cleaning robot to clean at a specific time and place.

[0586] "Terminal" refers to a device such as a smartphone or PC that a user uses to access and operate the system.

[0587] A "cleaning robot" is a machine that moves automatically and cleans designated areas.

[0588] "Status" refers to real-time information such as the cleaning robot's current operating status and remaining battery level.

[0589] "Notification" refers to the act of sending information from a server to a user's terminal or the information itself.

[0590] "Feedback" refers to the evaluations and opinions provided by users regarding the cleaning results of the cleaning robot.

[0591] "Real-time" refers to a situation in which information processing and communication occur almost simultaneously, and data is reflected without delay.

[0592] This invention is a system that allows users to operate a cleaning robot from a smartphone or personal computer (PC) to clean efficiently and effectively. The system utilizes a generative AI model to provide an optimal cleaning schedule and cleaning plan based on the user's requests.

[0593] The user launches a dedicated application on their smartphone or PC and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday." The device receives this request and sends it to the server, which interprets and analyzes the request and retrieves the necessary data (for example, past cleaning history and room layout information) from a database.

[0594] The server passes the interpreted and analyzed requests to the generative AI model, which generates an optimal cleaning schedule. The generated schedule is then sent to the user's device. The user can check the schedule on their device and make any necessary changes. Once the changes are complete and the user approves them, the final cleaning plan is generated.

[0595] This final cleaning plan is sent from the server to the cleaning robot. The cleaning robot operates automatically at the designated time and cleans while monitoring its surroundings in real time using obstacle sensors and cameras. The server monitors the status of the cleaning robot in real time, checking the battery condition and cleaning progress. If an abnormality is detected, the server immediately sends an alert to the user's device, and if the cleaning is progressing normally, it sends a notification that the cleaning is complete.

[0596] After the cleaning is completed, the user can review the results on the app and provide feedback, which is then sent to the server via the device, where it is passed to the generative AI model to be used to plan the next cleaning.

[0597] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[0598] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[0599] Prompt Sentence Examples

[0600] Say, "Clean the living room at 10am on Saturday."

[0601] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0602] Step 1:

[0603] User: The user launches a dedicated smartphone app and inputs a request via voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0604] Operation:

[0605] The user speaks into the smartphone's microphone or enters a request via text.

[0606] Input: "Clean the living room on Saturday at 10 AM"

[0607] Output: Request as text data

[0608] Step 2:

[0609] Terminal: Send this text data to the server.

[0610] Operation:

[0611] The app converts the speech into text and generates text data.

[0612] Input: A voice or text request from the user

[0613] Output: HTTPS request to the server

[0614] Step 3:

[0615] Server: Interprets and analyzes the received request.

[0616] Operation:

[0617] The server analyzes the received text data and extracts the necessary information.

[0618] Input: Request as text data

[0619] Output: Structured data as a result of analysis

[0620] Step 4:

[0621] Server: Based on the analysis results, retrieves past cleaning history and house layout information from the database.

[0622] Operation:

[0623] The server queries the database to retrieve the required data.

[0624] Input: Structured data of analysis results

[0625] Output: Past cleaning history and house layout information

[0626] Step 5:

[0627] Server: Passes the acquired data and analysis results to the AI ​​model to generate an optimal cleaning schedule.

[0628] Operation:

[0629] A generative AI model calculates the schedule based on past data and requests.

[0630] Input: Past cleaning history, house layout information, structured data of analysis results

[0631] Output: Optimal cleaning schedule

[0632] Step 6:

[0633] Server: Notifies the user of the generated schedule.

[0634] Operation:

[0635] The server sends the schedule in JSON format to the user's device.

[0636] Enter: your optimal cleaning schedule

[0637] Output: Notification to user terminal

[0638] Step 7:

[0639] Terminal: The user checks the schedule and adjusts it if necessary.

[0640] Operation:

[0641] The app displays the schedule to the user and provides an interface for modification.

[0642] Input: Schedule as notification to user device

[0643] Output: Revised or approved schedule

[0644] Step 8:

[0645] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[0646] Operation:

[0647] A generative AI model generates a detailed cleaning plan based on the approved schedule.

[0648] Input: Approved Schedule

[0649] Output: A detailed cleaning plan

[0650] Step 9:

[0651] Server: Sends the completed cleaning plan to the cleaning robot.

[0652] Operation:

[0653] The server transmits the cleaning plan to the cleaning robot via communication.

[0654] Input: Specific cleaning plan

[0655] Output: Instructions for the cleaning robot

[0656] Step 10:

[0657] Cleaning robot: Automatically starts cleaning at the specified time.

[0658] Operation:

[0659] The cleaning robot will start moving according to the schedule and perform cleaning.

[0660] Input: Instructions for the cleaning robot

[0661] Output: Cleaning run

[0662] Step 11:

[0663] Server: Monitors the status of the cleaning robot in real time.

[0664] Operation:

[0665] The server receives data from the cleaning robot and checks its battery status and progress.

[0666] Input: Status data from the cleaning robot

[0667] Output: Real-time monitoring information

[0668] Step 12:

[0669] Server: If an abnormality is detected, an alert is sent immediately to the user's device.

[0670] Operation:

[0671] If an abnormality is detected, an alert message is sent to the user terminal.

[0672] Input: Anomaly detection status data

[0673] Output: Alert notification to user terminal

[0674] Step 13:

[0675] Server: If cleaning is completed successfully, send a cleaning completion notification to the user.

[0676] Operation:

[0677] The server sends a "cleaning completed" notification to the user terminal.

[0678] Input: Cleaning completion status data

[0679] Output: Completion notification to user terminal

[0680] Step 14:

[0681] On the device: Users can view the cleaning results and provide feedback on the app.

[0682] Operation:

[0683] The user enters a rating and comment on the app's feedback screen.

[0684] Input: Check cleaning results and enter feedback

[0685] Output: Feedback data

[0686] Step 15:

[0687] Server: Collects feedback and passes it to the generative AI model to help plan the next cleaning.

[0688] Operation:

[0689] The server analyzes the feedback data and applies the feedback to the generative AI model.

[0690] Input: Feedback data

[0691] Output: Improved cleaning plan

[0692] (Application example 1)

[0693] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0694] Conventional cleaning robot management systems require users to input requests and check and modify schedules, which is time-consuming. Even when voice requests are accepted, subsequent processing is limited, making intuitive operation through visual devices difficult. Furthermore, the status of the cleaning robot and notifications in the event of an abnormality are often insufficient, resulting in a loss of user convenience.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0696] In this invention, the server includes means for receiving requests from a user, means for generating a cleaning schedule based on the received requests, means for controlling the cleaning robot according to the cleaning schedule, means for monitoring the status of the cleaning robot, means for notifying the user of the status information, means for collecting feedback from the user and reflecting it in the next cleaning plan, means for visually confirming and modifying the cleaning schedule through the user's visual device, and means for converting voice requests from the user into text and processing the requests. This allows the user to easily make requests through voice input and intuitively confirm and modify the cleaning schedule using the visual device in real time. Furthermore, since the status of the cleaning robot is monitored in real time and the user is immediately notified in the event of an abnormality, the cleaning robot can be managed more efficiently and effectively.

[0697] The "means for receiving a request from a user" refers to a device or software for receiving an operation instruction given by the user to the cleaning robot.

[0698] "Means for generating a cleaning schedule based on received requests" refers to an algorithm or program that analyzes user instructions and determines appropriate cleaning timings in accordance with those instructions.

[0699] The "means for controlling the cleaning robot in accordance with the cleaning schedule" is a control system for operating the cleaning robot in accordance with the generated schedule.

[0700] The "means for monitoring the status of the cleaning robot" is a monitoring system for obtaining real-time information such as the operating status and battery condition of the cleaning robot.

[0701] The "means for notifying the user of status information" is a notification system for informing the user of the current status or abnormalities of the cleaning robot.

[0702] The "means of collecting feedback from users and reflecting it in the next cleaning plan" is a data collection and analysis system that receives users' evaluations and requests and uses them in the next cleaning plan.

[0703] "Means for visually reviewing and modifying the cleaning schedule through a user's visual device" refers to an interface and software that allows a user to review and, if necessary, modify their cleaning plan using a visual device such as smart glasses or a head-mounted display.

[0704] The "means for converting voice requests from the user into text and processing the requests" is a voice recognition system that recognizes user instructions input by voice as text and processes them as control instructions for the cleaning robot.

[0705] This invention is a system that efficiently manages a cleaning robot using user voice instructions. By wearing a visual device such as smart glasses and inputting voice requests, the user can create and modify cleaning schedules and monitor the status of the cleaning robot in real time. Specific embodiments of this system are described below.

[0706] Hardware and software used

[0707] Smart glasses (e.g. Google Glass, Vuzix Blade)

[0708] Google Cloud Speech-to-Text API: A solution for converting voice input to text

[0709] Firebase Functions: Cloud functions for sending and processing requests

[0710] OpenAI GPT-4 model: Generative AI that analyzes requests and generates schedules and cleaning plans

[0711] iRobot API: An interface for controlling the cleaning robot

[0712] Firebase Realtime Database: A database for monitoring the status of the cleaning robot in real time.

[0713] Processing flow

[0714] Receiving and converting user requests

[0715] A user uses the smart glasses to input voice commands, such as "Clean the living room at 10 AM on Saturday," which are converted into text by the Google Cloud Speech-to-Text API through the smart glasses' microphone and sent to the server via Firebase Functions.

[0716] Generate a schedule

[0717] The server analyzes the received request and passes it to the OpenAI GPT-4 model. This generative AI model takes into account the request and past cleaning history to generate an optimal cleaning schedule. The generated schedule is visually displayed on the user's smart glasses.

[0718] Check and modify the schedule

[0719] The user can review the proposed schedule through the smart glasses and make any necessary changes, either by touchpad or voice input. Once the user finally approves the schedule, it is sent to the server and the cleaning plan is finalized.

[0720] Controlling the cleaning robot

[0721] The cleaning plan is sent to the cleaning robot via the iRobot API, and the cleaning robot starts cleaning at the designated time, checking its surroundings using obstacle sensors and cameras.

[0722] Status monitoring and notification

[0723] The status of the cleaning robot is monitored in real time using Firebase Realtime Database. If an abnormality is detected, the server sends an alert to the user's smart glasses, and if normal progress is made, a notification is sent when cleaning is complete.

[0724] Gathering and implementing feedback

[0725] After cleaning is complete, the user can check the results through the smart glasses and provide feedback, which is then sent back to the server and collected to be used to create the next cleaning plan.

[0726] Specific examples

[0727] For example, if a user provides a voice request such as "Clean the living room at 10 AM on Saturday," the following occurs:

[0728] 1. Voice command: "Clean the living room at 10 AM on Saturday."

[0729] 2. Text Transformation: "Clean the living room at 10 AM on Saturday."

[0730] 3. Send request: Send to server

[0731] 4. Analysis and generation: Generate schedules and cleaning plans using OpenAI GPT-4 models

[0732] 5. Notification and confirmation: The schedule is notified to the smart glasses and displayed as "Clean the living room at 10:00 AM on Saturday."

[0733] 6. Send cleaning plan: After approval, send it to the cleaning robot via iRobot API

[0734] 7. Real-time monitoring: The status of the cleaning robot is monitored in real time, and users are notified if an abnormality occurs.

[0735] This allows users to easily make requests through voice input and intuitively check and modify cleaning schedules in real time using visual devices.

[0736] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0737] Step 1:

[0738] The user speaks their request

[0739] Users input voice requests through the smart glasses, such as "Clean the living room at 10 a.m. on Saturday," and this voice input is captured by the smart glasses' microphone.

[0740] Input: Voice request (e.g., "Clean the living room at 10 AM on Saturday")

[0741] Output: Audio data

[0742] Step 2:

[0743] Transcription of voice requests

[0744] The smart glasses send the captured voice data to the Google Cloud Speech-to-Text API, which analyzes the voice data and converts it into corresponding text data.

[0745] Input: Audio data

[0746] Output: Text data (e.g. "Clean the living room at 10 AM on Saturday")

[0747] Step 3:

[0748] Sending a text request to the server

[0749] The smart glasses then send the converted text data to the server via Firebase Functions, which includes details such as the date, time, and location specified by the user.

[0750] Input: Text data (e.g., "Clean the living room at 10 AM on Saturday")

[0751] Output: Request data sent to the server

[0752] Step 4:

[0753] Schedule generation using generative AI models

[0754] The server analyzes the received request data and passes it to the OpenAI GPT-4 model, which then generates an optimal cleaning schedule based on the request and the user's cleaning history.

[0755] Input: Request data (e.g., "Clean the living room at 10 AM on Saturday"), past cleaning history

[0756] Output: Optimal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[0757] Step 5:

[0758] Schedule notification and confirmation

[0759] The generated cleaning schedule is sent to the user's smart glasses via Firebase Functions, where the user can view the visual display of the schedule and modify it using the touchpad or voice input.

[0760] Input: Your ideal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[0761] Output: Schedule, confirmation or correction data sent to the user's smart glasses

[0762] Step 6:

[0763] Send revised schedule and approve

[0764] Once the user confirms and modifies the schedule and finally approves it, the modified schedule is sent back to the server, which then receives the schedule and finalizes the cleaning plan.

[0765] Input: Check or correct data

[0766] Output: Approved cleaning schedule

[0767] Step 7:

[0768] Sending cleaning plans to cleaning robots

[0769] The server sends the determined cleaning plan to the cleaning robot via the iRobot API, and the cleaning robot automatically starts working at the specified time.

[0770] Input: Approved cleaning schedule

[0771] Output: The cleaning plan sent to the cleaning robot

[0772] Step 8:

[0773] Real-time monitoring of cleaning robot status

[0774] The status of the cleaning robot is monitored in real time using the Firebase Realtime Database, where progress and battery status are continuously checked.

[0775] Input: Vacuum robot status information

[0776] Output: Real-time status data

[0777] Step 9:

[0778] Status information notification and abnormality detection

[0779] If an abnormality is detected, the server will send a real-time alert to the user's smart glasses. Under normal circumstances, a cleaning completion notification will be sent.

[0780] Input: Real-time status data, anomaly detection data

[0781] Output: Alert notified to user or cleaning completion notification

[0782] Step 10:

[0783] Collecting and incorporating user feedback

[0784] After the cleaning is completed, the user can check the cleaning results through the smart glasses and provide feedback, which will be sent to the server and reflected in the generative AI model's next plan.

[0785] Input: Feedback data

[0786] Output: Feedback collected, data reflected in next cleaning plan

[0787] This allows users to easily operate the cleaning robot through voice input and manage it in real time using a visual device.

[0788] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0789] The present invention is a system that recognizes a user's emotions and adjusts the cleaning schedule and cleaning plan based on those emotions to provide a more suitable cleaning experience for the user. This system is equipped with an emotion engine that recognizes the user's emotions, analyzes the user's emotional state, and reflects the analysis results in the generation of the next cleaning plan and notification content.

[0790] Program processing explanation

[0791] Receiving a user request

[0792] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0793] Terminal: This request is sent to the server.

[0794] Generate a schedule

[0795] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[0796] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[0797] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[0798] Analysis and adjustment by emotion engine

[0799] Device: The user indicates their emotions through the camera on their smartphone or PC, either through voice or facial expression. For example, the user may have a dissatisfied expression or an angry tone of voice.

[0800] Server: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state. If they express dissatisfaction, it is assumed that there is a problem with the cleaning schedule or content.

[0801] Server: Based on the user's emotional state, the generative AI model readjusts the cleaning schedule and plan to best reflect it.

[0802] Create and review cleaning plans

[0803] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then confirms the final schedule.

[0804] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[0805] Server: Sends the completed cleaning plan to the cleaning robot.

[0806] Controlling the cleaning robot

[0807] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[0808] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[0809] Status monitoring and user notification

[0810] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[0811] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[0812] Cleaning completion and notification

[0813] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[0814] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[0815] Gathering user feedback

[0816] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[0817] Server: User feedback is passed to the emotion engine, which analyzes the user's emotional state again. The analysis results are reflected in the generation of the next cleaning plan and notification content.

[0818] Specific examples

[0819] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process goes like this: The user enters the request, which is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is then notified to the user, who can review it and make any necessary adjustments. The emotion engine analyzes the user's emotions and readjusts the schedule and plans accordingly.

[0820] The final cleaning plan is sent to the cleaning robot, which starts cleaning at the specified time. The cleaning robot is monitored in real time, and if any problems occur, an alert is sent to the user. After the cleaning is completed, the user can check the results and provide feedback. This feedback is reflected in the generation of the next cleaning plan, and the emotion engine provides a more personalized cleaning experience for the user.

[0821] As a result, the system can take the user's emotions into account and provide a more flexible and personalized cleaning experience.

[0822] The processing flow will be explained below.

[0823] Step 1:

[0824] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0825] Step 2:

[0826] Terminal: The input request is sent to the server as text data.

[0827] Step 3:

[0828] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[0829] Step 4:

[0830] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[0831] Step 5:

[0832] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[0833] Step 6:

[0834] On the device, the user checks the schedule and makes any necessary changes. Once the changes are complete, the user confirms the final schedule.

[0835] Step 7:

[0836] Server: Based on the final schedule, the generative AI model creates a specific cleaning plan.

[0837] Step 8:

[0838] Server: The emotion engine analyzes the voice and facial expression data provided by the user through the application to identify the user's emotional state.

[0839] Step 9:

[0840] Server: The emotion engine makes necessary adjustments to the cleaning schedule and plan based on the results of emotion analysis. For example, if the user expresses dissatisfaction, the cleaning plan will be reevaluated to improve efficiency.

[0841] Step 10:

[0842] Server: Sends the completed cleaning plan to the cleaning robot.

[0843] Step 11:

[0844] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[0845] Step 12:

[0846] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, allowing it to clean efficiently.

[0847] Step 13:

[0848] Server: Monitors the status of the cleaning robot in real time, checking cleaning progress and battery condition.

[0849] Step 14:

[0850] Server: If an anomaly is detected, for example if the cleaning robot gets stuck on an obstacle, the server immediately sends an alert to the user.

[0851] Step 15:

[0852] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[0853] Step 16:

[0854] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[0855] Step 17:

[0856] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[0857] Step 18:

[0858] Server: The user feedback is passed to the emotion engine, which analyzes the user's emotional state again. For example, if the user is satisfied, the same plan will be offered next time.

[0859] Step 19:

[0860] Server: The results of the user's emotion analysis are reflected in the generation of the next cleaning plan, and the emotion engine adjusts it to provide a cleaning experience that is even more suited to the user.

[0861] As described above, this system efficiently formulates cleaning plans based on user requests and uses an emotion engine to analyze and reflect the user's emotional state, providing a flexible and personalized cleaning experience.

[0862] Example 2

[0863] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0864] Existing cleaning robots and related systems have difficulty providing optimal cleaning plans for users because they are unable to fully reflect user emotions and feedback. Furthermore, they lack systems for quickly responding to abnormalities that occur during the cleaning robot's operation. This can lead to an unsatisfactory user experience.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0866] In this invention, the server includes a means for receiving requests from a user, a means for analyzing the received requests using a generative AI model to generate a cleaning schedule, and a means for recognizing the user's emotions and adjusting the cleaning schedule based on the user's emotional state. This allows for an optimal cleaning schedule that reflects the user's requests and emotions, enabling a more personalized cleaning experience for the user. It also enables rapid response through real-time monitoring and anomaly detection.

[0867] The "means for receiving requests from users" refers to a mechanism that allows the system to receive requests in voice or text format sent by users using smartphones or personal computer terminals.

[0868] "Means for analyzing using a generative AI model and generating a cleaning schedule" refers to a function that uses a generative AI model to analyze received user requests and automatically create an optimal cleaning schedule based on them.

[0869] "Means for recognizing the user's emotions and adjusting the cleaning schedule based on their emotional state" refers to a system that uses sensors such as cameras and microphones to analyze the user's emotions from their facial expressions and voice, and then optimizes the existing cleaning schedule based on the results of that analysis.

[0870] "Means for controlling the cleaning robot according to the adjusted cleaning schedule" means hardware and software mechanisms for directing and managing the operation of the cleaning robot according to the optimized cleaning schedule.

[0871] "Means for monitoring the status of the cleaning robot in real time" refers to a function that acquires and monitors the operating status, battery status, sensor information, etc. of the cleaning robot in real time.

[0872] The "means for notifying the user of status information" is a mechanism for notifying the user of the current cleaning status, whether or not there are any abnormalities, etc., based on status information monitored in real time.

[0873] "Means for collecting feedback from users and reflecting it in the next cleaning plan along with the results of the analysis of their emotional state" is a function that collects feedback provided by users and reflects it, including the results of the emotion analysis by the emotion engine, in optimizing the next cleaning schedule and cleaning plan.

[0874] This invention relates to a system that recognizes a user's emotions and adjusts cleaning schedules and plans based on them. The system receives requests from the user and generates an optimal cleaning schedule using a generative AI model. It then analyzes the user's emotional state using an emotion engine and readjusts the cleaning schedule based on the analysis results. It also has a function to monitor the operation of the cleaning robot and notify the user of the status in real time.

[0875] A user logs in to the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text. For example, a user may input a request such as "Clean the living room at 10:00 AM on Saturday." This request is then sent from the terminal to the server.

[0876] The server receives the request and analyzes it using a generative AI model. At this time, the voice data is converted into text and input as a prompt to the generative AI model. For example, a prompt such as, "Based on the user's request, please schedule cleaning of the living room for 10:00 AM on Saturday. Also, please consider past cleaning history data and propose the optimal plan" is given to the generative AI model.

[0877] The generative AI model generates an optimal cleaning schedule based on the request and past cleaning history data, and the server notifies the user's device. The user can review this schedule and make any necessary changes. Once changes are complete, the final schedule is confirmed.

[0878] The device captures the user's facial expressions and voice through a camera and microphone. The emotion engine on the server analyzes this data to identify the user's emotional state. It identifies emotions such as dissatisfaction or joy and adjusts the cleaning schedule and plan accordingly.

[0879] The determined cleaning schedule is sent from the server to the cleaning robot. The cleaning robot starts working at the designated time and begins cleaning the living room. The cleaning robot uses obstacle sensors and cameras to clean efficiently.

[0880] The server monitors the status of the cleaning robot in real time, checking its operation and battery condition. If an abnormality occurs, an alert is sent to the user immediately. When cleaning is complete, the server generates a "Cleaning Completed" notification and sends it to the user's device. The user can check this notification and view detailed cleaning results through the application.

[0881] Users can provide feedback on their cleaning results and input it within the application. This feedback is passed to the emotion engine on the server, which again analyzes the user's emotional state. The analysis results are reflected in the generation of the next cleaning plan and notifications, providing a more personalized cleaning experience.

[0882] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0883] Step 1:

[0884] Input: A user logs into the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[0885] Specific operation: After the user inputs a request, the device sends the request to the server. In the case of voice input, the device converts the voice data into text data.

[0886] Output: The input request is sent to the server in text format.

[0887] Step 2:

[0888] Input: The server receives the request from the device.

[0889] What happens: The server parses this request and prepares it to be input as a prompt to the generative AI model.

[0890] Data processing and calculation: The voice data is converted into text, and a prompt sentence is generated in a format suitable for the generative AI model.

[0891] Output: The generative AI model is given a prompt such as, "Based on the user's request, please schedule the living room cleaning for 10 AM on Saturday. Also, please take into account past cleaning history data and suggest the optimal plan."

[0892] Step 3:

[0893] Input: The prompt passed to the generative AI model from the server.

[0894] Specific operation: The generative AI model analyzes the request content based on the prompt text and generates an optimal cleaning schedule.

[0895] Data processing and calculation: The request details are compared with past cleaning history data to generate the optimal cleaning schedule.

[0896] Output: The generated optimal cleaning schedule is returned to the server.

[0897] Step 4:

[0898] Input: Optimal cleaning schedule from a generative AI model.

[0899] Specific operation: The server notifies the user's device of the generated cleaning schedule.

[0900] Output: A cleaning schedule notification is sent to the user's device.

[0901] Step 5:

[0902] Input: Cleaning schedule notification sent to the user's device.

[0903] What happens next: The user reviews the proposed schedule, makes any necessary adjustments, and then confirms the final schedule.

[0904] Output: The modified or confirmed schedule is sent to the server.

[0905] Step 6:

[0906] Input: Confirmed cleaning schedule.

[0907] Specific operation: The server sends the determined cleaning schedule to the cleaning robot.

[0908] Output: The cleaning schedule sent to the cleaning robot.

[0909] Step 7:

[0910] Input: The cleaning schedule sent to the cleaning robot.

[0911] Specific operation: The cleaning robot will automatically start working at the specified time and clean the living room.

[0912] Output: The operating status of the cleaning robot is sent to the server in real time.

[0913] Step 8:

[0914] Input: The operating status of the cleaning robot.

[0915] Specific operation: The server monitors the status of the cleaning robot in real time and sends an alert to the user if an abnormality occurs.

[0916] Output: The user is notified of the operation status of the cleaning robot and the monitoring results of the battery status.

[0917] Step 9:

[0918] Input: Notification that the robot vacuum cleaner has completed cleaning.

[0919] Specific operation: The server generates a notification that "cleaning is complete" and sends it to the user's device.

[0920] Output: A cleaning completion notification is sent to the user's device.

[0921] Step 10:

[0922] Input: Notification after cleaning is completed and cleaning result.

[0923] What it does: The user reviews the cleaning results and provides feedback within the app.

[0924] Output: Feedback is sent to the server and reflected in the next cleaning plan.

[0925] Step 11:

[0926] Input: User feedback.

[0927] Specific operation: The server's emotion engine analyzes the feedback and the user's emotional state and reflects it in the next cleaning plan.

[0928] Output: Emotional state analysis and feedback will be incorporated into the next cleaning plan.

[0929] (Application example 2)

[0930] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0931] Conventional work device control systems generate work schedules and control devices based on user requests, but do not consider the user's emotional state. As a result, they are unable to provide effective work plans when the user is dissatisfied or when a response tailored to a specific situation is required. Especially in brick-and-mortar stores, understanding the customer's emotional state leads to improved service, so responses that reflect emotions are required.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing and analyzing the user's emotions, means for adjusting the work schedule and work plan based on the analyzed emotional information, and means for collecting feedback from the user and reflecting it in the next work plan. This makes it possible to provide a more effective and personalized work plan that corresponds to the user's emotional state.

[0933] "User" means any person or entity that uses the System.

[0934] A "request" refers to a specific request for work or service sent by a user.

[0935] "Work Schedule" refers to the timeline and sequence of work to be created based on received requests.

[0936] "Working device" refers to hardware or robotic systems that perform work based on a specified schedule.

[0937] "Status" indicates status information such as the current working state of the working device, progress, and whether or not there is a problem.

[0938] "Emotion" refers to a user's mental state or mood, and is generally analyzed from facial expressions, tone of voice, etc.

[0939] "Analysis" refers to the process of deriving specific information or results from input data.

[0940] "Adjusting work schedules and plans" refers to modifying and optimizing existing schedules and plans based on analyzed data.

[0941] "Feedback" refers to the evaluations and impressions collected from users after completing a task.

[0942] 1. System Overview

[0943] The system for implementing this invention captures the user's emotions and adjusts the behavior of a customer service robot based on those emotions. The user sends a request using a PC or smartphone, and the customer service robot takes appropriate action in response to the request. This system analyzes the user's facial expressions and voice to provide optimal service in real time.

[0944] 2. Hardware and Software Configuration

[0945] The server implements this system using the following hardware and software:

[0946] Customer service robot: A robotic device that handles customer service in brick-and-mortar stores

[0947] Camera: A device for capturing the user's facial expressions

[0948] Microphone: A device for capturing the user's voice.

[0949] OpenCV: Image processing library, used for face detection and facial expression analysis

[0950] PyTorch: A machine learning library for running emotion recognition models

[0951] Google Cloud Speech-to-Text: Speech recognition API, converts user speech into text

[0952] paho.mqtt: MQTT communication module, sending emotion data in real time

[0953] 3. Processing Flow

[0954] The server receives requests from users and generates and adjusts work schedules based on the requests by using the following means:

[0955] First, the system captures the user's facial expressions using a camera and analyzes their emotions using OpenCV and PyTorch. It also collects the user's voice using a microphone and converts it into text using Google Cloud Speech-to-Text. This data is then sent to a server, where an optimal work schedule is generated using a generative AI model.

[0956] 4. Generative AI Model and Prompts

[0957] Based on the generated emotion data, the generative AI model builds optimized work schedules and plans.

[0958] As an example of a prompt sentence, the following sentence is fed into the generative AI model:

[0959] User emotions detected: [negative, positive]

[0960] Based on this prompt, the server generates the optimal response based on the user's emotions and reflects it in the customer service robot.

[0961] 5. Specific Examples

[0962] For example, if a user "looks busy" in a physical store, a customer service robot can be instructed to respond quickly by saying, "Let me know if there's anything I can help you with." This is achieved by a generative AI model that analyzes the user's emotions from the prompt text and generates an appropriate response.

[0963] In this way, the system can grasp the user's emotional state in real time and provide appropriate services, thereby significantly improving the user experience.

[0964] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0965] Step 1:

[0966] The terminal receives a request from the user. The input is a request (voice or text) sent by the user using a smartphone or PC. This request is sent to the server.

[0967] Step 2:

[0968] The server analyzes the received request. As input, it receives the user's textual request, which is passed to the generative AI model. Here, the audio data is converted to text using Google Cloud Speech-to-Text. As output, the request content is analyzed and the data required for further processing is generated.

[0969] Step 3:

[0970] The server uses a generative AI model to generate an optimal work schedule, taking into account the request content and past data. The inputs are the analyzed request data and past work history data. This data is then input into the generative AI model to process the data and generate an optimal schedule. The output is the generated work schedule.

[0971] Step 4:

[0972] The server sends the generated work schedule to the terminal. The input is the generated work schedule. The output is a confirmation notification sent to the user's terminal. The user checks this and makes any necessary corrections.

[0973] Step 5:

[0974] The device uses a camera and microphone to recognize and analyze the user's emotions. The input is the user's facial expressions and voice. The facial expression data captured by the camera is analyzed using OpenCV and PyTorch, and the voice data is converted to text using Google Cloud Speech-to-Text. The output is the analyzed emotion data.

[0975] Step 6:

[0976] The server adjusts the work schedule and work plan using a generative AI model based on the analyzed emotion data. The inputs are the analyzed emotion data and the generated work schedule. For data calculations, the emotion data is input into the generative AI model as a prompt sentence, and the work schedule is adjusted. An example of a prompt sentence is "User emotions detected: [negative, positive]". The output is an adjusted work schedule.

[0977] Step 7:

[0978] The server creates a final work plan based on the adjusted work schedule and sends it to the terminal. The input is the adjusted work schedule. The output is the final work plan sent to the user's terminal.

[0979] Step 8:

[0980] The terminal sends the final work plan to the customer service robot and has it execute it. The input is the final work plan sent from the server. The output is that the customer service robot starts the specified work.

[0981] Step 9:

[0982] The server monitors the status of the customer service robot in real time and sends an alert to the user if an abnormality is detected. The input is the status information of the customer service robot. The output is an alert sent to the user's device if an abnormality is detected.

[0983] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0984] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0985] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0986] [Third embodiment]

[0987] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0988] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0989] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0990] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0991] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0992] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0993] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0994] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0995] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0996] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0997] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0998] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0999] The system is designed to enable users to operate the cleaning robot from a smartphone or PC terminal to clean efficiently and effectively, and utilizes generative AI models to provide optimal cleaning schedules and plans based on user requests.

[1000] Program processing explanation

[1001] Receiving a user request

[1002] Device: The user launches the application and enters a request via voice or text, such as "Clean the living room at 10 AM on Saturday."

[1003] Terminal: This request is sent to the server.

[1004] Generate a schedule

[1005] Server: The server parses the received request and passes it to the generative AI model.

[1006] Server: The generative AI model takes into account the request and past cleaning history to generate an optimal schedule.

[1007] Server: The generated schedule is notified to the user and they are asked to confirm it.

[1008] Create and review cleaning plans

[1009] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then approves the final schedule.

[1010] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[1011] Server: Sends the completed cleaning plan to the cleaning robot.

[1012] Controlling the cleaning robot

[1013] Cleaning robot: The cleaning robot will automatically start working at the specified time.

[1014] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, efficiently cleaning the target area.

[1015] Status monitoring and user notification

[1016] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[1017] Server: If an abnormality is detected, it will send an alert to the user. If the process is normal, it will send a notification that the cleaning is complete.

[1018] Gathering user feedback

[1019] On-device: After cleaning is complete, the user can check the results in the app and provide feedback if necessary.

[1020] Server: Feedback is collected and passed to the generative AI model, which then influences the creation of the next cleaning plan.

[1021] Specific examples

[1022] For example, if a user makes a request to "clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[1023] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[1024] The processing flow will be explained below.

[1025] Step 1:

[1026] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1027] Step 2:

[1028] Terminal: The input request is sent to the server as text data.

[1029] Step 3:

[1030] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[1031] Step 4:

[1032] Server: The generative AI model takes into account the request content and past cleaning history data to generate the optimal cleaning schedule.

[1033] Step 5:

[1034] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[1035] Step 6:

[1036] Terminal: The user can check the notified schedule and make any necessary changes. Once the changes are complete, the final schedule is confirmed.

[1037] Step 7:

[1038] Server: Based on the finalized schedule, the generative AI model creates a specific cleaning plan and sends it to the cleaning robot.

[1039] Step 8:

[1040] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[1041] Step 9:

[1042] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[1043] Step 10:

[1044] Server: The server monitors the status of the cleaning robot in real time, checking cleaning progress and battery level.

[1045] Step 11:

[1046] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[1047] Step 12:

[1048] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[1049] Step 13:

[1050] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[1051] Step 14:

[1052] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[1053] Step 15:

[1054] Server: Collects user feedback, provides it to the generative AI model, and reflects it in creating the next cleaning plan.

[1055] As described above, this system efficiently formulates cleaning plans based on user requests and optimally controls the cleaning robot, providing users with a convenient and reliable cleaning experience.

[1056] Example 1

[1057] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1058] Manually setting cleaning schedules and operating cleaning robots is time-consuming and difficult for users to clean effectively and efficiently. Furthermore, if an abnormality occurs during cleaning, it is difficult to immediately respond, and feedback on the cleaning results is not reflected in the next cleaning plan.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1060] In this invention, the server includes means for receiving requests from a user, means for analyzing the received requests, means for notifying the user's terminal of a schedule generated based on the analysis results, means for the user to modify and approve the schedule, means for controlling the cleaning robot according to the approved cleaning schedule, means for monitoring the status of the cleaning robot in real time, means for notifying the user of status information and abnormalities, and means for collecting feedback from the user and reflecting it in the next cleaning plan. This reduces the user's effort and enables effective and efficient cleaning. Furthermore, by being able to respond quickly to abnormalities when they occur and reflecting feedback in the next cleaning plan, the quality of cleaning can be improved.

[1061] "User" refers to an individual or organization that uses this system to operate a cleaning robot.

[1062] A "request" refers to a cleaning instruction or request that a user makes to the system via a smartphone or other device.

[1063] "Parsing" refers to the process of understanding the received request and converting it into an appropriate data format.

[1064] "Generative AI model" refers to an artificial intelligence model that generates optimal cleaning schedules and plans based on past data and current instructions.

[1065] A "schedule" refers to a timetable for a cleaning robot to clean at a specific time and place.

[1066] "Terminal" refers to a device such as a smartphone or PC that a user uses to access and operate the system.

[1067] A "cleaning robot" is a machine that moves automatically and cleans designated areas.

[1068] "Status" refers to real-time information such as the cleaning robot's current operating status and remaining battery level.

[1069] "Notification" refers to the act of sending information from a server to a user's terminal or the information itself.

[1070] "Feedback" refers to the evaluations and opinions provided by users regarding the cleaning results of the cleaning robot.

[1071] "Real-time" refers to a situation in which information processing and communication occur almost simultaneously, and data is reflected without delay.

[1072] This invention is a system that allows users to operate a cleaning robot from a smartphone or personal computer (PC) to clean efficiently and effectively. The system utilizes a generative AI model to provide an optimal cleaning schedule and cleaning plan based on the user's requests.

[1073] The user launches a dedicated application on their smartphone or PC and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday." The device receives this request and sends it to the server, which interprets and analyzes the request and retrieves the necessary data (for example, past cleaning history and room layout information) from a database.

[1074] The server passes the interpreted and analyzed requests to the generative AI model, which generates an optimal cleaning schedule. The generated schedule is then sent to the user's device. The user can check the schedule on their device and make any necessary changes. Once the changes are complete and the user approves them, the final cleaning plan is generated.

[1075] This final cleaning plan is sent from the server to the cleaning robot. The cleaning robot operates automatically at the designated time and cleans while monitoring its surroundings in real time using obstacle sensors and cameras. The server monitors the status of the cleaning robot in real time, checking the battery condition and cleaning progress. If an abnormality is detected, the server immediately sends an alert to the user's device, and if the cleaning is progressing normally, it sends a notification that the cleaning is complete.

[1076] After the cleaning is completed, the user can review the results on the app and provide feedback, which is then sent to the server via the device, where it is passed to the generative AI model to be used to plan the next cleaning.

[1077] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[1078] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[1079] Prompt Sentence Examples

[1080] Say, "Clean the living room at 10am on Saturday."

[1081] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1082] Step 1:

[1083] User: The user launches a dedicated smartphone app and inputs a request via voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1084] Operation:

[1085] The user speaks into the smartphone's microphone or enters a request via text.

[1086] Input: "Clean the living room on Saturday at 10 AM"

[1087] Output: Request as text data

[1088] Step 2:

[1089] Terminal: Send this text data to the server.

[1090] Operation:

[1091] The app converts the speech into text and generates text data.

[1092] Input: A voice or text request from the user

[1093] Output: HTTPS request to the server

[1094] Step 3:

[1095] Server: Interprets and analyzes the received request.

[1096] Operation:

[1097] The server analyzes the received text data and extracts the necessary information.

[1098] Input: Request as text data

[1099] Output: Structured data as a result of analysis

[1100] Step 4:

[1101] Server: Based on the analysis results, retrieves past cleaning history and house layout information from the database.

[1102] Operation:

[1103] The server queries the database to retrieve the required data.

[1104] Input: Structured data of analysis results

[1105] Output: Past cleaning history and house layout information

[1106] Step 5:

[1107] Server: Passes the acquired data and analysis results to the AI ​​model to generate an optimal cleaning schedule.

[1108] Operation:

[1109] A generative AI model calculates the schedule based on past data and requests.

[1110] Input: Past cleaning history, house layout information, structured data of analysis results

[1111] Output: Optimal cleaning schedule

[1112] Step 6:

[1113] Server: Notifies the user of the generated schedule.

[1114] Operation:

[1115] The server sends the schedule in JSON format to the user's device.

[1116] Enter: your optimal cleaning schedule

[1117] Output: Notification to user terminal

[1118] Step 7:

[1119] Terminal: The user checks the schedule and adjusts it if necessary.

[1120] Operation:

[1121] The app displays the schedule to the user and provides an interface for modification.

[1122] Input: Schedule as notification to user device

[1123] Output: Revised or approved schedule

[1124] Step 8:

[1125] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[1126] Operation:

[1127] A generative AI model generates a detailed cleaning plan based on the approved schedule.

[1128] Input: Approved Schedule

[1129] Output: A detailed cleaning plan

[1130] Step 9:

[1131] Server: Sends the completed cleaning plan to the cleaning robot.

[1132] Operation:

[1133] The server transmits the cleaning plan to the cleaning robot via communication.

[1134] Input: Specific cleaning plan

[1135] Output: Instructions for the cleaning robot

[1136] Step 10:

[1137] Cleaning robot: Automatically starts cleaning at the specified time.

[1138] Operation:

[1139] The cleaning robot will start moving according to the schedule and perform cleaning.

[1140] Input: Instructions for the cleaning robot

[1141] Output: Cleaning run

[1142] Step 11:

[1143] Server: Monitors the status of the cleaning robot in real time.

[1144] Operation:

[1145] The server receives data from the cleaning robot and checks its battery status and progress.

[1146] Input: Status data from the cleaning robot

[1147] Output: Real-time monitoring information

[1148] Step 12:

[1149] Server: If an abnormality is detected, an alert is sent immediately to the user's device.

[1150] Operation:

[1151] If an abnormality is detected, an alert message is sent to the user terminal.

[1152] Input: Anomaly detection status data

[1153] Output: Alert notification to user terminal

[1154] Step 13:

[1155] Server: If cleaning is completed successfully, send a cleaning completion notification to the user.

[1156] Operation:

[1157] The server sends a "cleaning completed" notification to the user terminal.

[1158] Input: Cleaning completion status data

[1159] Output: Completion notification to user terminal

[1160] Step 14:

[1161] On the device: Users can view the cleaning results and provide feedback on the app.

[1162] Operation:

[1163] The user enters a rating and comment on the app's feedback screen.

[1164] Input: Check cleaning results and enter feedback

[1165] Output: Feedback data

[1166] Step 15:

[1167] Server: Collects feedback and passes it to the generative AI model to help plan the next cleaning.

[1168] Operation:

[1169] The server analyzes the feedback data and applies the feedback to the generative AI model.

[1170] Input: Feedback data

[1171] Output: Improved cleaning plan

[1172] (Application example 1)

[1173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1174] Conventional cleaning robot management systems require users to input requests and check and modify schedules, which is time-consuming. Even when voice requests are accepted, subsequent processing is limited, making intuitive operation through visual devices difficult. Furthermore, the status of the cleaning robot and notifications in the event of an abnormality are often insufficient, resulting in a loss of user convenience.

[1175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1176] In this invention, the server includes means for receiving requests from a user, means for generating a cleaning schedule based on the received requests, means for controlling the cleaning robot according to the cleaning schedule, means for monitoring the status of the cleaning robot, means for notifying the user of the status information, means for collecting feedback from the user and reflecting it in the next cleaning plan, means for visually confirming and modifying the cleaning schedule through the user's visual device, and means for converting voice requests from the user into text and processing the requests. This allows the user to easily make requests through voice input and intuitively confirm and modify the cleaning schedule using the visual device in real time. Furthermore, since the status of the cleaning robot is monitored in real time and the user is immediately notified in the event of an abnormality, the cleaning robot can be managed more efficiently and effectively.

[1177] The "means for receiving a request from a user" refers to a device or software for receiving an operation instruction given by the user to the cleaning robot.

[1178] "Means for generating a cleaning schedule based on received requests" refers to an algorithm or program that analyzes user instructions and determines appropriate cleaning timings in accordance with those instructions.

[1179] The "means for controlling the cleaning robot in accordance with the cleaning schedule" is a control system for operating the cleaning robot in accordance with the generated schedule.

[1180] The "means for monitoring the status of the cleaning robot" is a monitoring system for obtaining real-time information such as the operating status and battery condition of the cleaning robot.

[1181] The "means for notifying the user of status information" is a notification system for informing the user of the current status or abnormalities of the cleaning robot.

[1182] The "means of collecting feedback from users and reflecting it in the next cleaning plan" is a data collection and analysis system that receives users' evaluations and requests and uses them in the next cleaning plan.

[1183] "Means for visually reviewing and modifying the cleaning schedule through a user's visual device" refers to an interface and software that allows a user to review and, if necessary, modify their cleaning plan using a visual device such as smart glasses or a head-mounted display.

[1184] The "means for converting voice requests from the user into text and processing the requests" is a voice recognition system that recognizes user instructions input by voice as text and processes them as control instructions for the cleaning robot.

[1185] This invention is a system that efficiently manages a cleaning robot using user voice instructions. By wearing a visual device such as smart glasses and inputting voice requests, the user can create and modify cleaning schedules and monitor the status of the cleaning robot in real time. Specific embodiments of this system are described below.

[1186] Hardware and software used

[1187] Smart glasses (e.g. Google Glass, Vuzix Blade)

[1188] Google Cloud Speech-to-Text API: A solution for converting voice input to text

[1189] Firebase Functions: Cloud functions for sending and processing requests

[1190] OpenAI GPT-4 model: Generative AI that analyzes requests and generates schedules and cleaning plans

[1191] iRobot API: An interface for controlling the cleaning robot

[1192] Firebase Realtime Database: A database for monitoring the status of the cleaning robot in real time.

[1193] Processing flow

[1194] Receiving and converting user requests

[1195] A user uses the smart glasses to input voice commands, such as "Clean the living room at 10 AM on Saturday," which are converted into text by the Google Cloud Speech-to-Text API through the smart glasses' microphone and sent to the server via Firebase Functions.

[1196] Generate a schedule

[1197] The server analyzes the received request and passes it to the OpenAI GPT-4 model. This generative AI model takes into account the request and past cleaning history to generate an optimal cleaning schedule. The generated schedule is visually displayed on the user's smart glasses.

[1198] Check and modify the schedule

[1199] The user can review the proposed schedule through the smart glasses and make any necessary changes, either by touchpad or voice input. Once the user finally approves the schedule, it is sent to the server and the cleaning plan is finalized.

[1200] Controlling the cleaning robot

[1201] The cleaning plan is sent to the cleaning robot via the iRobot API, and the cleaning robot starts cleaning at the designated time, checking its surroundings using obstacle sensors and cameras.

[1202] Status monitoring and notification

[1203] The status of the cleaning robot is monitored in real time using Firebase Realtime Database. If an abnormality is detected, the server sends an alert to the user's smart glasses, and if normal progress is made, a notification is sent when cleaning is complete.

[1204] Gathering and implementing feedback

[1205] After cleaning is complete, the user can check the results through the smart glasses and provide feedback, which is then sent back to the server and collected to be used to create the next cleaning plan.

[1206] Specific examples

[1207] For example, if a user provides a voice request such as "Clean the living room at 10 AM on Saturday," the following occurs:

[1208] 1. Voice command: "Clean the living room at 10 AM on Saturday."

[1209] 2. Text Transformation: "Clean the living room at 10 AM on Saturday."

[1210] 3. Send request: Send to server

[1211] 4. Analysis and generation: Generate schedules and cleaning plans using OpenAI GPT-4 models

[1212] 5. Notification and confirmation: The schedule is notified to the smart glasses and displayed as "Clean the living room at 10:00 AM on Saturday."

[1213] 6. Send cleaning plan: After approval, send it to the cleaning robot via iRobot API

[1214] 7. Real-time monitoring: The status of the cleaning robot is monitored in real time, and users are notified if an abnormality occurs.

[1215] This allows users to easily make requests through voice input and intuitively check and modify cleaning schedules in real time using visual devices.

[1216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1217] Step 1:

[1218] The user speaks their request

[1219] Users input voice requests through the smart glasses, such as "Clean the living room at 10 a.m. on Saturday," and this voice input is captured by the smart glasses' microphone.

[1220] Input: Voice request (e.g., "Clean the living room at 10 AM on Saturday")

[1221] Output: Audio data

[1222] Step 2:

[1223] Transcription of voice requests

[1224] The smart glasses send the captured voice data to the Google Cloud Speech-to-Text API, which analyzes the voice data and converts it into corresponding text data.

[1225] Input: Audio data

[1226] Output: Text data (e.g. "Clean the living room at 10 AM on Saturday")

[1227] Step 3:

[1228] Sending a text request to the server

[1229] The smart glasses then send the converted text data to the server via Firebase Functions, which includes details such as the date, time, and location specified by the user.

[1230] Input: Text data (e.g., "Clean the living room at 10 AM on Saturday")

[1231] Output: Request data sent to the server

[1232] Step 4:

[1233] Schedule generation using generative AI models

[1234] The server analyzes the received request data and passes it to the OpenAI GPT-4 model, which then generates an optimal cleaning schedule based on the request and the user's cleaning history.

[1235] Input: Request data (e.g., "Clean the living room at 10 AM on Saturday"), past cleaning history

[1236] Output: Optimal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[1237] Step 5:

[1238] Schedule notification and confirmation

[1239] The generated cleaning schedule is sent to the user's smart glasses via Firebase Functions, where the user can view the visual display of the schedule and modify it using the touchpad or voice input.

[1240] Input: Your ideal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[1241] Output: Schedule, confirmation or correction data sent to the user's smart glasses

[1242] Step 6:

[1243] Send revised schedule and approve

[1244] Once the user confirms and modifies the schedule and finally approves it, the modified schedule is sent back to the server, which then receives the schedule and finalizes the cleaning plan.

[1245] Input: Check or correct data

[1246] Output: Approved cleaning schedule

[1247] Step 7:

[1248] Sending cleaning plans to cleaning robots

[1249] The server sends the determined cleaning plan to the cleaning robot via the iRobot API, and the cleaning robot automatically starts working at the specified time.

[1250] Input: Approved cleaning schedule

[1251] Output: The cleaning plan sent to the cleaning robot

[1252] Step 8:

[1253] Real-time monitoring of cleaning robot status

[1254] The status of the cleaning robot is monitored in real time using the Firebase Realtime Database, where progress and battery status are continuously checked.

[1255] Input: Vacuum robot status information

[1256] Output: Real-time status data

[1257] Step 9:

[1258] Status information notification and abnormality detection

[1259] If an abnormality is detected, the server will send a real-time alert to the user's smart glasses. Under normal circumstances, a cleaning completion notification will be sent.

[1260] Input: Real-time status data, anomaly detection data

[1261] Output: Alert notified to user or cleaning completion notification

[1262] Step 10:

[1263] Collecting and incorporating user feedback

[1264] After the cleaning is completed, the user can check the cleaning results through the smart glasses and provide feedback, which will be sent to the server and reflected in the generative AI model's next plan.

[1265] Input: Feedback data

[1266] Output: Feedback collected, data reflected in next cleaning plan

[1267] This allows users to easily operate the cleaning robot through voice input and manage it in real time using a visual device.

[1268] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1269] The present invention is a system that recognizes a user's emotions and adjusts the cleaning schedule and cleaning plan based on those emotions to provide a more suitable cleaning experience for the user. This system is equipped with an emotion engine that recognizes the user's emotions, analyzes the user's emotional state, and reflects the analysis results in the generation of the next cleaning plan and notification content.

[1270] Program processing explanation

[1271] Receiving a user request

[1272] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1273] Terminal: This request is sent to the server.

[1274] Generate a schedule

[1275] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[1276] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[1277] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[1278] Analysis and adjustment by emotion engine

[1279] Device: The user indicates their emotions through the camera on their smartphone or PC, either through voice or facial expression. For example, the user may have a dissatisfied expression or an angry tone of voice.

[1280] Server: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state. If they express dissatisfaction, it is assumed that there is a problem with the cleaning schedule or content.

[1281] Server: Based on the user's emotional state, the generative AI model readjusts the cleaning schedule and plan to best reflect it.

[1282] Create and review cleaning plans

[1283] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then confirms the final schedule.

[1284] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[1285] Server: Sends the completed cleaning plan to the cleaning robot.

[1286] Controlling the cleaning robot

[1287] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[1288] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[1289] Status monitoring and user notification

[1290] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[1291] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[1292] Cleaning completion and notification

[1293] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[1294] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[1295] Gathering user feedback

[1296] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[1297] Server: User feedback is passed to the emotion engine, which analyzes the user's emotional state again. The analysis results are reflected in the generation of the next cleaning plan and notification content.

[1298] Specific examples

[1299] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process goes like this: The user enters the request, which is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is then notified to the user, who can review it and make any necessary adjustments. The emotion engine analyzes the user's emotions and readjusts the schedule and plans accordingly.

[1300] The final cleaning plan is sent to the cleaning robot, which starts cleaning at the specified time. The cleaning robot is monitored in real time, and if any problems occur, an alert is sent to the user. After the cleaning is completed, the user can check the results and provide feedback. This feedback is reflected in the generation of the next cleaning plan, and the emotion engine provides a more personalized cleaning experience for the user.

[1301] As a result, the system can take the user's emotions into account and provide a more flexible and personalized cleaning experience.

[1302] The processing flow will be explained below.

[1303] Step 1:

[1304] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1305] Step 2:

[1306] Terminal: The input request is sent to the server as text data.

[1307] Step 3:

[1308] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[1309] Step 4:

[1310] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[1311] Step 5:

[1312] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[1313] Step 6:

[1314] On the device, the user checks the schedule and makes any necessary changes. Once the changes are complete, the user confirms the final schedule.

[1315] Step 7:

[1316] Server: Based on the final schedule, the generative AI model creates a specific cleaning plan.

[1317] Step 8:

[1318] Server: The emotion engine analyzes the voice and facial expression data provided by the user through the application to identify the user's emotional state.

[1319] Step 9:

[1320] Server: The emotion engine makes necessary adjustments to the cleaning schedule and plan based on the results of emotion analysis. For example, if the user expresses dissatisfaction, the cleaning plan will be reevaluated to improve efficiency.

[1321] Step 10:

[1322] Server: Sends the completed cleaning plan to the cleaning robot.

[1323] Step 11:

[1324] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[1325] Step 12:

[1326] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, allowing it to clean efficiently.

[1327] Step 13:

[1328] Server: Monitors the status of the cleaning robot in real time, checking cleaning progress and battery condition.

[1329] Step 14:

[1330] Server: If an anomaly is detected, for example if the cleaning robot gets stuck on an obstacle, the server immediately sends an alert to the user.

[1331] Step 15:

[1332] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[1333] Step 16:

[1334] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[1335] Step 17:

[1336] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[1337] Step 18:

[1338] Server: The user feedback is passed to the emotion engine, which analyzes the user's emotional state again. For example, if the user is satisfied, the same plan will be offered next time.

[1339] Step 19:

[1340] Server: The results of the user's emotion analysis are reflected in the generation of the next cleaning plan, and the emotion engine adjusts it to provide a cleaning experience that is even more suited to the user.

[1341] As described above, this system efficiently formulates cleaning plans based on user requests and uses an emotion engine to analyze and reflect the user's emotional state, providing a flexible and personalized cleaning experience.

[1342] Example 2

[1343] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1344] Existing cleaning robots and related systems have difficulty providing optimal cleaning plans for users because they are unable to fully reflect user emotions and feedback. Furthermore, they lack systems for quickly responding to abnormalities that occur during the cleaning robot's operation. This can lead to an unsatisfactory user experience.

[1345] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1346] In this invention, the server includes a means for receiving requests from a user, a means for analyzing the received requests using a generative AI model to generate a cleaning schedule, and a means for recognizing the user's emotions and adjusting the cleaning schedule based on the user's emotional state. This allows for an optimal cleaning schedule that reflects the user's requests and emotions, enabling a more personalized cleaning experience for the user. It also enables rapid response through real-time monitoring and anomaly detection.

[1347] The "means for receiving requests from users" refers to a mechanism that allows the system to receive requests in voice or text format sent by users using smartphones or personal computer terminals.

[1348] "Means for analyzing using a generative AI model and generating a cleaning schedule" refers to a function that uses a generative AI model to analyze received user requests and automatically create an optimal cleaning schedule based on them.

[1349] "Means for recognizing the user's emotions and adjusting the cleaning schedule based on their emotional state" refers to a system that uses sensors such as cameras and microphones to analyze the user's emotions from their facial expressions and voice, and then optimizes the existing cleaning schedule based on the results of that analysis.

[1350] "Means for controlling the cleaning robot according to the adjusted cleaning schedule" means hardware and software mechanisms for directing and managing the operation of the cleaning robot according to the optimized cleaning schedule.

[1351] "Means for monitoring the status of the cleaning robot in real time" refers to a function that acquires and monitors the operating status, battery status, sensor information, etc. of the cleaning robot in real time.

[1352] The "means for notifying the user of status information" is a mechanism for notifying the user of the current cleaning status, whether or not there are any abnormalities, etc., based on status information monitored in real time.

[1353] "Means for collecting feedback from users and reflecting it in the next cleaning plan along with the results of the analysis of their emotional state" is a function that collects feedback provided by users and reflects it, including the results of the emotion analysis by the emotion engine, in optimizing the next cleaning schedule and cleaning plan.

[1354] This invention relates to a system that recognizes a user's emotions and adjusts cleaning schedules and plans based on them. The system receives requests from the user and generates an optimal cleaning schedule using a generative AI model. It then analyzes the user's emotional state using an emotion engine and readjusts the cleaning schedule based on the analysis results. It also has a function to monitor the operation of the cleaning robot and notify the user of the status in real time.

[1355] A user logs in to the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text. For example, a user may input a request such as "Clean the living room at 10:00 AM on Saturday." This request is then sent from the terminal to the server.

[1356] The server receives the request and analyzes it using a generative AI model. At this time, the voice data is converted into text and input as a prompt to the generative AI model. For example, a prompt such as, "Based on the user's request, please schedule cleaning of the living room for 10:00 AM on Saturday. Also, please consider past cleaning history data and propose the optimal plan" is given to the generative AI model.

[1357] The generative AI model generates an optimal cleaning schedule based on the request and past cleaning history data, and the server notifies the user's device. The user can review this schedule and make any necessary changes. Once changes are complete, the final schedule is confirmed.

[1358] The device captures the user's facial expressions and voice through a camera and microphone. The emotion engine on the server analyzes this data to identify the user's emotional state. It identifies emotions such as dissatisfaction or joy and adjusts the cleaning schedule and plan accordingly.

[1359] The determined cleaning schedule is sent from the server to the cleaning robot. The cleaning robot starts working at the designated time and begins cleaning the living room. The cleaning robot uses obstacle sensors and cameras to clean efficiently.

[1360] The server monitors the status of the cleaning robot in real time, checking its operation and battery condition. If an abnormality occurs, an alert is sent to the user immediately. When cleaning is complete, the server generates a "Cleaning Completed" notification and sends it to the user's device. The user can check this notification and view detailed cleaning results through the application.

[1361] Users can provide feedback on their cleaning results and input it within the application. This feedback is passed to the emotion engine on the server, which again analyzes the user's emotional state. The analysis results are reflected in the generation of the next cleaning plan and notifications, providing a more personalized cleaning experience.

[1362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1363] Step 1:

[1364] Input: A user logs into the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1365] Specific operation: After the user inputs a request, the device sends the request to the server. In the case of voice input, the device converts the voice data into text data.

[1366] Output: The input request is sent to the server in text format.

[1367] Step 2:

[1368] Input: The server receives the request from the device.

[1369] What happens: The server parses this request and prepares it to be input as a prompt to the generative AI model.

[1370] Data processing and calculation: The voice data is converted into text, and a prompt sentence is generated in a format suitable for the generative AI model.

[1371] Output: The generative AI model is given a prompt such as, "Based on the user's request, please schedule the living room cleaning for 10 AM on Saturday. Also, please take into account past cleaning history data and suggest the optimal plan."

[1372] Step 3:

[1373] Input: The prompt passed to the generative AI model from the server.

[1374] Specific operation: The generative AI model analyzes the request content based on the prompt text and generates an optimal cleaning schedule.

[1375] Data processing and calculation: The request details are compared with past cleaning history data to generate the optimal cleaning schedule.

[1376] Output: The generated optimal cleaning schedule is returned to the server.

[1377] Step 4:

[1378] Input: Optimal cleaning schedule from a generative AI model.

[1379] Specific operation: The server notifies the user's device of the generated cleaning schedule.

[1380] Output: A cleaning schedule notification is sent to the user's device.

[1381] Step 5:

[1382] Input: Cleaning schedule notification sent to the user's device.

[1383] What happens next: The user reviews the proposed schedule, makes any necessary adjustments, and then confirms the final schedule.

[1384] Output: The modified or confirmed schedule is sent to the server.

[1385] Step 6:

[1386] Input: Confirmed cleaning schedule.

[1387] Specific operation: The server sends the determined cleaning schedule to the cleaning robot.

[1388] Output: The cleaning schedule sent to the cleaning robot.

[1389] Step 7:

[1390] Input: The cleaning schedule sent to the cleaning robot.

[1391] Specific operation: The cleaning robot will automatically start working at the specified time and clean the living room.

[1392] Output: The operating status of the cleaning robot is sent to the server in real time.

[1393] Step 8:

[1394] Input: The operating status of the cleaning robot.

[1395] Specific operation: The server monitors the status of the cleaning robot in real time and sends an alert to the user if an abnormality occurs.

[1396] Output: The user is notified of the operation status of the cleaning robot and the monitoring results of the battery status.

[1397] Step 9:

[1398] Input: Notification that the robot vacuum cleaner has completed cleaning.

[1399] Specific operation: The server generates a notification that "cleaning is complete" and sends it to the user's device.

[1400] Output: A cleaning completion notification is sent to the user's device.

[1401] Step 10:

[1402] Input: Notification after cleaning is completed and cleaning result.

[1403] What it does: The user reviews the cleaning results and provides feedback within the app.

[1404] Output: Feedback is sent to the server and reflected in the next cleaning plan.

[1405] Step 11:

[1406] Input: User feedback.

[1407] Specific operation: The server's emotion engine analyzes the feedback and the user's emotional state and reflects it in the next cleaning plan.

[1408] Output: Emotional state analysis and feedback will be incorporated into the next cleaning plan.

[1409] (Application example 2)

[1410] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1411] Conventional work device control systems generate work schedules and control devices based on user requests, but do not consider the user's emotional state. As a result, they are unable to provide effective work plans when the user is dissatisfied or when a response tailored to a specific situation is required. Especially in brick-and-mortar stores, understanding the customer's emotional state leads to improved service, so responses that reflect emotions are required.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing and analyzing the user's emotions, means for adjusting the work schedule and work plan based on the analyzed emotional information, and means for collecting feedback from the user and reflecting it in the next work plan. This makes it possible to provide a more effective and personalized work plan that corresponds to the user's emotional state.

[1413] "User" means any person or entity that uses the System.

[1414] A "request" refers to a specific request for work or service sent by a user.

[1415] "Work Schedule" refers to the timeline and sequence of work to be created based on received requests.

[1416] "Working device" refers to hardware or robotic systems that perform work based on a specified schedule.

[1417] "Status" indicates status information such as the current working state of the working device, progress, and whether or not there is a problem.

[1418] "Emotion" refers to a user's mental state or mood, and is generally analyzed from facial expressions, tone of voice, etc.

[1419] "Analysis" refers to the process of deriving specific information or results from input data.

[1420] "Adjusting work schedules and plans" refers to modifying and optimizing existing schedules and plans based on analyzed data.

[1421] "Feedback" refers to the evaluations and impressions collected from users after completing a task.

[1422] 1. System Overview

[1423] The system for implementing this invention captures the user's emotions and adjusts the behavior of a customer service robot based on those emotions. The user sends a request using a PC or smartphone, and the customer service robot takes appropriate action in response to the request. This system analyzes the user's facial expressions and voice to provide optimal service in real time.

[1424] 2. Hardware and Software Configuration

[1425] The server implements this system using the following hardware and software:

[1426] Customer service robot: A robotic device that handles customer service in brick-and-mortar stores

[1427] Camera: A device for capturing the user's facial expressions

[1428] Microphone: A device for capturing the user's voice.

[1429] OpenCV: Image processing library, used for face detection and facial expression analysis

[1430] PyTorch: A machine learning library for running emotion recognition models

[1431] Google Cloud Speech-to-Text: Speech recognition API, converts user speech into text

[1432] paho.mqtt: MQTT communication module, sending emotion data in real time

[1433] 3. Processing Flow

[1434] The server receives requests from users and generates and adjusts work schedules based on the requests by using the following means:

[1435] First, the system captures the user's facial expressions using a camera and analyzes their emotions using OpenCV and PyTorch. It also collects the user's voice using a microphone and converts it into text using Google Cloud Speech-to-Text. This data is then sent to a server, where an optimal work schedule is generated using a generative AI model.

[1436] 4. Generative AI Model and Prompts

[1437] Based on the generated emotion data, the generative AI model builds optimized work schedules and plans.

[1438] As an example of a prompt sentence, the following sentence is fed into the generative AI model:

[1439] User emotions detected: [negative, positive]

[1440] Based on this prompt, the server generates the optimal response based on the user's emotions and reflects it in the customer service robot.

[1441] 5. Specific Examples

[1442] For example, if a user "looks busy" in a physical store, a customer service robot can be instructed to respond quickly by saying, "Let me know if there's anything I can help you with." This is achieved by a generative AI model that analyzes the user's emotions from the prompt text and generates an appropriate response.

[1443] In this way, the system can grasp the user's emotional state in real time and provide appropriate services, thereby significantly improving the user experience.

[1444] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1445] Step 1:

[1446] The terminal receives a request from the user. The input is a request (voice or text) sent by the user using a smartphone or PC. This request is sent to the server.

[1447] Step 2:

[1448] The server analyzes the received request. As input, it receives the user's textual request, which is passed to the generative AI model. Here, the audio data is converted to text using Google Cloud Speech-to-Text. As output, the request content is analyzed and the data required for further processing is generated.

[1449] Step 3:

[1450] The server uses a generative AI model to generate an optimal work schedule, taking into account the request content and past data. The inputs are the analyzed request data and past work history data. This data is then input into the generative AI model to process the data and generate an optimal schedule. The output is the generated work schedule.

[1451] Step 4:

[1452] The server sends the generated work schedule to the terminal. The input is the generated work schedule. The output is a confirmation notification sent to the user's terminal. The user checks this and makes any necessary corrections.

[1453] Step 5:

[1454] The device uses a camera and microphone to recognize and analyze the user's emotions. The input is the user's facial expressions and voice. The facial expression data captured by the camera is analyzed using OpenCV and PyTorch, and the voice data is converted to text using Google Cloud Speech-to-Text. The output is the analyzed emotion data.

[1455] Step 6:

[1456] The server adjusts the work schedule and work plan using a generative AI model based on the analyzed emotion data. The inputs are the analyzed emotion data and the generated work schedule. For data calculations, the emotion data is input into the generative AI model as a prompt sentence, and the work schedule is adjusted. An example of a prompt sentence is "User emotions detected: [negative, positive]". The output is an adjusted work schedule.

[1457] Step 7:

[1458] The server creates a final work plan based on the adjusted work schedule and sends it to the terminal. The input is the adjusted work schedule. The output is the final work plan sent to the user's terminal.

[1459] Step 8:

[1460] The terminal sends the final work plan to the customer service robot and has it execute it. The input is the final work plan sent from the server. The output is that the customer service robot starts the specified work.

[1461] Step 9:

[1462] The server monitors the status of the customer service robot in real time and sends an alert to the user if an abnormality is detected. The input is the status information of the customer service robot. The output is an alert sent to the user's device if an abnormality is detected.

[1463] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1464] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1465] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1466] [Fourth embodiment]

[1467] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1468] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1469] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1470] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1471] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1472] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1473] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1474] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1475] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1478] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1479] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1480] The system is designed to enable users to operate the cleaning robot from a smartphone or PC terminal to clean efficiently and effectively, and utilizes generative AI models to provide optimal cleaning schedules and plans based on user requests.

[1481] Program processing explanation

[1482] Receiving a user request

[1483] Device: The user launches the application and enters a request via voice or text, such as "Clean the living room at 10 AM on Saturday."

[1484] Terminal: This request is sent to the server.

[1485] Generate a schedule

[1486] Server: The server parses the received request and passes it to the generative AI model.

[1487] Server: The generative AI model takes into account the request and past cleaning history to generate an optimal schedule.

[1488] Server: The generated schedule is notified to the user and they are asked to confirm it.

[1489] Create and review cleaning plans

[1490] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then approves the final schedule.

[1491] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[1492] Server: Sends the completed cleaning plan to the cleaning robot.

[1493] Controlling the cleaning robot

[1494] Cleaning robot: The cleaning robot will automatically start working at the specified time.

[1495] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, efficiently cleaning the target area.

[1496] Status monitoring and user notification

[1497] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[1498] Server: If an abnormality is detected, it will send an alert to the user. If the process is normal, it will send a notification that the cleaning is complete.

[1499] Gathering user feedback

[1500] On-device: After cleaning is complete, the user can check the results in the app and provide feedback if necessary.

[1501] Server: Feedback is collected and passed to the generative AI model, which then influences the creation of the next cleaning plan.

[1502] Specific examples

[1503] For example, if a user makes a request to "clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[1504] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[1505] The processing flow will be explained below.

[1506] Step 1:

[1507] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1508] Step 2:

[1509] Terminal: The input request is sent to the server as text data.

[1510] Step 3:

[1511] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[1512] Step 4:

[1513] Server: The generative AI model takes into account the request content and past cleaning history data to generate the optimal cleaning schedule.

[1514] Step 5:

[1515] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[1516] Step 6:

[1517] Terminal: The user can check the notified schedule and make any necessary changes. Once the changes are complete, the final schedule is confirmed.

[1518] Step 7:

[1519] Server: Based on the finalized schedule, the generative AI model creates a specific cleaning plan and sends it to the cleaning robot.

[1520] Step 8:

[1521] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[1522] Step 9:

[1523] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[1524] Step 10:

[1525] Server: The server monitors the status of the cleaning robot in real time, checking cleaning progress and battery level.

[1526] Step 11:

[1527] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[1528] Step 12:

[1529] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[1530] Step 13:

[1531] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[1532] Step 14:

[1533] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[1534] Step 15:

[1535] Server: Collects user feedback, provides it to the generative AI model, and reflects it in creating the next cleaning plan.

[1536] As described above, this system efficiently formulates cleaning plans based on user requests and optimally controls the cleaning robot, providing users with a convenient and reliable cleaning experience.

[1537] Example 1

[1538] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1539] Manually setting cleaning schedules and operating cleaning robots is time-consuming and difficult for users to clean effectively and efficiently. Furthermore, if an abnormality occurs during cleaning, it is difficult to immediately respond, and feedback on the cleaning results is not reflected in the next cleaning plan.

[1540] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1541] In this invention, the server includes means for receiving requests from a user, means for analyzing the received requests, means for notifying the user's terminal of a schedule generated based on the analysis results, means for the user to modify and approve the schedule, means for controlling the cleaning robot according to the approved cleaning schedule, means for monitoring the status of the cleaning robot in real time, means for notifying the user of status information and abnormalities, and means for collecting feedback from the user and reflecting it in the next cleaning plan. This reduces the user's effort and enables effective and efficient cleaning. Furthermore, by being able to respond quickly to abnormalities when they occur and reflecting feedback in the next cleaning plan, the quality of cleaning can be improved.

[1542] "User" refers to an individual or organization that uses this system to operate a cleaning robot.

[1543] A "request" refers to a cleaning instruction or request that a user makes to the system via a smartphone or other device.

[1544] "Parsing" refers to the process of understanding the received request and converting it into an appropriate data format.

[1545] "Generative AI model" refers to an artificial intelligence model that generates optimal cleaning schedules and plans based on past data and current instructions.

[1546] A "schedule" refers to a timetable for a cleaning robot to clean at a specific time and place.

[1547] "Terminal" refers to a device such as a smartphone or PC that a user uses to access and operate the system.

[1548] A "cleaning robot" is a machine that moves automatically and cleans designated areas.

[1549] "Status" refers to real-time information such as the cleaning robot's current operating status and remaining battery level.

[1550] "Notification" refers to the act of sending information from a server to a user's terminal or the information itself.

[1551] "Feedback" refers to the evaluations and opinions provided by users regarding the cleaning results of the cleaning robot.

[1552] "Real-time" refers to a situation in which information processing and communication occur almost simultaneously, and data is reflected without delay.

[1553] This invention is a system that allows users to operate a cleaning robot from a smartphone or personal computer (PC) to clean efficiently and effectively. The system utilizes a generative AI model to provide an optimal cleaning schedule and cleaning plan based on the user's requests.

[1554] The user launches a dedicated application on their smartphone or PC and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday." The device receives this request and sends it to the server, which interprets and analyzes the request and retrieves the necessary data (for example, past cleaning history and room layout information) from a database.

[1555] The server passes the interpreted and analyzed requests to the generative AI model, which generates an optimal cleaning schedule. The generated schedule is then sent to the user's device. The user can check the schedule on their device and make any necessary changes. Once the changes are complete and the user approves them, the final cleaning plan is generated.

[1556] This final cleaning plan is sent from the server to the cleaning robot. The cleaning robot operates automatically at the designated time and cleans while monitoring its surroundings in real time using obstacle sensors and cameras. The server monitors the status of the cleaning robot in real time, checking the battery condition and cleaning progress. If an abnormality is detected, the server immediately sends an alert to the user's device, and if the cleaning is progressing normally, it sends a notification that the cleaning is complete.

[1557] After the cleaning is completed, the user can review the results on the app and provide feedback, which is then sent to the server via the device, where it is passed to the generative AI model to be used to plan the next cleaning.

[1558] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process would proceed as follows: The user enters the request on their smartphone and it is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is notified to the user, who can review it and make any necessary adjustments. Finally, the cleaning plan is finalized and sent to the cleaning robot. At 10 AM on Saturday, the cleaning robot will automatically start cleaning the living room. If any abnormalities occur during cleaning, the server will send an alert to the user. Once the cleaning is complete, the user is notified and can review the results and provide feedback.

[1559] This system allows users to easily operate the cleaning robot and clean effectively, and by linking it with a generative AI model, it optimizes cleaning schedules and plans to achieve efficient cleaning.

[1560] Prompt Sentence Examples

[1561] Say, "Clean the living room at 10am on Saturday."

[1562] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1563] Step 1:

[1564] User: The user launches a dedicated smartphone app and inputs a request via voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1565] Operation:

[1566] The user speaks into the smartphone's microphone or enters a request via text.

[1567] Input: "Clean the living room on Saturday at 10 AM"

[1568] Output: Request as text data

[1569] Step 2:

[1570] Terminal: Send this text data to the server.

[1571] Operation:

[1572] The app converts the speech into text and generates text data.

[1573] Input: A voice or text request from the user

[1574] Output: HTTPS request to the server

[1575] Step 3:

[1576] Server: Interprets and analyzes the received request.

[1577] Operation:

[1578] The server analyzes the received text data and extracts the necessary information.

[1579] Input: Request as text data

[1580] Output: Structured data as a result of analysis

[1581] Step 4:

[1582] Server: Based on the analysis results, retrieves past cleaning history and house layout information from the database.

[1583] Operation:

[1584] The server queries the database to retrieve the required data.

[1585] Input: Structured data of analysis results

[1586] Output: Past cleaning history and house layout information

[1587] Step 5:

[1588] Server: Passes the acquired data and analysis results to the AI ​​model to generate an optimal cleaning schedule.

[1589] Operation:

[1590] A generative AI model calculates the schedule based on past data and requests.

[1591] Input: Past cleaning history, house layout information, structured data of analysis results

[1592] Output: Optimal cleaning schedule

[1593] Step 6:

[1594] Server: Notifies the user of the generated schedule.

[1595] Operation:

[1596] The server sends the schedule in JSON format to the user's device.

[1597] Enter: your optimal cleaning schedule

[1598] Output: Notification to user terminal

[1599] Step 7:

[1600] Terminal: The user checks the schedule and adjusts it if necessary.

[1601] Operation:

[1602] The app displays the schedule to the user and provides an interface for modification.

[1603] Input: Schedule as notification to user device

[1604] Output: Revised or approved schedule

[1605] Step 8:

[1606] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[1607] Operation:

[1608] A generative AI model generates a detailed cleaning plan based on the approved schedule.

[1609] Input: Approved Schedule

[1610] Output: A detailed cleaning plan

[1611] Step 9:

[1612] Server: Sends the completed cleaning plan to the cleaning robot.

[1613] Operation:

[1614] The server transmits the cleaning plan to the cleaning robot via communication.

[1615] Input: Specific cleaning plan

[1616] Output: Instructions for the cleaning robot

[1617] Step 10:

[1618] Cleaning robot: Automatically starts cleaning at the specified time.

[1619] Operation:

[1620] The cleaning robot will start moving according to the schedule and perform cleaning.

[1621] Input: Instructions for the cleaning robot

[1622] Output: Cleaning run

[1623] Step 11:

[1624] Server: Monitors the status of the cleaning robot in real time.

[1625] Operation:

[1626] The server receives data from the cleaning robot and checks its battery status and progress.

[1627] Input: Status data from the cleaning robot

[1628] Output: Real-time monitoring information

[1629] Step 12:

[1630] Server: If an abnormality is detected, an alert is sent immediately to the user's device.

[1631] Operation:

[1632] If an abnormality is detected, an alert message is sent to the user terminal.

[1633] Input: Anomaly detection status data

[1634] Output: Alert notification to user terminal

[1635] Step 13:

[1636] Server: If cleaning is completed successfully, send a cleaning completion notification to the user.

[1637] Operation:

[1638] The server sends a "cleaning completed" notification to the user terminal.

[1639] Input: Cleaning completion status data

[1640] Output: Completion notification to user terminal

[1641] Step 14:

[1642] On the device: Users can view the cleaning results and provide feedback on the app.

[1643] Operation:

[1644] The user enters a rating and comment on the app's feedback screen.

[1645] Input: Check cleaning results and enter feedback

[1646] Output: Feedback data

[1647] Step 15:

[1648] Server: Collects feedback and passes it to the generative AI model to help plan the next cleaning.

[1649] Operation:

[1650] The server analyzes the feedback data and applies the feedback to the generative AI model.

[1651] Input: Feedback data

[1652] Output: Improved cleaning plan

[1653] (Application example 1)

[1654] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1655] Conventional cleaning robot management systems require users to input requests and check and modify schedules, which is time-consuming. Even when voice requests are accepted, subsequent processing is limited, making intuitive operation through visual devices difficult. Furthermore, the status of the cleaning robot and notifications in the event of an abnormality are often insufficient, resulting in a loss of user convenience.

[1656] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1657] In this invention, the server includes means for receiving requests from a user, means for generating a cleaning schedule based on the received requests, means for controlling the cleaning robot according to the cleaning schedule, means for monitoring the status of the cleaning robot, means for notifying the user of the status information, means for collecting feedback from the user and reflecting it in the next cleaning plan, means for visually confirming and modifying the cleaning schedule through the user's visual device, and means for converting voice requests from the user into text and processing the requests. This allows the user to easily make requests through voice input and intuitively confirm and modify the cleaning schedule using the visual device in real time. Furthermore, since the status of the cleaning robot is monitored in real time and the user is immediately notified in the event of an abnormality, the cleaning robot can be managed more efficiently and effectively.

[1658] The "means for receiving a request from a user" refers to a device or software for receiving an operation instruction given by the user to the cleaning robot.

[1659] "Means for generating a cleaning schedule based on received requests" refers to an algorithm or program that analyzes user instructions and determines appropriate cleaning timings in accordance with those instructions.

[1660] The "means for controlling the cleaning robot in accordance with the cleaning schedule" is a control system for operating the cleaning robot in accordance with the generated schedule.

[1661] The "means for monitoring the status of the cleaning robot" is a monitoring system for obtaining real-time information such as the operating status and battery condition of the cleaning robot.

[1662] The "means for notifying the user of status information" is a notification system for informing the user of the current status or abnormalities of the cleaning robot.

[1663] The "means of collecting feedback from users and reflecting it in the next cleaning plan" is a data collection and analysis system that receives users' evaluations and requests and uses them in the next cleaning plan.

[1664] "Means for visually reviewing and modifying the cleaning schedule through a user's visual device" refers to an interface and software that allows a user to review and, if necessary, modify their cleaning plan using a visual device such as smart glasses or a head-mounted display.

[1665] The "means for converting voice requests from the user into text and processing the requests" is a voice recognition system that recognizes user instructions input by voice as text and processes them as control instructions for the cleaning robot.

[1666] This invention is a system that efficiently manages a cleaning robot using user voice instructions. By wearing a visual device such as smart glasses and inputting voice requests, the user can create and modify cleaning schedules and monitor the status of the cleaning robot in real time. Specific embodiments of this system are described below.

[1667] Hardware and software used

[1668] Smart glasses (e.g. Google Glass, Vuzix Blade)

[1669] Google Cloud Speech-to-Text API: A solution for converting voice input to text

[1670] Firebase Functions: Cloud functions for sending and processing requests

[1671] OpenAI GPT-4 model: Generative AI that analyzes requests and generates schedules and cleaning plans

[1672] iRobot API: An interface for controlling the cleaning robot

[1673] Firebase Realtime Database: A database for monitoring the status of the cleaning robot in real time.

[1674] Processing flow

[1675] Receiving and converting user requests

[1676] A user uses the smart glasses to input voice commands, such as "Clean the living room at 10 AM on Saturday," which are converted into text by the Google Cloud Speech-to-Text API through the smart glasses' microphone and sent to the server via Firebase Functions.

[1677] Generate a schedule

[1678] The server analyzes the received request and passes it to the OpenAI GPT-4 model. This generative AI model takes into account the request and past cleaning history to generate an optimal cleaning schedule. The generated schedule is visually displayed on the user's smart glasses.

[1679] Check and modify the schedule

[1680] The user can review the proposed schedule through the smart glasses and make any necessary changes, either by touchpad or voice input. Once the user finally approves the schedule, it is sent to the server and the cleaning plan is finalized.

[1681] Controlling the cleaning robot

[1682] The cleaning plan is sent to the cleaning robot via the iRobot API, and the cleaning robot starts cleaning at the designated time, checking its surroundings using obstacle sensors and cameras.

[1683] Status monitoring and notification

[1684] The status of the cleaning robot is monitored in real time using Firebase Realtime Database. If an abnormality is detected, the server sends an alert to the user's smart glasses, and if normal progress is made, a notification is sent when cleaning is complete.

[1685] Gathering and implementing feedback

[1686] After cleaning is complete, the user can check the results through the smart glasses and provide feedback, which is then sent back to the server and collected to be used to create the next cleaning plan.

[1687] Specific examples

[1688] For example, if a user provides a voice request such as "Clean the living room at 10 AM on Saturday," the following occurs:

[1689] 1. Voice command: "Clean the living room at 10 AM on Saturday."

[1690] 2. Text Transformation: "Clean the living room at 10 AM on Saturday."

[1691] 3. Send request: Send to server

[1692] 4. Analysis and generation: Generate schedules and cleaning plans using OpenAI GPT-4 models

[1693] 5. Notification and confirmation: The schedule is notified to the smart glasses and displayed as "Clean the living room at 10:00 AM on Saturday."

[1694] 6. Send cleaning plan: After approval, send it to the cleaning robot via iRobot API

[1695] 7. Real-time monitoring: The status of the cleaning robot is monitored in real time, and users are notified if an abnormality occurs.

[1696] This allows users to easily make requests through voice input and intuitively check and modify cleaning schedules in real time using visual devices.

[1697] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1698] Step 1:

[1699] The user speaks their request

[1700] Users input voice requests through the smart glasses, such as "Clean the living room at 10 a.m. on Saturday," and this voice input is captured by the smart glasses' microphone.

[1701] Input: Voice request (e.g., "Clean the living room at 10 AM on Saturday")

[1702] Output: Audio data

[1703] Step 2:

[1704] Transcription of voice requests

[1705] The smart glasses send the captured voice data to the Google Cloud Speech-to-Text API, which analyzes the voice data and converts it into corresponding text data.

[1706] Input: Audio data

[1707] Output: Text data (e.g. "Clean the living room at 10 AM on Saturday")

[1708] Step 3:

[1709] Sending a text request to the server

[1710] The smart glasses then send the converted text data to the server via Firebase Functions, which includes details such as the date, time, and location specified by the user.

[1711] Input: Text data (e.g., "Clean the living room at 10 AM on Saturday")

[1712] Output: Request data sent to the server

[1713] Step 4:

[1714] Schedule generation using generative AI models

[1715] The server analyzes the received request data and passes it to the OpenAI GPT-4 model, which then generates an optimal cleaning schedule based on the request and the user's cleaning history.

[1716] Input: Request data (e.g., "Clean the living room at 10 AM on Saturday"), past cleaning history

[1717] Output: Optimal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[1718] Step 5:

[1719] Schedule notification and confirmation

[1720] The generated cleaning schedule is sent to the user's smart glasses via Firebase Functions, where the user can view the visual display of the schedule and modify it using the touchpad or voice input.

[1721] Input: Your ideal cleaning schedule (e.g., "Clean the living room at 10 AM on Saturday")

[1722] Output: Schedule, confirmation or correction data sent to the user's smart glasses

[1723] Step 6:

[1724] Send revised schedule and approve

[1725] Once the user confirms and modifies the schedule and finally approves it, the modified schedule is sent back to the server, which then receives the schedule and finalizes the cleaning plan.

[1726] Input: Check or correct data

[1727] Output: Approved cleaning schedule

[1728] Step 7:

[1729] Sending cleaning plans to cleaning robots

[1730] The server sends the determined cleaning plan to the cleaning robot via the iRobot API, and the cleaning robot automatically starts working at the specified time.

[1731] Input: Approved cleaning schedule

[1732] Output: The cleaning plan sent to the cleaning robot

[1733] Step 8:

[1734] Real-time monitoring of cleaning robot status

[1735] The status of the cleaning robot is monitored in real time using the Firebase Realtime Database, where progress and battery status are continuously checked.

[1736] Input: Vacuum robot status information

[1737] Output: Real-time status data

[1738] Step 9:

[1739] Status information notification and abnormality detection

[1740] If an abnormality is detected, the server will send a real-time alert to the user's smart glasses. Under normal circumstances, a cleaning completion notification will be sent.

[1741] Input: Real-time status data, anomaly detection data

[1742] Output: Alert notified to user or cleaning completion notification

[1743] Step 10:

[1744] Collecting and incorporating user feedback

[1745] After the cleaning is completed, the user can check the cleaning results through the smart glasses and provide feedback, which will be sent to the server and reflected in the generative AI model's next plan.

[1746] Input: Feedback data

[1747] Output: Feedback collected, data reflected in next cleaning plan

[1748] This allows users to easily operate the cleaning robot through voice input and manage it in real time using a visual device.

[1749] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1750] The present invention is a system that recognizes a user's emotions and adjusts the cleaning schedule and cleaning plan based on those emotions to provide a more suitable cleaning experience for the user. This system is equipped with an emotion engine that recognizes the user's emotions, analyzes the user's emotional state, and reflects the analysis results in the generation of the next cleaning plan and notification content.

[1751] Program processing explanation

[1752] Receiving a user request

[1753] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1754] Terminal: This request is sent to the server.

[1755] Generate a schedule

[1756] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[1757] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[1758] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[1759] Analysis and adjustment by emotion engine

[1760] Device: The user indicates their emotions through the camera on their smartphone or PC, either through voice or facial expression. For example, the user may have a dissatisfied expression or an angry tone of voice.

[1761] Server: The emotion engine analyzes the user's voice and facial expressions to identify their emotional state. If they express dissatisfaction, it is assumed that there is a problem with the cleaning schedule or content.

[1762] Server: Based on the user's emotional state, the generative AI model readjusts the cleaning schedule and plan to best reflect it.

[1763] Create and review cleaning plans

[1764] Terminal: The user reviews the proposed schedule, makes any necessary modifications, and then confirms the final schedule.

[1765] Server: Based on the approved schedule, the generative AI model creates a specific cleaning plan.

[1766] Server: Sends the completed cleaning plan to the cleaning robot.

[1767] Controlling the cleaning robot

[1768] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[1769] Cleaning robot: The cleaning robot uses obstacle sensors and cameras to check its surroundings while cleaning, allowing it to clean efficiently.

[1770] Status monitoring and user notification

[1771] Server: Monitors the status of the cleaning robot in real time, checking the cleaning progress and battery status.

[1772] Server: If an anomaly is detected or a problem occurs, such as getting caught on an obstacle, the server immediately sends an alert to the user.

[1773] Cleaning completion and notification

[1774] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[1775] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[1776] Gathering user feedback

[1777] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[1778] Server: User feedback is passed to the emotion engine, which analyzes the user's emotional state again. The analysis results are reflected in the generation of the next cleaning plan and notification content.

[1779] Specific examples

[1780] For example, if a user requests "Clean the living room at 10 AM on Saturday," the process goes like this: The user enters the request, which is sent to the server. The server uses a generative AI model to analyze the request and generate an optimal cleaning schedule. This schedule is then notified to the user, who can review it and make any necessary adjustments. The emotion engine analyzes the user's emotions and readjusts the schedule and plans accordingly.

[1781] The final cleaning plan is sent to the cleaning robot, which starts cleaning at the specified time. The cleaning robot is monitored in real time, and if any problems occur, an alert is sent to the user. After the cleaning is completed, the user can check the results and provide feedback. This feedback is reflected in the generation of the next cleaning plan, and the emotion engine provides a more personalized cleaning experience for the user.

[1782] As a result, the system can take the user's emotions into account and provide a more flexible and personalized cleaning experience.

[1783] The processing flow will be explained below.

[1784] Step 1:

[1785] Device: The user logs into the cleaning robot application using a smartphone or PC terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1786] Step 2:

[1787] Terminal: The input request is sent to the server as text data.

[1788] Step 3:

[1789] Server: The server parses the incoming request and passes it to a generative AI model, which converts the speech data into text so that the request can be accurately understood.

[1790] Step 4:

[1791] Server: The generative AI model generates an optimal cleaning schedule based on the request and historical cleaning data.

[1792] Step 5:

[1793] Server: The generated cleaning schedule is sent to the user's device as a notification for confirmation.

[1794] Step 6:

[1795] On the device, the user checks the schedule and makes any necessary changes. Once the changes are complete, the user confirms the final schedule.

[1796] Step 7:

[1797] Server: Based on the final schedule, the generative AI model creates a specific cleaning plan.

[1798] Step 8:

[1799] Server: The emotion engine analyzes the voice and facial expression data provided by the user through the application to identify the user's emotional state.

[1800] Step 9:

[1801] Server: The emotion engine makes necessary adjustments to the cleaning schedule and plan based on the results of emotion analysis. For example, if the user expresses dissatisfaction, the cleaning plan will be reevaluated to improve efficiency.

[1802] Step 10:

[1803] Server: Sends the completed cleaning plan to the cleaning robot.

[1804] Step 11:

[1805] Cleaning robot: The cleaning robot will automatically start working at the specified time and begin cleaning the living room.

[1806] Step 12:

[1807] Cleaning robot: Using obstacle sensors and cameras, the robot checks its surroundings while cleaning, allowing it to clean efficiently.

[1808] Step 13:

[1809] Server: Monitors the status of the cleaning robot in real time, checking cleaning progress and battery condition.

[1810] Step 14:

[1811] Server: If an anomaly is detected, for example if the cleaning robot gets stuck on an obstacle, the server immediately sends an alert to the user.

[1812] Step 15:

[1813] Server: When cleaning is complete, the server generates a notification to the user saying "Cleaning is complete" and sends it to the user's device.

[1814] Step 16:

[1815] On the device: The user checks the cleaning result notification and displays the detailed cleaning results through the application.

[1816] Step 17:

[1817] Terminal: The user provides feedback on the cleaning results and enters it within the application.

[1818] Step 18:

[1819] Server: The user feedback is passed to the emotion engine, which analyzes the user's emotional state again. For example, if the user is satisfied, the same plan will be offered next time.

[1820] Step 19:

[1821] Server: The results of the user's emotion analysis are reflected in the generation of the next cleaning plan, and the emotion engine adjusts it to provide a cleaning experience that is even more suited to the user.

[1822] As described above, this system efficiently formulates cleaning plans based on user requests and uses an emotion engine to analyze and reflect the user's emotional state, providing a flexible and personalized cleaning experience.

[1823] Example 2

[1824] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1825] Existing cleaning robots and related systems have difficulty providing optimal cleaning plans for users because they are unable to fully reflect user emotions and feedback. Furthermore, they lack systems for quickly responding to abnormalities that occur during the cleaning robot's operation. This can lead to an unsatisfactory user experience.

[1826] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1827] In this invention, the server includes a means for receiving requests from a user, a means for analyzing the received requests using a generative AI model to generate a cleaning schedule, and a means for recognizing the user's emotions and adjusting the cleaning schedule based on the user's emotional state. This allows for an optimal cleaning schedule that reflects the user's requests and emotions, enabling a more personalized cleaning experience for the user. It also enables rapid response through real-time monitoring and anomaly detection.

[1828] The "means for receiving requests from users" refers to a mechanism that allows the system to receive requests in voice or text format sent by users using smartphones or personal computer terminals.

[1829] "Means for analyzing using a generative AI model and generating a cleaning schedule" refers to a function that uses a generative AI model to analyze received user requests and automatically create an optimal cleaning schedule based on them.

[1830] "Means for recognizing the user's emotions and adjusting the cleaning schedule based on their emotional state" refers to a system that uses sensors such as cameras and microphones to analyze the user's emotions from their facial expressions and voice, and then optimizes the existing cleaning schedule based on the results of that analysis.

[1831] "Means for controlling the cleaning robot according to the adjusted cleaning schedule" means hardware and software mechanisms for directing and managing the operation of the cleaning robot according to the optimized cleaning schedule.

[1832] "Means for monitoring the status of the cleaning robot in real time" refers to a function that acquires and monitors the operating status, battery status, sensor information, etc. of the cleaning robot in real time.

[1833] The "means for notifying the user of status information" is a mechanism for notifying the user of the current cleaning status, whether or not there are any abnormalities, etc., based on status information monitored in real time.

[1834] "Means for collecting feedback from users and reflecting it in the next cleaning plan along with the results of the analysis of their emotional state" is a function that collects feedback provided by users and reflects it, including the results of the emotion analysis by the emotion engine, in optimizing the next cleaning schedule and cleaning plan.

[1835] This invention relates to a system that recognizes a user's emotions and adjusts cleaning schedules and plans based on them. The system receives requests from the user and generates an optimal cleaning schedule using a generative AI model. It then analyzes the user's emotional state using an emotion engine and readjusts the cleaning schedule based on the analysis results. It also has a function to monitor the operation of the cleaning robot and notify the user of the status in real time.

[1836] A user logs in to the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text. For example, a user may input a request such as "Clean the living room at 10:00 AM on Saturday." This request is then sent from the terminal to the server.

[1837] The server receives the request and analyzes it using a generative AI model. At this time, the voice data is converted into text and input as a prompt to the generative AI model. For example, a prompt such as, "Based on the user's request, please schedule cleaning of the living room for 10:00 AM on Saturday. Also, please consider past cleaning history data and propose the optimal plan" is given to the generative AI model.

[1838] The generative AI model generates an optimal cleaning schedule based on the request and past cleaning history data, and the server notifies the user's device. The user can review this schedule and make any necessary changes. Once changes are complete, the final schedule is confirmed.

[1839] The device captures the user's facial expressions and voice through a camera and microphone. The emotion engine on the server analyzes this data to identify the user's emotional state. It identifies emotions such as dissatisfaction or joy and adjusts the cleaning schedule and plan accordingly.

[1840] The determined cleaning schedule is sent from the server to the cleaning robot. The cleaning robot starts working at the designated time and begins cleaning the living room. The cleaning robot uses obstacle sensors and cameras to clean efficiently.

[1841] The server monitors the status of the cleaning robot in real time, checking its operation and battery condition. If an abnormality occurs, an alert is sent to the user immediately. When cleaning is complete, the server generates a "Cleaning Completed" notification and sends it to the user's device. The user can check this notification and view detailed cleaning results through the application.

[1842] Users can provide feedback on their cleaning results and input it within the application. This feedback is passed to the emotion engine on the server, which again analyzes the user's emotional state. The analysis results are reflected in the generation of the next cleaning plan and notifications, providing a more personalized cleaning experience.

[1843] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1844] Step 1:

[1845] Input: A user logs into the cleaning robot application using a smartphone or personal computer terminal and inputs a request by voice or text, such as "Clean the living room at 10:00 AM on Saturday."

[1846] Specific operation: After the user inputs a request, the device sends the request to the server. In the case of voice input, the device converts the voice data into text data.

[1847] Output: The input request is sent to the server in text format.

[1848] Step 2:

[1849] Input: The server receives the request from the device.

[1850] What happens: The server parses this request and prepares it to be input as a prompt to the generative AI model.

[1851] Data processing and calculation: The voice data is converted into text, and a prompt sentence is generated in a format suitable for the generative AI model.

[1852] Output: The generative AI model is given a prompt such as, "Based on the user's request, please schedule the living room cleaning for 10 AM on Saturday. Also, please take into account past cleaning history data and suggest the optimal plan."

[1853] Step 3:

[1854] Input: The prompt passed to the generative AI model from the server.

[1855] Specific operation: The generative AI model analyzes the request content based on the prompt text and generates an optimal cleaning schedule.

[1856] Data processing and calculation: The request details are compared with past cleaning history data to generate the optimal cleaning schedule.

[1857] Output: The generated optimal cleaning schedule is returned to the server.

[1858] Step 4:

[1859] Input: Optimal cleaning schedule from a generative AI model.

[1860] Specific operation: The server notifies the user's device of the generated cleaning schedule.

[1861] Output: A cleaning schedule notification is sent to the user's device.

[1862] Step 5:

[1863] Input: Cleaning schedule notification sent to the user's device.

[1864] What happens next: The user reviews the proposed schedule, makes any necessary adjustments, and then confirms the final schedule.

[1865] Output: The modified or confirmed schedule is sent to the server.

[1866] Step 6:

[1867] Input: Confirmed cleaning schedule.

[1868] Specific operation: The server sends the determined cleaning schedule to the cleaning robot.

[1869] Output: The cleaning schedule sent to the cleaning robot.

[1870] Step 7:

[1871] Input: The cleaning schedule sent to the cleaning robot.

[1872] Specific operation: The cleaning robot will automatically start working at the specified time and clean the living room.

[1873] Output: The operating status of the cleaning robot is sent to the server in real time.

[1874] Step 8:

[1875] Input: The operating status of the cleaning robot.

[1876] Specific operation: The server monitors the status of the cleaning robot in real time and sends an alert to the user if an abnormality occurs.

[1877] Output: The user is notified of the operation status of the cleaning robot and the monitoring results of the battery status.

[1878] Step 9:

[1879] Input: Notification that the robot vacuum cleaner has completed cleaning.

[1880] Specific operation: The server generates a notification that "cleaning is complete" and sends it to the user's device.

[1881] Output: A cleaning completion notification is sent to the user's device.

[1882] Step 10:

[1883] Input: Notification after cleaning is completed and cleaning result.

[1884] What it does: The user reviews the cleaning results and provides feedback within the app.

[1885] Output: Feedback is sent to the server and reflected in the next cleaning plan.

[1886] Step 11:

[1887] Input: User feedback.

[1888] Specific operation: The server's emotion engine analyzes the feedback and the user's emotional state and reflects it in the next cleaning plan.

[1889] Output: Emotional state analysis and feedback will be incorporated into the next cleaning plan.

[1890] (Application example 2)

[1891] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1892] Conventional work device control systems generate work schedules and control devices based on user requests, but do not consider the user's emotional state. As a result, they are unable to provide effective work plans when the user is dissatisfied or when a response tailored to a specific situation is required. Especially in brick-and-mortar stores, understanding the customer's emotional state leads to improved service, so responses that reflect emotions are required.

[1893] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing and analyzing the user's emotions, means for adjusting the work schedule and work plan based on the analyzed emotional information, and means for collecting feedback from the user and reflecting it in the next work plan. This makes it possible to provide a more effective and personalized work plan that corresponds to the user's emotional state.

[1894] "User" means any person or entity that uses the System.

[1895] A "request" refers to a specific request for work or service sent by a user.

[1896] "Work Schedule" refers to the timeline and sequence of work to be created based on received requests.

[1897] "Working device" refers to hardware or robotic systems that perform work based on a specified schedule.

[1898] "Status" indicates status information such as the current working state of the working device, progress, and whether or not there is a problem.

[1899] "Emotion" refers to a user's mental state or mood, and is generally analyzed from facial expressions, tone of voice, etc.

[1900] "Analysis" refers to the process of deriving specific information or results from input data.

[1901] "Adjusting work schedules and plans" refers to modifying and optimizing existing schedules and plans based on analyzed data.

[1902] "Feedback" refers to the evaluations and impressions collected from users after completing a task.

[1903] 1. System Overview

[1904] The system for implementing this invention captures the user's emotions and adjusts the behavior of a customer service robot based on those emotions. The user sends a request using a PC or smartphone, and the customer service robot takes appropriate action in response to the request. This system analyzes the user's facial expressions and voice to provide optimal service in real time.

[1905] 2. Hardware and Software Configuration

[1906] The server implements this system using the following hardware and software:

[1907] Customer service robot: A robotic device that handles customer service in brick-and-mortar stores

[1908] Camera: A device for capturing the user's facial expressions

[1909] Microphone: A device for capturing the user's voice.

[1910] OpenCV: Image processing library, used for face detection and facial expression analysis

[1911] PyTorch: A machine learning library for running emotion recognition models

[1912] Google Cloud Speech-to-Text: Speech recognition API, converts user speech into text

[1913] paho.mqtt: MQTT communication module, sending emotion data in real time

[1914] 3. Processing Flow

[1915] The server receives requests from users and generates and adjusts work schedules based on the requests by using the following means:

[1916] First, the system captures the user's facial expressions using a camera and analyzes their emotions using OpenCV and PyTorch. It also collects the user's voice using a microphone and converts it into text using Google Cloud Speech-to-Text. This data is then sent to a server, where an optimal work schedule is generated using a generative AI model.

[1917] 4. Generative AI Model and Prompts

[1918] Based on the generated emotion data, the generative AI model builds optimized work schedules and plans.

[1919] As an example of a prompt sentence, the following sentence is fed into the generative AI model:

[1920] User emotions detected: [negative, positive]

[1921] Based on this prompt, the server generates the optimal response based on the user's emotions and reflects it in the customer service robot.

[1922] 5. Specific Examples

[1923] For example, if a user "looks busy" in a physical store, a customer service robot can be instructed to respond quickly by saying, "Let me know if there's anything I can help you with." This is achieved by a generative AI model that analyzes the user's emotions from the prompt text and generates an appropriate response.

[1924] In this way, the system can grasp the user's emotional state in real time and provide appropriate services, thereby significantly improving the user experience.

[1925] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1926] Step 1:

[1927] The terminal receives a request from the user. The input is a request (voice or text) sent by the user using a smartphone or PC. This request is sent to the server.

[1928] Step 2:

[1929] The server analyzes the received request. As input, it receives the user's textual request, which is passed to the generative AI model. Here, the audio data is converted to text using Google Cloud Speech-to-Text. As output, the request content is analyzed and the data required for further processing is generated.

[1930] Step 3:

[1931] The server uses a generative AI model to generate an optimal work schedule, taking into account the request content and past data. The inputs are the analyzed request data and past work history data. This data is then input into the generative AI model to process the data and generate an optimal schedule. The output is the generated work schedule.

[1932] Step 4:

[1933] The server sends the generated work schedule to the terminal. The input is the generated work schedule. The output is a confirmation notification sent to the user's terminal. The user checks this and makes any necessary corrections.

[1934] Step 5:

[1935] The device uses a camera and microphone to recognize and analyze the user's emotions. The input is the user's facial expressions and voice. The facial expression data captured by the camera is analyzed using OpenCV and PyTorch, and the voice data is converted to text using Google Cloud Speech-to-Text. The output is the analyzed emotion data.

[1936] Step 6:

[1937] The server adjusts the work schedule and work plan using a generative AI model based on the analyzed emotion data. The inputs are the analyzed emotion data and the generated work schedule. For data calculations, the emotion data is input into the generative AI model as a prompt sentence, and the work schedule is adjusted. An example of a prompt sentence is "User emotions detected: [negative, positive]". The output is an adjusted work schedule.

[1938] Step 7:

[1939] The server creates a final work plan based on the adjusted work schedule and sends it to the terminal. The input is the adjusted work schedule. The output is the final work plan sent to the user's terminal.

[1940] Step 8:

[1941] The terminal sends the final work plan to the customer service robot and has it execute it. The input is the final work plan sent from the server. The output is that the customer service robot starts the specified work.

[1942] Step 9:

[1943] The server monitors the status of the customer service robot in real time and sends an alert to the user if an abnormality is detected. The input is the status information of the customer service robot. The output is an alert sent to the user's device if an abnormality is detected.

[1944] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1945] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1946] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1947] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1948] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1949] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1950] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1951] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1952] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1953] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1954] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1955] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1956] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1957] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1958] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1959] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1960] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1961] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1962] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1963] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1964] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1965] The following is further disclosed regarding the above embodiment.

[1966] (Claim 1)

[1967] means for receiving a request from a user;

[1968] means for generating a cleaning schedule based on the received requests;

[1969] means for controlling the cleaning robot according to a cleaning schedule;

[1970] a means for monitoring the status of the cleaning robot;

[1971] means for notifying a user of status information;

[1972] The system includes a means to collect feedback from users and incorporate it into the next cleaning plan.

[1973] (Claim 2)

[1974] 10. The system of claim 1, further comprising means for analyzing received user requests by voice or text.

[1975] (Claim 3)

[1976] 2. The system according to claim 1, further comprising means for monitoring the operation of the cleaning robot and sending an alert to a user when an abnormality is detected.

[1977] "Example 1"

[1978] (Claim 1)

[1979] means for receiving a request from a user;

[1980] a means for analyzing the received request;

[1981] a means for notifying a user terminal of a schedule generated based on the analysis results;

[1982] A means for users to modify and approve schedules;

[1983] means for controlling the cleaning robot in accordance with an approved cleaning schedule;

[1984] a means for monitoring the status of the cleaning robot in real time;

[1985] means for notifying a user of status information and abnormalities;

[1986] A system that includes a means to collect feedback from users and incorporate it into the next cleaning plan.

[1987] (Claim 2)

[1988] The system of claim 1, further comprising means for analyzing a received user request by voice or text and passing the analysis results to the generative AI model.

[1989] (Claim 3)

[1990] 2. The system according to claim 1, further comprising means for monitoring the operation of the cleaning robot and sending an alert to a user when an abnormality is detected.

[1991] "Application Example 1"

[1992] (Claim 1)

[1993] means for receiving a request from a user;

[1994] means for generating a cleaning schedule based on the received requests;

[1995] means for controlling the cleaning robot according to a cleaning schedule;

[1996] a means for monitoring the status of the cleaning robot;

[1997] means for notifying a user of status information;

[1998] A way to collect feedback from users and reflect it in the next cleaning plan,

[1999] a means for visually reviewing and modifying the cleaning schedule through a user's visual device;

[2000] The system includes a means for converting a voice request from a user into text and processing the request.

[2001] (Claim 2)

[2002] 10. The system of claim 1, further comprising means for analyzing received user requests by voice or text.

[2003] (Claim 3)

[2004] 2. The system according to claim 1, further comprising means for monitoring the operation of the cleaning robot and sending an alert to a user when an abnormality is detected.

[2005] "Example 2: Combining Emotion Engines"

[2006] (Claim 1)

[2007] means for receiving a request from a user;

[2008] A means for analyzing the received request using a generative AI model and generating a cleaning schedule;

[2009] means for recognizing a user's emotions and adjusting a cleaning schedule based on the user's emotional state;

[2010] means for controlling the cleaning robot according to the adjusted cleaning schedule;

[2011] a means for monitoring the status of the cleaning robot in real time;

[2012] means for notifying a user of status information;

[2013] The system includes a means for collecting feedback from users and incorporating it into the next cleaning plan along with the results of an analysis of their emotional state.

[2014] (Claim 2)

[2015] The system of claim 1, further comprising means for analyzing received user requests by voice or text and generating an optimal cleaning schedule using a generative AI model.

[2016] (Claim 3)

[2017] 2. The system according to claim 1, further comprising means for monitoring the operation of the cleaning robot and sending an alert to a user when an abnormality is detected.

[2018] "Application example 2 when combining emotion engines"

[2019] (Claim 1)

[2020] means for receiving a request from a user;

[2021] means for generating a work schedule based on the received requests;

[2022] a means for controlling the work device according to a work schedule;

[2023] means for monitoring the status of the work implement;

[2024] means for notifying a user of status information;

[2025] means for recognizing and analyzing user emotions;

[2026] A means for adjusting work schedules and work plans based on the analyzed emotional information;

[2027] A system that includes a means of gathering user feedback and incorporating it into future work plans.

[2028] (Claim 2)

[2029] 10. The system of claim 1, further comprising means for analyzing received user requests by voice or text.

[2030] (Claim 3)

[2031] 2. The system according to claim 1, further comprising means for monitoring the operation of the work device and sending an alert to a user when an abnormality is detected. [Explanation of symbols]

[2032] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving a request from a user; means for generating a cleaning schedule based on the received requests; means for controlling the cleaning robot according to a cleaning schedule; a means for monitoring the status of the cleaning robot; means for notifying a user of status information; The system includes a means to collect feedback from users and incorporate it into the next cleaning plan.

2. 10. The system of claim 1, further comprising means for analyzing received user requests by voice or text.

3. The system according to claim 1 , further comprising means for monitoring the operation of the cleaning robot and sending an alert to a user when an abnormality is detected.

Images