System

Abstract

A system is provided.SOLUTION: An autonomous mobile cleaning device equipped with a generative artificial intelligence model and controlled remotely from a user, comprising: means for analyzing a cleaning command received from a user terminal and initiating a corresponding cleaning operation; means for reporting a current cleaning status and location information in response to a status check request from the user terminal; and means for storing a corresponding cleaning area and generating an efficient schedule.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current autonomous mobile cleaning devices often lack the means for users to flexibly and efficiently control them from a remote location. Furthermore, current devices have difficulty responding to users' specific needs and questions in natural language, and users have limited means to understand the device's status in real time. As a result, users are unable to fully utilize the device's functions, and may not achieve the desired results. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing an autonomous mobile cleaning device equipped with a generative artificial intelligence model. Specifically, the device includes a means for analyzing cleaning commands from a user terminal and initiating the corresponding cleaning operation, and a means for reporting the current cleaning status and location information in response to a status check request from the user terminal. It also includes a means for storing the corresponding cleaning area and generating an efficient schedule. Furthermore, the device can provide an optimal cleaning plan based on the user's schedule and address specific stains or rooms. The device also has a function for answering questions in natural language using the generative artificial intelligence model, making it easier for users to understand the status of the device in real time. This allows users to flexibly and efficiently control the cleaning device from a remote location.

[0006] A "generative artificial intelligence model" is a type of artificial intelligence that includes algorithms that perform natural language processing based on user input and generate appropriate responses and instructions.

[0007] An "autonomous mobile cleaning device" refers to a device that can automatically move around within a certain area and clean.

[0008] A "user terminal" is a device operated by a user, and includes a smartphone, tablet, computer, etc.

[0009] A "cleaning command" refers to an input by a user to instruct a cleaning device to perform a specific action.

[0010] A "cleaning operation" refers to a cleaning action that the cleaning device performs based on instructions from the user.

[0011] A "status check request" refers to a request sent by a user to check the current status and location information of the cleaning device.

[0012] "Cleaning status" is information that indicates what the cleaning device is currently doing, such as "cleaning" or "standby."

[0013] "Location information" is data that indicates the current physical location of the cleaning device.

[0014] "Cleaning area" refers to the area that the cleaning device is to clean, including a specific room or location.

[0015] "Generating a schedule" refers to the cleaning device determining the optimal cleaning time and sequence and creating an action plan based on that.

[0016] "Ability to answer questions in natural language" refers to the ability of a generative artificial intelligence model to understand and return appropriate responses to questions or instructions from users in natural language. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This system is based on an autonomous mobile cleaning device equipped with a generative artificial intelligence model, and is also composed of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[0039] Components

[0040] 1. Cleaning device

[0041] The cleaning device is a piece of hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[0042] 2. User Device

[0043] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[0044] 3. Server

[0045] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[0046] System Operation Overview

[0047] 1. User sends command

[0048] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[0049] 2. Receiving and parsing commands

[0050] The user terminal transmits the input command to the server.

[0051] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[0052] 3. Sending commands to the cleaning device

[0053] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[0054] Once the cleaning device receives the command, it will begin cleaning the designated area.

[0055] 4. Status Check and Reporting

[0056] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[0057] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[0058] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[0059] Specific examples

[0060] Example 1: Cleaning the living room

[0061] User: Type "Clean the living room" into their smartphone.

[0062] Terminal: Sends input to the server.

[0063] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[0064] Cleaning device: Receive instructions and start cleaning the living room.

[0065] Server: Notifies the user device that cleaning has started.

[0066] Device: Displays to the user "Cleaning of the living room has begun."

[0067] Example 2: Checking the cleaning status

[0068] User: Type "Tell me how the cleaning is going" on your smartphone.

[0069] Device: Sends a status check request to the server.

[0070] Server: Transmits the request to the cleaning device.

[0071] Cleaning device: Sends current status and location information back to the server.

[0072] Server: Returns the received information to the user terminal.

[0073] Device: Shows the user "You're currently cleaning the living room."

[0074] This system allows users to operate cleaning devices flexibly and efficiently even from remote locations and check their status in real time, greatly improving the convenience of cleaning work.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the kitchen."

[0078] Step 2:

[0079] The terminal receives the user's input and sends a request of the form "command: clean area kitchen" to the server.

[0080] Step 3:

[0081] The server receives the request and extracts the command part ("clean area kitchen") through the handle_user_request method.

[0082] Step 4:

[0083] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[0084] Step 5:

[0085] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[0086] Step 6:

[0087] A generative artificial intelligence model analyzes the command "clean area kitchen" and generates the response "Cleaning kitchen."

[0088] Step 7:

[0089] The cleaning device sends the generated response to the server.

[0090] Step 8:

[0091] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Kitchen cleaning has started."

[0092] Step 9:

[0093] The terminal displays the response received from the server on the screen and notifies the user.

[0094] Step 10:

[0095] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[0096] Step 11:

[0097] The device sends a request called "status" to the server.

[0098] Step 12:

[0099] The server receives the status check request and invokes the report_status method of the cleaning device.

[0100] Step 13:

[0101] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., kitchen) to the server via the report_status method.

[0102] Step 14:

[0103] The server returns the received status information to the user terminal.

[0104] Step 15:

[0105] The terminal displays the status information received from the server on the screen and notifies the user, for example, "The kitchen is currently being cleaned."

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] Conventional autonomous mobile cleaning devices have had issues such as difficulty for users to control the cleaning device efficiently from a remote location, insufficient real-time confirmation of the cleaning status, insufficient status reporting, difficulty in providing optimal cleaning plans for specific stains or rooms, and difficulty in responding to questions from users in natural language.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes a means for analyzing cleaning commands received from a user-operated device and initiating the corresponding cleaning operation, a means for reporting the current cleaning status and location information in response to a status check request from the device, a means for storing the corresponding cleaning area and generating an efficient schedule, a means for incorporating a generative artificial intelligence model and responding to questions from the user in natural language, and a means for communicating data between the user device, the cleaning device, and the server. This allows users to efficiently control the cleaning device and check its status in real time, even from remote locations. It also provides optimal cleaning plans for specific stains or rooms and enables question and answering in natural language, greatly improving the convenience of cleaning work.

[0111] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new information and responses based on data, and is particularly used in natural language processing and image generation.

[0112] An "autonomous mobile cleaning device" is a device that has the function of cleaning while autonomously moving around a specific area, and performs various operations based on instructions from a user.

[0113] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC, and is a means for communicating with a cleaning device or a server.

[0114] The "server" is a central control device that mediates communication between the user terminal and the cleaning device, and processes and analyzes various requests.

[0115] A "cleaning command" is an instruction that a user inputs from a terminal and sends to the cleaning device, and includes instructions to start cleaning or to clean a specific area.

[0116] A "status confirmation request" is a request sent from a terminal by a user to check the current cleaning status and location information of the cleaning device.

[0117] An "efficient schedule" is an optimal cleaning plan generated based on the area to be cleaned and the user's schedule, and is designed to maximize the efficiency and effectiveness of cleaning.

[0118] The "function to answer questions in natural language" is a function that uses a generative artificial intelligence model to generate appropriate responses to questions or instructions from the user in natural language and responds.

[0119] "Data communication" is the process of exchanging information between different devices over a communication network.

[0120] The present invention is a system that mainly comprises an autonomous mobile cleaning device equipped with a generative artificial intelligence model, a user terminal, and a server. The details of each component and the operation of the entire system are explained below.

[0121] Components

[0122] 1. Autonomous mobile cleaning device

[0123] The cleaning device is hardware that can move around autonomously and clean. Using a built-in generative artificial intelligence model (e.g., OpenAI GPT-3), it can respond to cleaning commands and questions from the user in natural language. It also uses sensors (e.g., LIDAR and infrared sensors) to understand the surrounding environment and plan an efficient cleaning route.

[0124] 2. User Device

[0125] The user terminal is a device operated by the user, such as a smartphone, tablet, or PC. The user can issue cleaning commands and check the status through a dedicated application or web interface. Communication is via Wi-Fi or LTE.

[0126] 3. Server

[0127] The server is a central control device that mediates communication between the user device and the cleaning device and processes various requests. It analyzes the received cleaning commands and determines and instructs the specific cleaning operations. The server also reports the current status and location information of the cleaning device to the user device.

[0128] System Operation Overview

[0129] Entering and sending commands

[0130] Users use a device such as a smartphone to input and send cleaning commands, such as "Clean the living room," into a dedicated app or web interface.

[0131] Receiving and parsing commands

[0132] The user terminal transmits the input command to the server.

[0133] The server analyzes the received command using a natural language processing engine (e.g., Google Cloud Natural Language API), inputs specific instructions such as "clean the living room" as a prompt sentence to the generative AI model, and obtains the generated detailed cleaning plan.

[0134] Sending commands to cleaning devices

[0135] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[0136] The cleaning device receives instructions and begins cleaning the designated area (e.g., living room), using built-in generative AI models and sensors to plan and execute an efficient cleaning route.

[0137] Checking and reporting status

[0138] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[0139] The user terminal sends this request to the server.

[0140] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[0141] The cleaning device sends its current status and location information back to the server.

[0142] The server returns the received information to the user terminal.

[0143] The user terminal displays to the user, "Currently cleaning the living room."

[0144] Specific operation example

[0145] Example 1: Cleaning the living room

[0146] The user types "clean the living room" into their smartphone.

[0147] The terminal sends this command to the server.

[0148] The server receives the command, analyzes it, and inputs the "clean the living room" instruction into the generative AI model. Example prompt: "What is the best route to efficiently clean the living room?"

[0149] The server transmits specific instructions to the cleaning device based on the generated cleaning plan.

[0150] The vacuum cleaner receives instructions and begins cleaning the living room, using sensors and generative AI models to clean efficiently.

[0151] Example 2: Checking the cleaning status

[0152] The user types "Tell me about the cleaning status" on their smartphone.

[0153] The terminal sends a status confirmation request to the server.

[0154] The server communicates the request to the cleaning device.

[0155] The cleaning device sends its current status and location information back to the server.

[0156] The server returns the information to the user terminal.

[0157] The device displays to the user, "Currently cleaning the living room."

[0158] With this mechanism, the system of the present invention significantly improves user convenience and enables efficient and flexible cleaning work.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Specific processing steps of the program

[0161] Step 1:

[0162] The user enters a command

[0163] Users use a dedicated app on their smartphone or tablet to input cleaning commands such as "clean the living room."

[0164] Input: User's cleaning command (e.g., "Clean the living room")

[0165] Output: The entered command is recorded on the user's terminal and prepared for transmission.

[0166] Step 2:

[0167] Sending commands from the user terminal to the server

[0168] The device sends the entered cleaning command to a server via the internet, using Wi-Fi or LTE for communication.

[0169] Input: Cleaning command entered by the user

[0170] Output: Cleaning command is sent to the server

[0171] Step 3:

[0172] The server receives the command

[0173] The server receives a cleaning command sent from a user terminal.

[0174] Input: Cleaning command sent from user terminal

[0175] Output: The received cleaning command is queued in the server.

[0176] Step 4:

[0177] The server parses the command

[0178] The server analyzes the received cleaning command using a natural language processing engine (e.g., Google Cloud Natural Language API).

[0179] Input: Cleaning command received

[0180] Data processing: A natural language processing engine is used to analyze the meaning of commands and convert them into specific cleaning instructions.

[0181] Output: Specific cleaning instructions as a result of the analysis (e.g., "Start cleaning the living room")

[0182] Step 5:

[0183] The server sends a prompt to the generative AI model

[0184] Based on the analysis results, the server sends prompts to a generative AI model (e.g., OpenAI GPT-3) to generate a detailed cleaning plan.

[0185] Input: Specific cleaning instructions as analysis results

[0186] Data Computation: Generative AI model generates optimal cleaning route and plan based on prompts

[0187] Output: Generated cleaning route and plan

[0188] Step 6:

[0189] The server sends commands to the cleaning device.

[0190] The server instructs the cleaning device on specific cleaning areas and operations based on the generated cleaning route and plan.

[0191] Input: Generated cleaning route and plan

[0192] Output: Specific cleaning instructions are sent to the cleaning device

[0193] Step 7:

[0194] The cleaning device starts cleaning

[0195] Based on the instructions received, the cleaning device begins cleaning the designated area (e.g., the living room), planning and executing an efficient cleaning route using built-in sensors and generative AI models.

[0196] Input: Specific cleaning instructions from the server

[0197] Data calculation: Cleaning is performed while dynamically correcting the actual cleaning route based on sensor information.

[0198] Output: Dust and dirt in the area is removed

[0199] Step 8:

[0200] User submits a status check request

[0201] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[0202] Input: Status Check Request

[0203] Output: The request is logged on the user's terminal and ready to be sent.

[0204] Step 9:

[0205] The user device sends a request to the server

[0206] The terminal sends this request to the server.

[0207] Input: User status check request

[0208] Output: The request is transmitted to the server

[0209] Step 10:

[0210] Server requests status

[0211] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[0212] Input: User status check request

[0213] Output: The current cleaning status and location information request are sent to the cleaning device.

[0214] Step 11:

[0215] The cleaning device sends status information back to the server

[0216] The cleaning device sends its current status and location information back to the server.

[0217] Input: Status information request from the server

[0218] Data calculation: Collects current cleaning progress and location information

[0219] Output: Sending collected status and location information

[0220] Step 12:

[0221] The server sends status information to the user terminal.

[0222] The server returns the received information to the user terminal.

[0223] Input: Status and location information from the cleaning device

[0224] Output: Information is transmitted to the user terminal

[0225] Step 13:

[0226] The user terminal displays the status

[0227] The terminal displays to the user, "Currently cleaning the living room."

[0228] Input: Status and location information from the server

[0229] Output: Status information is displayed to the user

[0230] This allows the user to operate the cleaning device efficiently and in real time even from a remote location and check the current cleaning status.

[0231] (Application example 1)

[0232] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0233] Modern self-driving vehicles have advanced autonomous driving capabilities, but cleaning the interior and exterior of the vehicle and performing regular maintenance require manual labor, which is a hassle for users. Furthermore, diagnosing the vehicle's condition and identifying repair areas requires specialized knowledge, which is inconvenient for users. There is a need to solve these problems and provide a system that efficiently automates the cleaning and maintenance of self-driving vehicles.

[0234] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0235] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status confirmation request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, means for automatically cleaning the interior and exterior of the vehicle and diagnosing the vehicle status, and means for scheduling cleaning maintenance and automatically identifying and reporting areas requiring repairs. This automates vehicle cleaning and maintenance work, allowing users to flexibly manage these tasks from a remote location.

[0236] A "generative artificial intelligence model" is an artificial intelligence technique that can generate new information and content from data.

[0237] An "autonomous mobile cleaning device" is a hardware device that can clean while moving around on its own.

[0238] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[0239] A "cleaning command" is an instruction or command given by a user to a cleaning device.

[0240] A "status confirmation request" is a request made by the user to confirm the current status and location information of the cleaning device.

[0241] A "cleaning zone" is a particular area that a cleaning device is to clean.

[0242] The "means for generating a schedule" is a function that determines the time and order for the cleaning device to perform cleaning work efficiently.

[0243] "Automatic cleaning inside and outside the vehicle" is a function that automatically cleans the inside and outside of the vehicle.

[0244] "Means for diagnosing the condition of the vehicle" refers to a function that checks the condition of each part and system of the vehicle and detects abnormalities.

[0245] "Cleaning and maintenance scheduling" is a function that allows you to create a schedule for systematically cleaning and maintaining your vehicle.

[0246] The "means for automatically identifying and reporting parts requiring repair" is a function for automatically detecting parts of a vehicle that are faulty and reporting the same to the user.

[0247] This invention is a system that automates cleaning and maintenance work for self-driving vehicles using an autonomous mobile cleaning device equipped with a generative artificial intelligence model. The system is composed of a user terminal, a server, and the autonomous mobile cleaning device.

[0248] Components

[0249] 1. Autonomous mobile cleaning device

[0250] The cleaning device is a hardware device that moves autonomously and cleans the inside and outside of a vehicle. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[0251] 2. User Device

[0252] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. Using this terminal, the user can remotely issue cleaning commands and check the status.

[0253] 3. Server

[0254] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[0255] System Operation Overview

[0256] 1. User sends command

[0257] Users input and send cleaning commands such as "clean the interior of the car" or "perform a maintenance check" using smart glasses or a smartphone.

[0258] 2. Receiving and parsing commands

[0259] The server receives commands sent from the user terminal, analyzes the contents of the commands, and then identifies cleaning areas and diagnostic points inside and outside the vehicle.

[0260] 3. Sending commands to the cleaning device

[0261] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving the instructions, the cleaning device begins cleaning the designated area. At the same time, it also performs a vehicle condition diagnosis.

[0262] 4. Status Check and Reporting

[0263] When a user wants to check the current status of the cleaning device, the user sends a command such as "Tell me the current situation" from the user terminal. The server transmits this request to the cleaning device, collects the current cleaning status and location information, and reports it to the user terminal.

[0264] Hardware and software used

[0265] Hardware:

[0266] Autonomous mobile cleaning device mounted on a self-driving vehicle

[0267] Vehicle condition diagnostic sensors

[0268] software:

[0269] Python

[0270] Flask (server-side request processing)

[0271] Generative AI model

[0272] Specific examples

[0273] 1. Specific scenario

[0274] Sending cleaning commands:

[0275] A user types into a smartphone app, "Start cleaning the car interior at 9:00 tomorrow morning."

[0276] Request a maintenance check:

[0277] When getting into the vehicle, enter the command "Tell me where there is a risk of breakdown" to obtain the necessary information.

[0278] 2. Example prompts for generative AI models

[0279] "Schedule my car to be cleaned tomorrow at 9am"

[0280] "Perform a maintenance check and report any abnormalities."

[0281] This system enables users to efficiently manage cleaning and maintenance work on autonomous vehicles from remote locations, significantly reducing the burden on users.

[0282] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0283] Step 1:

[0284] The user uses a user terminal such as a smartphone to input and send cleaning commands and maintenance check instructions.

[0285] Input: The user enters a prompt statement such as "Start cleaning the car tomorrow morning at 9am."

[0286] Output: Commands are sent from the user terminal to the server.

[0287] Step 2:

[0288] The server analyzes the commands received from the user terminal and determines the corresponding cleaning actions and maintenance checks.

[0289] Input: Cleaning command data sent from the user terminal.

[0290] Data processing / calculation: The server uses a generative artificial intelligence model to analyze and interpret commands in natural language.

[0291] Output: The interpreted commands are directed to specific cleaning actions or maintenance checks.

[0292] Step 3:

[0293] Based on the analysis results, the server instructs the autonomous mobile cleaning device on specific cleaning areas and maintenance operations.

[0294] Input: Parsed cleaning actions and maintenance check instructions.

[0295] Output: Specific cleaning area and operation instructions for the cleaning device.

[0296] Step 4:

[0297] The autonomous mobile cleaning device begins cleaning in the designated area and diagnoses the condition of the inside and outside of the vehicle.

[0298] Input: Cleaning and diagnostic instructions sent from the server.

[0299] Specific operation: The cleaning device locates itself and starts cleaning, while at the same time checking for any abnormalities in the vehicle using a diagnostic sensor.

[0300] Output: Cleaning progress and vehicle status data.

[0301] Step 5:

[0302] The cleaning device and diagnostic sensors transmit collected data to a server.

[0303] Input: Cleaning progress and vehicle condition data.

[0304] Data processing / calculation: The server analyzes these data and generates information to report to the user.

[0305] Output: Parsed cleaning and diagnostic result data.

[0306] Step 6:

[0307] The server reports the current status and diagnostic results of the cleaning device to the user terminal.

[0308] Input: Cleaning progress and vehicle condition data.

[0309] Specific operation: The server organizes these data and sends them to the user terminal.

[0310] Output: Reports to the user device such as "Cleaning is complete" or "There are areas at risk of failure."

[0311] Step 7:

[0312] The user checks the report results on a smartphone or smart glasses and gives additional instructions if necessary.

[0313] Input: Report message from the server.

[0314] Specific operation: The user checks the report results and inputs new cleaning commands or maintenance instructions.

[0315] Output: New cleaning commands or maintenance instructions are sent to the server again.

[0316] Through these steps, users can efficiently and automatically manage the cleaning and maintenance of their autonomous vehicles.

[0317] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0318] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[0319] Components

[0320] 1. Cleaning device

[0321] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly.

[0322] 2. User Device

[0323] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[0324] 3. Server

[0325] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[0326] System Operation Overview

[0327] 1. User sends command

[0328] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[0329] 2. Receiving and parsing commands

[0330] The user terminal transmits the input command to the server.

[0331] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[0332] 3. Sending commands to the cleaning device

[0333] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[0334] Once the cleaning device receives the command, it will begin cleaning the designated area.

[0335] 4. Status Check and Reporting

[0336] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[0337] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[0338] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[0339] Overview of Emotion Engine Operation

[0340] 1. User Emotion Recognition

[0341] The cleaning device is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions.

[0342] For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[0343] 2. Emotion-based adjustment of cleaning behavior

[0344] The cleaning device adjusts its cleaning schedule and frequency based on the emotions it recognizes.

[0345] For example, if the user is frustrated, the cleaning device may change its operation to finish cleaning quietly and quickly.

[0346] 3. Generating Emotional Feedback

[0347] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[0348] For example, a notification such as "The user is tired, so the living room will be cleaned quietly" is sent to the user terminal.

[0349] Specific examples

[0350] Example 1: Cleaning the living room

[0351] User: Type "Clean the living room" into their smartphone.

[0352] Terminal: Sends input to the server.

[0353] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[0354] Cleaning device: Receive instructions and start cleaning the living room.

[0355] Server: Notifies the user device that cleaning has started.

[0356] Device: Displays to the user "Cleaning of the living room has begun."

[0357] Example 2: Checking the cleaning status

[0358] User: Type "Tell me how the cleaning is going" on your smartphone.

[0359] Device: Sends a status check request to the server.

[0360] Server: Transmits the request to the cleaning device.

[0361] Cleaning device: Sends current status and location information back to the server.

[0362] Server: Returns the received information to the user terminal.

[0363] Device: Shows the user "You're currently cleaning the living room."

[0364] Example 3: Adjusting cleaning behavior through emotion recognition

[0365] User: In a tired voice, types, "Clean the living room."

[0366] Terminal: Sends input to the server.

[0367] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[0368] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[0369] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[0370] Server: Notifies the user device of emotional feedback from the cleaning device.

[0371] Device: Shows the user "Quietly clean the living room."

[0372] This system allows users to operate the cleaning device flexibly and efficiently even from a remote location, and check the status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning operations according to the user's emotions, thereby improving user satisfaction.

[0373] The processing flow will be explained below.

[0374] Step 1:

[0375] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the living room."

[0376] Step 2:

[0377] The terminal receives the user's input and sends a request of the form "command: clean area living room" to the server.

[0378] Step 3:

[0379] The server receives the request and extracts the command part ("clean area living room") through the handle_user_request method.

[0380] Step 4:

[0381] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[0382] Step 5:

[0383] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[0384] Step 6:

[0385] A generative artificial intelligence model analyzes the command and generates the response "Cleaning living room."

[0386] Step 7:

[0387] The cleaning device sends the generated response to the server.

[0388] Step 8:

[0389] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Cleaning of the living room has started."

[0390] Step 9:

[0391] The terminal displays the response received from the server on the screen and notifies the user.

[0392] Step 10:

[0393] The cleaning device uses a built-in emotion engine to recognize emotions from the user's voice and facial expressions. For example, if the user says "Clean up" in a tired voice, the emotion engine will recognize this as "fatigue."

[0394] Step 11:

[0395] The cleaning device automatically adjusts its cleaning plan based on the emotions it recognizes, for example, recognizing fatigue and changing settings to clean with reduced noise.

[0396] Step 12:

[0397] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[0398] Step 13:

[0399] The device sends a request called "status" to the server.

[0400] Step 14:

[0401] The server receives the status check request and invokes the report_status method of the cleaning device.

[0402] Step 15:

[0403] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., living room) to the server via the report_status method.

[0404] Step 16:

[0405] The server returns the received status information to the user terminal.

[0406] Step 17:

[0407] The device displays the status information received from the server on the screen and notifies the user, for example, "The living room is currently being cleaned."

[0408] Step 18:

[0409] The cleaning device generates feedback from the emotion engine and notifies the user terminal via the server. For example, a message such as "Because the user is tired, the living room will be cleaned quietly" is displayed.

[0410] Example 2

[0411] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0412] Conventional autonomous mobile cleaning devices simply receive and execute cleaning commands from the user, making it difficult to respond flexibly to emotions and situations. Furthermore, they often fail to efficiently check the status of the cleaning device or address specific stains in the room. Furthermore, because the cleaning device lacks the ability to understand the user's emotions and adapt its behavior based on those emotions, it is unable to provide a satisfactory service to some users.

[0413] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0414] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, and means including an emotion engine for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, thereby enabling real-time response to commands from the user and flexible cleaning operation according to the user's emotions.

[0415] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new content or results from data, and is applied to tasks such as natural language processing and image generation.

[0416] An "autonomous mobile cleaning device" is a cleaning device that has the ability to clean while moving around on its own, and is equipped with sensors, cameras, and mapping functions inside.

[0417] A "user terminal" is a device operated by a user, and includes smartphones, tablets, personal computers, etc.

[0418] The "server" is a central control unit that handles communication and data processing between the user terminal and the cleaning device, and has functions such as command analysis, instruction transmission, and status management.

[0419] An "emotion engine" refers to an algorithm or system that recognizes emotions from a user's voice and facial expressions and adjusts the device's behavior based on those emotions.

[0420] A "status check request" is a request sent by a user to check the current operating status and location information of the cleaning device.

[0421] A "cleaning command" is an instruction that a user sends to a cleaning device to instruct the device to perform a specific cleaning operation.

[0422] "Cleaning area" refers to the particular location or room in which the cleaning device performs cleaning.

[0423] An "efficient schedule" refers to a work plan generated to allow the cleaning device to clean as efficiently as possible, minimizing waste of time and energy.

[0424] The present invention is a system that mainly consists of an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and is composed of a user terminal and a server. The details of each component and the operation of the entire system are explained below.

[0425] Components

[0426] 1. Autonomous mobile cleaning device

[0427] The cleaning device is a piece of hardware that has a built-in generative artificial intelligence model and emotion engine, and moves around autonomously to clean. The device is equipped with sensors, cameras, and mapping functions, and operates efficiently within a designated cleaning area.

[0428] 2. User Device

[0429] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, which allows the user to send cleaning commands and check the status remotely.

[0430] 3. Server

[0431] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes data. The server analyzes commands from the user and sends specific operational instructions to the cleaning device. It also monitors the operating status and location information of the cleaning device.

[0432] System Operation

[0433] User command sending

[0434] A user inputs and sends a cleaning command such as "Clean the living room" from their smartphone. The user device then sends the command to the server.

[0435] Receiving and parsing commands

[0436] The server receives commands sent from the user terminal and analyzes the content using a text analysis engine. For example, if the command "clean the living room" is received, the server generates a cleaning area and an action plan based on the content.

[0437] Sending commands to cleaning devices

[0438] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving these instructions, the cleaning device begins cleaning the specified area.

[0439] Checking and reporting status

[0440] When a user wants to check the current status of the cleaning device, they send a status check request such as "Tell me the cleaning status" from the user terminal. The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The server then returns the received information to the user terminal.

[0441] Emotion Engine Operation

[0442] User Emotion Recognition

[0443] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[0444] Emotion-based adjustment of cleaning behavior

[0445] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is irritated, the cleaning device will change its behavior to clean quietly and quickly.

[0446] Generating Emotional Feedback

[0447] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[0448] Specific examples

[0449] Example 1: Cleaning the living room

[0450] The user types "clean the living room" into their smartphone.

[0451] The terminal sends this command to the server.

[0452] The server parses the command and generates the instruction "Clean the living room."

[0453] The cleaning device receives the parsed instructions and begins cleaning the living room.

[0454] The server notifies the user terminal of the status from the cleaning device and displays to the user, "Cleaning of the living room has started."

[0455] Example 2: Checking the cleaning status

[0456] The user types "Tell me about the cleaning status" on their smartphone.

[0457] The terminal sends a status confirmation request to the server.

[0458] The server communicates a request to the cleaning device to obtain the current status and location of the cleaning device.

[0459] The server returns the received information to the user terminal and displays to the user, "Currently cleaning the living room."

[0460] Example 3: Adjusting cleaning behavior through emotion recognition

[0461] The user types in a tired voice, "Clean the living room."

[0462] The terminal sends this input to the server.

[0463] The server analyzes the command and sends an instruction to "clean the living room" to the cleaning device.

[0464] The cleaning device receives instructions and the emotion engine recognizes the user's fatigue.

[0465] The cleaning device is adjusted to complete the cleaning operation quietly and quickly.

[0466] The server notifies the user terminal of the emotion feedback from the cleaning device and displays to the user, "Quietly cleans the living room."

[0467] With the above operations, users can operate the cleaning device flexibly and efficiently even from a remote location and check the status in real time. In addition, by incorporating an emotion engine, the device can provide appropriate cleaning operations according to the user's emotions, improving user satisfaction.

[0468] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0469] Program processing steps

[0470] Step 1: User command input

[0471] explanation

[0472] A user uses a user terminal such as a smartphone or tablet to input specific cleaning commands to the cleaning device.

[0473] input

[0474] The user enters a prompt sentence such as "Clean the living room."

[0475] Data processing and calculation

[0476] The terminal receives voice or text input from the user and converts the voice data to text data as needed.

[0477] output

[0478] A textual cleaning command is generated.

[0479] Specific actions

[0480] The user starts the application on the terminal and inputs cleaning commands using voice recognition or keyboard input.

[0481] Step 2: Send the command

[0482] explanation

[0483] The user terminal transmits the input command to the server.

[0484] input

[0485] Textual cleaning commands.

[0486] Data processing and calculation

[0487] The device converts the generated cleaning command into JSON format data and sends it to the server using an HTTP request.

[0488] output

[0489] The request data in JSON format is sent to the server.

[0490] Specific actions

[0491] The device sends an HTTP POST request to the server's API endpoint using Wi-Fi or mobile data.

[0492] Step 3: Command Analysis

[0493] explanation

[0494] The server analyzes the received command and generates specific operating instructions for the cleaning device.

[0495] input

[0496] Cleaning commands in JSON format.

[0497] Data processing and calculation

[0498] The server analyzes the command using a natural language processing engine and generates a specific cleaning area and cleaning route. For example, from the command "Clean the living room," it extracts the coordinate data of the living room and the cleaning route.

[0499] output

[0500] Specific operating instructions for the cleaning device (cleaning area and route information).

[0501] Specific actions

[0502] The server uses an internal text analysis engine to parse the received commands and calculate the appropriate cleaning route.

[0503] Step 4: Sending commands to the cleaning device

[0504] explanation

[0505] Based on the analysis results, the server sends specific cleaning areas and operation instructions to the cleaning device.

[0506] input

[0507] Specific operating instructions for the cleaning device.

[0508] Data processing and calculation

[0509] The server converts the generated operation instructions into a protocol that can be understood by the cleaning device.

[0510] output

[0511] Cleaning instruction data is transmitted in a format that can be understood by the cleaning device.

[0512] Specific actions

[0513] The server uses an HTTP request to send instruction data to the communication module of the cleaning device.

[0514] Step 5: Start cleaning

[0515] explanation

[0516] The cleaning device receives instructions from the server and begins cleaning the designated area.

[0517] input

[0518] Cleaning instruction data.

[0519] Data processing and calculation

[0520] Based on the instructions it receives, the cleaning device works in conjunction with an internal mapping sensor to check its own position while cleaning.

[0521] output

[0522] The cleaning operation begins.

[0523] Specific actions

[0524] The cleaning device activates the operation module and performs cleaning while scanning the designated area.

[0525] Step 6: Check and report status

[0526] explanation

[0527] When a user wants to check the current status of the cleaning device, he sends a request to the server, which retrieves the status information from the cleaning device and returns it to the user terminal.

[0528] input

[0529] Status requests (e.g., "How's the cleaning going?").

[0530] Data processing and calculation

[0531] The server passes the request to the cleaning device, which then returns its current status and location information to the server.

[0532] output

[0533] The current status and location of the cleaning device is displayed on the user terminal.

[0534] Specific actions

[0535] The server receives a request from the user terminal, sends a "status check" instruction to the cleaning device, and returns a response from the cleaning device to the user terminal.

[0536] Step 7: Emotion Recognition

[0537] explanation

[0538] The cleaning device uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[0539] input

[0540] User voice and facial expression data.

[0541] Data processing and calculation

[0542] The cleaning device uses an emotion engine to extract emotions from the user's voice and facial expressions and assign appropriate emotion labels.

[0543] output

[0544] Recognized emotion data (e.g., "fatigue").

[0545] Specific actions

[0546] The cleaning device uses a built-in microphone and camera to analyze the user's emotions and stores the results in its internal memory.

[0547] Step 8: Emotionally Based Behavior Adjustments

[0548] explanation

[0549] The cleaning device adjusts its cleaning behavior based on the recognized emotion.

[0550] input

[0551] Emotional data (e.g., "fatigue").

[0552] Data processing and calculation

[0553] The cleaning device optimizes its operating parameters (volume, speed, etc.) based on emotion data.

[0554] output

[0555] The cleaning operation is performed based on the adjusted operation parameters.

[0556] Specific actions

[0557] The cleaning device switches to a quiet mode and adjusts the cleaning speed to perform operations that are considerate of the user's feelings.

[0558] The above are the specific processing steps and details of the system.

[0559] (Application example 2)

[0560] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0561] Conventional autonomous mobile cleaning devices have problems such as inefficient cleaning instructions and status checks, and inability to adapt to user emotions and store conditions. In particular, cleaning activities in brick-and-mortar stores require frequent cleaning to maintain cleanliness, and streamlining cleaning work is required to reduce the burden on store staff. Furthermore, a system that can appropriately adjust cleaning operations according to the store's congestion status and staff stress levels is desired.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, and means for analyzing the command input from the user and reporting the operating status and battery status of the cleaning robot in real time. This allows users to flexibly and efficiently operate the cleaning device from a remote location and check the status in real time. In particular, in physical stores, appropriate cleaning operations can be provided according to the store's congestion status and the staff's stress level, thereby improving the efficiency of cleaning work and reducing the burden on store staff.

[0563] A "generative artificial intelligence model" is an artificial intelligence model that has the ability to generate new data and information based on given data.

[0564] An "autonomous mobile cleaning device" is a device that moves autonomously in response to instructions from a user and cleans a designated area.

[0565] A "user terminal" is a remote control device operated by a user, such as a smartphone, tablet, or PC.

[0566] A "cleaning command" is a command that a user inputs to instruct the autonomously mobile cleaning device to perform cleaning.

[0567] A "status confirmation request" is a request sent by a user to find out the current cleaning status and location information of the autonomously mobile cleaning device.

[0568] The "corresponding cleaning area" is a specific area that the autonomously mobile cleaning device has memorized as a cleaning target.

[0569] An "efficient schedule" is a time and motion plan designed to optimize cleaning tasks for an autonomous mobile cleaning device.

[0570] The "emotion engine" is an engine that has the function of recognizing the user's emotional state from their voice, facial expression, etc., and adjusting cleaning operations according to that state.

[0571] "Battery status" is information indicating the current remaining battery power and operating time of the autonomously mobile cleaning device.

[0572] "Real-time" is a term that refers to information being transmitted and processed almost immediately.

[0573] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[0574] Components

[0575] 1. Cleaning device

[0576] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly. The main hardware used includes a LiDAR sensor, camera, and microphone.

[0577] 2. User Device

[0578] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. This allows the user to remotely issue cleaning commands and check the status. Commands and requests are sent from the user terminal to the server.

[0579] 3. Server

[0580] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes various requests. The server analyzes commands from the cleaning device and reports their status. The main software used includes Node.js and Java. MySQL and PostgreSQL are used as databases.

[0581] System Operation Overview

[0582] 1. User sends command

[0583] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[0584] 2. Receiving and parsing commands

[0585] The user terminal sends the input command to the server, which receives the command and analyzes it to determine the corresponding cleaning action.

[0586] 3. Sending commands to the cleaning device

[0587] Based on the analysis results, the server instructs the cleaning device on the specific cleaning area and operation. Upon receiving the instruction, the cleaning device begins cleaning the specified area.

[0588] 4. Status Check and Reporting

[0589] When a user wants to check the current status of the cleaning device, the user terminal sends a status check request such as "Tell me the cleaning status." The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The cleaning device returns its current status and location information to the server, which then transmits it to the user terminal.

[0590] Overview of Emotion Engine Operation

[0591] 1. User Emotion Recognition

[0592] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[0593] 2. Emotion-based adjustment of cleaning behavior

[0594] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is tired, the cleaning device will change its behavior to clean quietly and quickly.

[0595] 3. Generating Emotional Feedback

[0596] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[0597] Specific examples

[0598] Example 1: Cleaning the living room

[0599] User: Type "Clean the living room" into their smartphone.

[0600] Terminal: Sends input to the server.

[0601] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[0602] Cleaning device: Receive instructions and start cleaning the living room.

[0603] Server: Notifies the user device that cleaning has started.

[0604] Device: Displays to the user "Cleaning of the living room has begun."

[0605] Example 2: Checking the cleaning status

[0606] User: Type "Tell me how the cleaning is going" on your smartphone.

[0607] Device: Sends a status check request to the server.

[0608] Server: Transmits the request to the cleaning device.

[0609] Cleaning device: Sends current status and location information back to the server.

[0610] Server: Returns the received information to the user terminal.

[0611] Device: Shows the user "You're currently cleaning the living room."

[0612] Example 3: Adjusting cleaning behavior through emotion recognition

[0613] User: In a tired voice, types, "Clean the living room."

[0614] Terminal: Sends input to the server.

[0615] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[0616] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[0617] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[0618] Server: Notifies the user device of emotional feedback from the cleaning device.

[0619] Device: Shows the user "Quietly clean the living room."

[0620] Prompt Sentence Examples

[0621] "It promptly cleans the shopping area specified by the user, reports battery status and current location information in real time, and makes appropriate adjustments based on the user's voice and facial expressions."

[0622] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[0623] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0624] Step 1:

[0625] User command sending

[0626] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[0627] Input: Cleaning commands entered by the user.

[0628] Output: A cleaning command is sent from the user terminal to the server.

[0629] In this step, commands are entered through the user terminal interface and sent to the server as internal API requests.

[0630] Step 2:

[0631] Receiving and parsing commands

[0632] The server receives and analyzes the command sent from the user terminal.

[0633] Input: Cleaning command sent from user terminal.

[0634] Output: Analyzed cleaning area and action instructions.

[0635] The server analyzes the command and determines the corresponding cleaning action. This analysis is done using a natural language processing model (e.g., a generative AI model). The analysis result is then sent to the next step.

[0636] Step 3:

[0637] Sending commands to cleaning devices

[0638] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[0639] Input: Analyzed cleaning area and action instructions.

[0640] Output: Operation instructions to the cleaning device.

[0641] The server sends an HTTP request to the cleaning device, informing it of the required cleaning instructions, and the cleaning device begins to operate according to the received instructions.

[0642] Step 4:

[0643] Emotional awareness and regulation

[0644] The cleaning device performs processing to recognize the user's emotions.

[0645] Input: User's voice and facial expressions.

[0646] Output: Adjustment of cleaning behavior based on recognized emotions.

[0647] The cleaning device uses a built-in microphone and camera to recognize the user's emotional state, and the recognized emotional data is processed by an emotion engine to determine an appropriate cleaning plan.

[0648] Step 5:

[0649] Checking and reporting status

[0650] When a user wants to check the current cleaning status, the user sends a request from the user terminal saying "Tell me the cleaning status."

[0651] Input: Status check request from user terminal.

[0652] Output: Current cleaning status and location.

[0653] The server receives this request and queries the cleaning device about its current status. The cleaning device returns the current cleaning status to the server, which then transmits it to the user terminal.

[0654] Step 6:

[0655] Generating Emotional Feedback

[0656] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[0657] Input: Perceived user emotion.

[0658] Output: Emotion feedback notification to the user device.

[0659] The cleaning device generates an appropriate feedback message based on the emotion data processed by the emotion engine and transmits it to the user terminal via the server.

[0660] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[0661] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0662] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0663] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0664] [Second embodiment]

[0665] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0666] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0667] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0668] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0669] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0670] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0671] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0672] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0673] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0674] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0675] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0676] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0677] This system is based on an autonomous mobile cleaning device equipped with a generative artificial intelligence model, and is also composed of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[0678] Components

[0679] 1. Cleaning device

[0680] The cleaning device is a piece of hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[0681] 2. User Device

[0682] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[0683] 3. Server

[0684] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[0685] System Operation Overview

[0686] 1. User sends command

[0687] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[0688] 2. Receiving and parsing commands

[0689] The user terminal transmits the input command to the server.

[0690] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[0691] 3. Sending commands to the cleaning device

[0692] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[0693] Once the cleaning device receives the command, it will begin cleaning the designated area.

[0694] 4. Status Check and Reporting

[0695] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[0696] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[0697] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[0698] Specific examples

[0699] Example 1: Cleaning the living room

[0700] User: Type "Clean the living room" into their smartphone.

[0701] Terminal: Sends input to the server.

[0702] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[0703] Cleaning device: Receive instructions and start cleaning the living room.

[0704] Server: Notifies the user device that cleaning has started.

[0705] Device: Displays to the user "Cleaning of the living room has begun."

[0706] Example 2: Checking the cleaning status

[0707] User: Type "Tell me how the cleaning is going" on your smartphone.

[0708] Device: Sends a status check request to the server.

[0709] Server: Transmits the request to the cleaning device.

[0710] Cleaning device: Sends current status and location information back to the server.

[0711] Server: Returns the received information to the user terminal.

[0712] Device: Shows the user "You're currently cleaning the living room."

[0713] This system allows users to operate cleaning devices flexibly and efficiently even from remote locations and check their status in real time, greatly improving the convenience of cleaning work.

[0714] The processing flow will be explained below.

[0715] Step 1:

[0716] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the kitchen."

[0717] Step 2:

[0718] The terminal receives the user's input and sends a request of the form "command: clean area kitchen" to the server.

[0719] Step 3:

[0720] The server receives the request and extracts the command part ("clean area kitchen") through the handle_user_request method.

[0721] Step 4:

[0722] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[0723] Step 5:

[0724] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[0725] Step 6:

[0726] A generative artificial intelligence model analyzes the command "clean area kitchen" and generates the response "Cleaning kitchen."

[0727] Step 7:

[0728] The cleaning device sends the generated response to the server.

[0729] Step 8:

[0730] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Kitchen cleaning has started."

[0731] Step 9:

[0732] The terminal displays the response received from the server on the screen and notifies the user.

[0733] Step 10:

[0734] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[0735] Step 11:

[0736] The device sends a request called "status" to the server.

[0737] Step 12:

[0738] The server receives the status check request and invokes the report_status method of the cleaning device.

[0739] Step 13:

[0740] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., kitchen) to the server via the report_status method.

[0741] Step 14:

[0742] The server returns the received status information to the user terminal.

[0743] Step 15:

[0744] The terminal displays the status information received from the server on the screen and notifies the user, for example, "The kitchen is currently being cleaned."

[0745] Example 1

[0746] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0747] Conventional autonomous mobile cleaning devices have had issues such as difficulty for users to control the cleaning device efficiently from a remote location, insufficient real-time confirmation of the cleaning status, insufficient status reporting, difficulty in providing optimal cleaning plans for specific stains or rooms, and difficulty in responding to questions from users in natural language.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0749] In this invention, the server includes a means for analyzing cleaning commands received from a user-operated device and initiating the corresponding cleaning operation, a means for reporting the current cleaning status and location information in response to a status check request from the device, a means for storing the corresponding cleaning area and generating an efficient schedule, a means for incorporating a generative artificial intelligence model and responding to questions from the user in natural language, and a means for communicating data between the user device, the cleaning device, and the server. This allows users to efficiently control the cleaning device and check its status in real time, even from remote locations. It also provides optimal cleaning plans for specific stains or rooms and enables question and answering in natural language, greatly improving the convenience of cleaning work.

[0750] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new information and responses based on data, and is particularly used in natural language processing and image generation.

[0751] An "autonomous mobile cleaning device" is a device that has the function of cleaning while autonomously moving around a specific area, and performs various operations based on instructions from a user.

[0752] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC, and is a means for communicating with a cleaning device or a server.

[0753] The "server" is a central control device that mediates communication between the user terminal and the cleaning device, and processes and analyzes various requests.

[0754] A "cleaning command" is an instruction that a user inputs from a terminal and sends to the cleaning device, and includes instructions to start cleaning or to clean a specific area.

[0755] A "status confirmation request" is a request sent from a terminal by a user to check the current cleaning status and location information of the cleaning device.

[0756] An "efficient schedule" is an optimal cleaning plan generated based on the area to be cleaned and the user's schedule, and is designed to maximize the efficiency and effectiveness of cleaning.

[0757] The "function to answer questions in natural language" is a function that uses a generative artificial intelligence model to generate appropriate responses to questions or instructions from the user in natural language and responds.

[0758] "Data communication" is the process of exchanging information between different devices over a communication network.

[0759] The present invention is a system that mainly comprises an autonomous mobile cleaning device equipped with a generative artificial intelligence model, a user terminal, and a server. The details of each component and the operation of the entire system are explained below.

[0760] Components

[0761] 1. Autonomous mobile cleaning device

[0762] The cleaning device is hardware that can move around autonomously and clean. Using a built-in generative artificial intelligence model (e.g., OpenAI GPT-3), it can respond to cleaning commands and questions from the user in natural language. It also uses sensors (e.g., LIDAR and infrared sensors) to understand the surrounding environment and plan an efficient cleaning route.

[0763] 2. User Device

[0764] The user terminal is a device operated by the user, such as a smartphone, tablet, or PC. The user can issue cleaning commands and check the status through a dedicated application or web interface. Communication is via Wi-Fi or LTE.

[0765] 3. Server

[0766] The server is a central control device that mediates communication between the user device and the cleaning device and processes various requests. It analyzes the received cleaning commands and determines and instructs the specific cleaning operations. The server also reports the current status and location information of the cleaning device to the user device.

[0767] System Operation Overview

[0768] Entering and sending commands

[0769] Users use a device such as a smartphone to input and send cleaning commands, such as "Clean the living room," into a dedicated app or web interface.

[0770] Receiving and parsing commands

[0771] The user terminal transmits the input command to the server.

[0772] The server analyzes the received command using a natural language processing engine (e.g., Google Cloud Natural Language API), inputs specific instructions such as "clean the living room" as a prompt sentence to the generative AI model, and obtains the generated detailed cleaning plan.

[0773] Sending commands to cleaning devices

[0774] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[0775] The cleaning device receives instructions and begins cleaning the designated area (e.g., living room), using built-in generative AI models and sensors to plan and execute an efficient cleaning route.

[0776] Checking and reporting status

[0777] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[0778] The user terminal sends this request to the server.

[0779] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[0780] The cleaning device sends its current status and location information back to the server.

[0781] The server returns the received information to the user terminal.

[0782] The user terminal displays to the user, "Currently cleaning the living room."

[0783] Specific operation example

[0784] Example 1: Cleaning the living room

[0785] The user types "clean the living room" into their smartphone.

[0786] The terminal sends this command to the server.

[0787] The server receives the command, analyzes it, and inputs the "clean the living room" instruction into the generative AI model. Example prompt: "What is the best route to efficiently clean the living room?"

[0788] The server transmits specific instructions to the cleaning device based on the generated cleaning plan.

[0789] The vacuum cleaner receives instructions and begins cleaning the living room, using sensors and generative AI models to clean efficiently.

[0790] Example 2: Checking the cleaning status

[0791] The user types "Tell me about the cleaning status" on their smartphone.

[0792] The terminal sends a status confirmation request to the server.

[0793] The server communicates the request to the cleaning device.

[0794] The cleaning device sends its current status and location information back to the server.

[0795] The server returns the information to the user terminal.

[0796] The device displays to the user, "Currently cleaning the living room."

[0797] With this mechanism, the system of the present invention significantly improves user convenience and enables efficient and flexible cleaning work.

[0798] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0799] Specific processing steps of the program

[0800] Step 1:

[0801] The user enters a command

[0802] Users use a dedicated app on their smartphone or tablet to input cleaning commands such as "clean the living room."

[0803] Input: User's cleaning command (e.g., "Clean the living room")

[0804] Output: The entered command is recorded on the user's terminal and prepared for transmission.

[0805] Step 2:

[0806] Sending commands from the user terminal to the server

[0807] The device sends the entered cleaning command to a server via the internet, using Wi-Fi or LTE for communication.

[0808] Input: Cleaning command entered by the user

[0809] Output: Cleaning command is sent to the server

[0810] Step 3:

[0811] The server receives the command

[0812] The server receives a cleaning command sent from a user terminal.

[0813] Input: Cleaning command sent from user terminal

[0814] Output: The received cleaning command is queued in the server.

[0815] Step 4:

[0816] The server parses the command

[0817] The server analyzes the received cleaning command using a natural language processing engine (e.g., Google Cloud Natural Language API).

[0818] Input: Cleaning command received

[0819] Data processing: A natural language processing engine is used to analyze the meaning of commands and convert them into specific cleaning instructions.

[0820] Output: Specific cleaning instructions as a result of the analysis (e.g., "Start cleaning the living room")

[0821] Step 5:

[0822] The server sends a prompt to the generative AI model

[0823] Based on the analysis results, the server sends prompts to a generative AI model (e.g., OpenAI GPT-3) to generate a detailed cleaning plan.

[0824] Input: Specific cleaning instructions as analysis results

[0825] Data Computation: Generative AI model generates optimal cleaning route and plan based on prompts

[0826] Output: Generated cleaning route and plan

[0827] Step 6:

[0828] The server sends commands to the cleaning device.

[0829] The server instructs the cleaning device on specific cleaning areas and operations based on the generated cleaning route and plan.

[0830] Input: Generated cleaning route and plan

[0831] Output: Specific cleaning instructions are sent to the cleaning device

[0832] Step 7:

[0833] The cleaning device starts cleaning

[0834] Based on the instructions received, the cleaning device begins cleaning the designated area (e.g., the living room), planning and executing an efficient cleaning route using built-in sensors and generative AI models.

[0835] Input: Specific cleaning instructions from the server

[0836] Data calculation: Cleaning is performed while dynamically correcting the actual cleaning route based on sensor information.

[0837] Output: Dust and dirt in the area is removed

[0838] Step 8:

[0839] User submits a status check request

[0840] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[0841] Input: Status Check Request

[0842] Output: The request is logged on the user's terminal and ready to be sent.

[0843] Step 9:

[0844] The user device sends a request to the server

[0845] The terminal sends this request to the server.

[0846] Input: User status check request

[0847] Output: The request is transmitted to the server

[0848] Step 10:

[0849] Server requests status

[0850] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[0851] Input: User status check request

[0852] Output: The current cleaning status and location information request are sent to the cleaning device.

[0853] Step 11:

[0854] The cleaning device sends status information back to the server

[0855] The cleaning device sends its current status and location information back to the server.

[0856] Input: Status information request from the server

[0857] Data calculation: Collects current cleaning progress and location information

[0858] Output: Sending collected status and location information

[0859] Step 12:

[0860] The server sends status information to the user terminal.

[0861] The server returns the received information to the user terminal.

[0862] Input: Status and location information from the cleaning device

[0863] Output: Information is transmitted to the user terminal

[0864] Step 13:

[0865] The user terminal displays the status

[0866] The terminal displays to the user, "Currently cleaning the living room."

[0867] Input: Status and location information from the server

[0868] Output: Status information is displayed to the user

[0869] This allows the user to operate the cleaning device efficiently and in real time even from a remote location and check the current cleaning status.

[0870] (Application example 1)

[0871] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0872] Modern self-driving vehicles have advanced autonomous driving capabilities, but cleaning the interior and exterior of the vehicle and performing regular maintenance require manual labor, which is a hassle for users. Furthermore, diagnosing the vehicle's condition and identifying repair areas requires specialized knowledge, which is inconvenient for users. There is a need to solve these problems and provide a system that efficiently automates the cleaning and maintenance of self-driving vehicles.

[0873] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0874] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status confirmation request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, means for automatically cleaning the interior and exterior of the vehicle and diagnosing the vehicle status, and means for scheduling cleaning maintenance and automatically identifying and reporting areas requiring repairs. This automates vehicle cleaning and maintenance work, allowing users to flexibly manage these tasks from a remote location.

[0875] A "generative artificial intelligence model" is an artificial intelligence technique that can generate new information and content from data.

[0876] An "autonomous mobile cleaning device" is a hardware device that can clean while moving around on its own.

[0877] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[0878] A "cleaning command" is an instruction or command given by a user to a cleaning device.

[0879] A "status confirmation request" is a request made by the user to confirm the current status and location information of the cleaning device.

[0880] A "cleaning zone" is a particular area that a cleaning device is to clean.

[0881] The "means for generating a schedule" is a function that determines the time and order for the cleaning device to perform cleaning work efficiently.

[0882] "Automatic cleaning inside and outside the vehicle" is a function that automatically cleans the inside and outside of the vehicle.

[0883] "Means for diagnosing the condition of the vehicle" refers to a function that checks the condition of each part and system of the vehicle and detects abnormalities.

[0884] "Cleaning and maintenance scheduling" is a function that allows you to create a schedule for systematically cleaning and maintaining your vehicle.

[0885] The "means for automatically identifying and reporting parts requiring repair" is a function for automatically detecting parts of a vehicle that are faulty and reporting the same to the user.

[0886] This invention is a system that automates cleaning and maintenance work for self-driving vehicles using an autonomous mobile cleaning device equipped with a generative artificial intelligence model. The system is composed of a user terminal, a server, and the autonomous mobile cleaning device.

[0887] Components

[0888] 1. Autonomous mobile cleaning device

[0889] The cleaning device is a hardware device that moves autonomously and cleans the inside and outside of a vehicle. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[0890] 2. User Device

[0891] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. Using this terminal, the user can remotely issue cleaning commands and check the status.

[0892] 3. Server

[0893] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[0894] System Operation Overview

[0895] 1. User sends command

[0896] Users input and send cleaning commands such as "clean the interior of the car" or "perform a maintenance check" using smart glasses or a smartphone.

[0897] 2. Receiving and parsing commands

[0898] The server receives commands sent from the user terminal, analyzes the contents of the commands, and then identifies cleaning areas and diagnostic points inside and outside the vehicle.

[0899] 3. Sending commands to the cleaning device

[0900] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving the instructions, the cleaning device begins cleaning the designated area. At the same time, it also performs a vehicle condition diagnosis.

[0901] 4. Status Check and Reporting

[0902] When a user wants to check the current status of the cleaning device, the user sends a command such as "Tell me the current situation" from the user terminal. The server transmits this request to the cleaning device, collects the current cleaning status and location information, and reports it to the user terminal.

[0903] Hardware and software used

[0904] Hardware:

[0905] Autonomous mobile cleaning device mounted on a self-driving vehicle

[0906] Vehicle condition diagnostic sensors

[0907] software:

[0908] Python

[0909] Flask (server-side request processing)

[0910] Generative AI model

[0911] Specific examples

[0912] 1. Specific scenario

[0913] Sending cleaning commands:

[0914] A user types into a smartphone app, "Start cleaning the car interior at 9:00 tomorrow morning."

[0915] Request a maintenance check:

[0916] When getting into the vehicle, enter the command "Tell me where there is a risk of breakdown" to obtain the necessary information.

[0917] 2. Example prompts for generative AI models

[0918] "Schedule my car to be cleaned tomorrow at 9am"

[0919] "Perform a maintenance check and report any abnormalities."

[0920] This system enables users to efficiently manage cleaning and maintenance work on autonomous vehicles from remote locations, significantly reducing the burden on users.

[0921] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0922] Step 1:

[0923] The user uses a user terminal such as a smartphone to input and send cleaning commands and maintenance check instructions.

[0924] Input: The user enters a prompt statement such as "Start cleaning the car tomorrow morning at 9am."

[0925] Output: Commands are sent from the user terminal to the server.

[0926] Step 2:

[0927] The server analyzes the commands received from the user terminal and determines the corresponding cleaning actions and maintenance checks.

[0928] Input: Cleaning command data sent from the user terminal.

[0929] Data processing / calculation: The server uses a generative artificial intelligence model to analyze and interpret commands in natural language.

[0930] Output: The interpreted commands are directed to specific cleaning actions or maintenance checks.

[0931] Step 3:

[0932] Based on the analysis results, the server instructs the autonomous mobile cleaning device on specific cleaning areas and maintenance operations.

[0933] Input: Parsed cleaning actions and maintenance check instructions.

[0934] Output: Specific cleaning area and operation instructions for the cleaning device.

[0935] Step 4:

[0936] The autonomous mobile cleaning device begins cleaning in the designated area and diagnoses the condition of the inside and outside of the vehicle.

[0937] Input: Cleaning and diagnostic instructions sent from the server.

[0938] Specific operation: The cleaning device locates itself and starts cleaning, while at the same time checking for any abnormalities in the vehicle using a diagnostic sensor.

[0939] Output: Cleaning progress and vehicle status data.

[0940] Step 5:

[0941] The cleaning device and diagnostic sensors transmit collected data to a server.

[0942] Input: Cleaning progress and vehicle condition data.

[0943] Data processing / calculation: The server analyzes these data and generates information to report to the user.

[0944] Output: Parsed cleaning and diagnostic result data.

[0945] Step 6:

[0946] The server reports the current status and diagnostic results of the cleaning device to the user terminal.

[0947] Input: Cleaning progress and vehicle condition data.

[0948] Specific operation: The server organizes these data and sends them to the user terminal.

[0949] Output: Reports to the user device such as "Cleaning is complete" or "There are areas at risk of failure."

[0950] Step 7:

[0951] The user checks the report results on a smartphone or smart glasses and gives additional instructions if necessary.

[0952] Input: Report message from the server.

[0953] Specific operation: The user checks the report results and inputs new cleaning commands or maintenance instructions.

[0954] Output: New cleaning commands or maintenance instructions are sent to the server again.

[0955] Through these steps, users can efficiently and automatically manage the cleaning and maintenance of their autonomous vehicles.

[0956] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0957] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[0958] Components

[0959] 1. Cleaning device

[0960] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly.

[0961] 2. User Device

[0962] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[0963] 3. Server

[0964] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[0965] System Operation Overview

[0966] 1. User sends command

[0967] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[0968] 2. Receiving and parsing commands

[0969] The user terminal transmits the input command to the server.

[0970] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[0971] 3. Sending commands to the cleaning device

[0972] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[0973] Once the cleaning device receives the command, it will begin cleaning the designated area.

[0974] 4. Status Check and Reporting

[0975] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[0976] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[0977] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[0978] Overview of Emotion Engine Operation

[0979] 1. User Emotion Recognition

[0980] The cleaning device is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions.

[0981] For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[0982] 2. Emotion-based adjustment of cleaning behavior

[0983] The cleaning device adjusts its cleaning schedule and frequency based on the emotions it recognizes.

[0984] For example, if the user is frustrated, the cleaning device may change its operation to finish cleaning quietly and quickly.

[0985] 3. Generating Emotional Feedback

[0986] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[0987] For example, a notification such as "The user is tired, so the living room will be cleaned quietly" is sent to the user terminal.

[0988] Specific examples

[0989] Example 1: Cleaning the living room

[0990] User: Type "Clean the living room" into their smartphone.

[0991] Terminal: Sends input to the server.

[0992] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[0993] Cleaning device: Receive instructions and start cleaning the living room.

[0994] Server: Notifies the user device that cleaning has started.

[0995] Device: Displays to the user "Cleaning of the living room has begun."

[0996] Example 2: Checking the cleaning status

[0997] User: Type "Tell me how the cleaning is going" on your smartphone.

[0998] Device: Sends a status check request to the server.

[0999] Server: Transmits the request to the cleaning device.

[1000] Cleaning device: Sends current status and location information back to the server.

[1001] Server: Returns the received information to the user terminal.

[1002] Device: Shows the user "You're currently cleaning the living room."

[1003] Example 3: Adjusting cleaning behavior through emotion recognition

[1004] User: In a tired voice, types, "Clean the living room."

[1005] Terminal: Sends input to the server.

[1006] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1007] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[1008] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[1009] Server: Notifies the user device of emotional feedback from the cleaning device.

[1010] Device: Shows the user "Quietly clean the living room."

[1011] This system allows users to operate the cleaning device flexibly and efficiently even from a remote location, and check the status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning operations according to the user's emotions, thereby improving user satisfaction.

[1012] The processing flow will be explained below.

[1013] Step 1:

[1014] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the living room."

[1015] Step 2:

[1016] The terminal receives the user's input and sends a request of the form "command: clean area living room" to the server.

[1017] Step 3:

[1018] The server receives the request and extracts the command part ("clean area living room") through the handle_user_request method.

[1019] Step 4:

[1020] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[1021] Step 5:

[1022] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[1023] Step 6:

[1024] A generative artificial intelligence model analyzes the command and generates the response "Cleaning living room."

[1025] Step 7:

[1026] The cleaning device sends the generated response to the server.

[1027] Step 8:

[1028] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Cleaning of the living room has started."

[1029] Step 9:

[1030] The terminal displays the response received from the server on the screen and notifies the user.

[1031] Step 10:

[1032] The cleaning device uses a built-in emotion engine to recognize emotions from the user's voice and facial expressions. For example, if the user says "Clean up" in a tired voice, the emotion engine will recognize this as "fatigue."

[1033] Step 11:

[1034] The cleaning device automatically adjusts its cleaning plan based on the emotions it recognizes, for example, recognizing fatigue and changing settings to clean with reduced noise.

[1035] Step 12:

[1036] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[1037] Step 13:

[1038] The device sends a request called "status" to the server.

[1039] Step 14:

[1040] The server receives the status check request and invokes the report_status method of the cleaning device.

[1041] Step 15:

[1042] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., living room) to the server via the report_status method.

[1043] Step 16:

[1044] The server returns the received status information to the user terminal.

[1045] Step 17:

[1046] The device displays the status information received from the server on the screen and notifies the user, for example, "The living room is currently being cleaned."

[1047] Step 18:

[1048] The cleaning device generates feedback from the emotion engine and notifies the user terminal via the server. For example, a message such as "Because the user is tired, the living room will be cleaned quietly" is displayed.

[1049] Example 2

[1050] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1051] Conventional autonomous mobile cleaning devices simply receive and execute cleaning commands from the user, making it difficult to respond flexibly to emotions and situations. Furthermore, they often fail to efficiently check the status of the cleaning device or address specific stains in the room. Furthermore, because the cleaning device lacks the ability to understand the user's emotions and adapt its behavior based on those emotions, it is unable to provide a satisfactory service to some users.

[1052] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1053] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, and means including an emotion engine for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, thereby enabling real-time response to commands from the user and flexible cleaning operation according to the user's emotions.

[1054] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new content or results from data, and is applied to tasks such as natural language processing and image generation.

[1055] An "autonomous mobile cleaning device" is a cleaning device that has the ability to clean while moving around on its own, and is equipped with sensors, cameras, and mapping functions inside.

[1056] A "user terminal" is a device operated by a user, and includes smartphones, tablets, personal computers, etc.

[1057] The "server" is a central control unit that handles communication and data processing between the user terminal and the cleaning device, and has functions such as command analysis, instruction transmission, and status management.

[1058] An "emotion engine" refers to an algorithm or system that recognizes emotions from a user's voice and facial expressions and adjusts the device's behavior based on those emotions.

[1059] A "status check request" is a request sent by a user to check the current operating status and location information of the cleaning device.

[1060] A "cleaning command" is an instruction that a user sends to a cleaning device to instruct the device to perform a specific cleaning operation.

[1061] "Cleaning area" refers to the particular location or room in which the cleaning device performs cleaning.

[1062] An "efficient schedule" refers to a work plan generated to allow the cleaning device to clean as efficiently as possible, minimizing waste of time and energy.

[1063] The present invention is a system that mainly consists of an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and is composed of a user terminal and a server. The details of each component and the operation of the entire system are explained below.

[1064] Components

[1065] 1. Autonomous mobile cleaning device

[1066] The cleaning device is a piece of hardware that has a built-in generative artificial intelligence model and emotion engine, and moves around autonomously to clean. The device is equipped with sensors, cameras, and mapping functions, and operates efficiently within a designated cleaning area.

[1067] 2. User Device

[1068] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, which allows the user to send cleaning commands and check the status remotely.

[1069] 3. Server

[1070] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes data. The server analyzes commands from the user and sends specific operational instructions to the cleaning device. It also monitors the operating status and location information of the cleaning device.

[1071] System Operation

[1072] User command sending

[1073] A user inputs and sends a cleaning command such as "Clean the living room" from their smartphone. The user device then sends the command to the server.

[1074] Receiving and parsing commands

[1075] The server receives commands sent from the user terminal and analyzes the content using a text analysis engine. For example, if the command "clean the living room" is received, the server generates a cleaning area and an action plan based on the content.

[1076] Sending commands to cleaning devices

[1077] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving these instructions, the cleaning device begins cleaning the specified area.

[1078] Checking and reporting status

[1079] When a user wants to check the current status of the cleaning device, they send a status check request such as "Tell me the cleaning status" from the user terminal. The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The server then returns the received information to the user terminal.

[1080] Emotion Engine Operation

[1081] User Emotion Recognition

[1082] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[1083] Emotion-based adjustment of cleaning behavior

[1084] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is irritated, the cleaning device will change its behavior to clean quietly and quickly.

[1085] Generating Emotional Feedback

[1086] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[1087] Specific examples

[1088] Example 1: Cleaning the living room

[1089] The user types "clean the living room" into their smartphone.

[1090] The terminal sends this command to the server.

[1091] The server parses the command and generates the instruction "Clean the living room."

[1092] The cleaning device receives the parsed instructions and begins cleaning the living room.

[1093] The server notifies the user terminal of the status from the cleaning device and displays to the user, "Cleaning of the living room has started."

[1094] Example 2: Checking the cleaning status

[1095] The user types "Tell me about the cleaning status" on their smartphone.

[1096] The terminal sends a status confirmation request to the server.

[1097] The server communicates a request to the cleaning device to obtain the current status and location of the cleaning device.

[1098] The server returns the received information to the user terminal and displays to the user, "Currently cleaning the living room."

[1099] Example 3: Adjusting cleaning behavior through emotion recognition

[1100] The user types in a tired voice, "Clean the living room."

[1101] The terminal sends this input to the server.

[1102] The server analyzes the command and sends an instruction to "clean the living room" to the cleaning device.

[1103] The cleaning device receives instructions and the emotion engine recognizes the user's fatigue.

[1104] The cleaning device is adjusted to complete the cleaning operation quietly and quickly.

[1105] The server notifies the user terminal of the emotion feedback from the cleaning device and displays to the user, "Quietly cleans the living room."

[1106] With the above operations, users can operate the cleaning device flexibly and efficiently even from a remote location and check the status in real time. In addition, by incorporating an emotion engine, the device can provide appropriate cleaning operations according to the user's emotions, improving user satisfaction.

[1107] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1108] Program processing steps

[1109] Step 1: User command input

[1110] explanation

[1111] A user uses a user terminal such as a smartphone or tablet to input specific cleaning commands to the cleaning device.

[1112] input

[1113] The user enters a prompt sentence such as "Clean the living room."

[1114] Data processing and calculation

[1115] The terminal receives voice or text input from the user and converts the voice data to text data as needed.

[1116] output

[1117] A textual cleaning command is generated.

[1118] Specific actions

[1119] The user starts the application on the terminal and inputs cleaning commands using voice recognition or keyboard input.

[1120] Step 2: Send the command

[1121] explanation

[1122] The user terminal transmits the input command to the server.

[1123] input

[1124] Textual cleaning commands.

[1125] Data processing and calculation

[1126] The device converts the generated cleaning command into JSON format data and sends it to the server using an HTTP request.

[1127] output

[1128] The request data in JSON format is sent to the server.

[1129] Specific actions

[1130] The device sends an HTTP POST request to the server's API endpoint using Wi-Fi or mobile data.

[1131] Step 3: Command Analysis

[1132] explanation

[1133] The server analyzes the received command and generates specific operating instructions for the cleaning device.

[1134] input

[1135] Cleaning commands in JSON format.

[1136] Data processing and calculation

[1137] The server analyzes the command using a natural language processing engine and generates a specific cleaning area and cleaning route. For example, from the command "Clean the living room," it extracts the coordinate data of the living room and the cleaning route.

[1138] output

[1139] Specific operating instructions for the cleaning device (cleaning area and route information).

[1140] Specific actions

[1141] The server uses an internal text analysis engine to parse the received commands and calculate the appropriate cleaning route.

[1142] Step 4: Sending commands to the cleaning device

[1143] explanation

[1144] Based on the analysis results, the server sends specific cleaning areas and operation instructions to the cleaning device.

[1145] input

[1146] Specific operating instructions for the cleaning device.

[1147] Data processing and calculation

[1148] The server converts the generated operation instructions into a protocol that can be understood by the cleaning device.

[1149] output

[1150] Cleaning instruction data is transmitted in a format that can be understood by the cleaning device.

[1151] Specific actions

[1152] The server uses an HTTP request to send instruction data to the communication module of the cleaning device.

[1153] Step 5: Start cleaning

[1154] explanation

[1155] The cleaning device receives instructions from the server and begins cleaning the designated area.

[1156] input

[1157] Cleaning instruction data.

[1158] Data processing and calculation

[1159] Based on the instructions it receives, the cleaning device works in conjunction with an internal mapping sensor to check its own position while cleaning.

[1160] output

[1161] The cleaning operation begins.

[1162] Specific actions

[1163] The cleaning device activates the operation module and performs cleaning while scanning the designated area.

[1164] Step 6: Check and report status

[1165] explanation

[1166] When a user wants to check the current status of the cleaning device, he sends a request to the server, which retrieves the status information from the cleaning device and returns it to the user terminal.

[1167] input

[1168] Status requests (e.g., "How's the cleaning going?").

[1169] Data processing and calculation

[1170] The server passes the request to the cleaning device, which then returns its current status and location information to the server.

[1171] output

[1172] The current status and location of the cleaning device is displayed on the user terminal.

[1173] Specific actions

[1174] The server receives a request from the user terminal, sends a "status check" instruction to the cleaning device, and returns a response from the cleaning device to the user terminal.

[1175] Step 7: Emotion Recognition

[1176] explanation

[1177] The cleaning device uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[1178] input

[1179] User voice and facial expression data.

[1180] Data processing and calculation

[1181] The cleaning device uses an emotion engine to extract emotions from the user's voice and facial expressions and assign appropriate emotion labels.

[1182] output

[1183] Recognized emotion data (e.g., "fatigue").

[1184] Specific actions

[1185] The cleaning device uses a built-in microphone and camera to analyze the user's emotions and stores the results in its internal memory.

[1186] Step 8: Emotionally Based Behavior Adjustments

[1187] explanation

[1188] The cleaning device adjusts its cleaning behavior based on the recognized emotion.

[1189] input

[1190] Emotional data (e.g., "fatigue").

[1191] Data processing and calculation

[1192] The cleaning device optimizes its operating parameters (volume, speed, etc.) based on emotion data.

[1193] output

[1194] The cleaning operation is performed based on the adjusted operation parameters.

[1195] Specific actions

[1196] The cleaning device switches to a quiet mode and adjusts the cleaning speed to perform operations that are considerate of the user's feelings.

[1197] The above are the specific processing steps and details of the system.

[1198] (Application example 2)

[1199] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1200] Conventional autonomous mobile cleaning devices have problems such as inefficient cleaning instructions and status checks, and inability to adapt to user emotions and store conditions. In particular, cleaning activities in brick-and-mortar stores require frequent cleaning to maintain cleanliness, and streamlining cleaning work is required to reduce the burden on store staff. Furthermore, a system that can appropriately adjust cleaning operations according to the store's congestion status and staff stress levels is desired.

[1201] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, and means for analyzing the command input from the user and reporting the operating status and battery status of the cleaning robot in real time. This allows users to flexibly and efficiently operate the cleaning device from a remote location and check the status in real time. In particular, in physical stores, appropriate cleaning operations can be provided according to the store's congestion status and the staff's stress level, thereby improving the efficiency of cleaning work and reducing the burden on store staff.

[1202] A "generative artificial intelligence model" is an artificial intelligence model that has the ability to generate new data and information based on given data.

[1203] An "autonomous mobile cleaning device" is a device that moves autonomously in response to instructions from a user and cleans a designated area.

[1204] A "user terminal" is a remote control device operated by a user, such as a smartphone, tablet, or PC.

[1205] A "cleaning command" is a command that a user inputs to instruct the autonomously mobile cleaning device to perform cleaning.

[1206] A "status confirmation request" is a request sent by a user to find out the current cleaning status and location information of the autonomously mobile cleaning device.

[1207] The "corresponding cleaning area" is a specific area that the autonomously mobile cleaning device has memorized as a cleaning target.

[1208] An "efficient schedule" is a time and motion plan designed to optimize cleaning tasks for an autonomous mobile cleaning device.

[1209] The "emotion engine" is an engine that has the function of recognizing the user's emotional state from their voice, facial expression, etc., and adjusting cleaning operations according to that state.

[1210] "Battery status" is information indicating the current remaining battery power and operating time of the autonomously mobile cleaning device.

[1211] "Real-time" is a term that refers to information being transmitted and processed almost immediately.

[1212] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[1213] Components

[1214] 1. Cleaning device

[1215] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly. The main hardware used includes a LiDAR sensor, camera, and microphone.

[1216] 2. User Device

[1217] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. This allows the user to remotely issue cleaning commands and check the status. Commands and requests are sent from the user terminal to the server.

[1218] 3. Server

[1219] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes various requests. The server analyzes commands from the cleaning device and reports their status. The main software used includes Node.js and Java. MySQL and PostgreSQL are used as databases.

[1220] System Operation Overview

[1221] 1. User sends command

[1222] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[1223] 2. Receiving and parsing commands

[1224] The user terminal sends the input command to the server, which receives the command and analyzes it to determine the corresponding cleaning action.

[1225] 3. Sending commands to the cleaning device

[1226] Based on the analysis results, the server instructs the cleaning device on the specific cleaning area and operation. Upon receiving the instruction, the cleaning device begins cleaning the specified area.

[1227] 4. Status Check and Reporting

[1228] When a user wants to check the current status of the cleaning device, the user terminal sends a status check request such as "Tell me the cleaning status." The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The cleaning device returns its current status and location information to the server, which then transmits it to the user terminal.

[1229] Overview of Emotion Engine Operation

[1230] 1. User Emotion Recognition

[1231] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[1232] 2. Emotion-based adjustment of cleaning behavior

[1233] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is tired, the cleaning device will change its behavior to clean quietly and quickly.

[1234] 3. Generating Emotional Feedback

[1235] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[1236] Specific examples

[1237] Example 1: Cleaning the living room

[1238] User: Type "Clean the living room" into their smartphone.

[1239] Terminal: Sends input to the server.

[1240] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1241] Cleaning device: Receive instructions and start cleaning the living room.

[1242] Server: Notifies the user device that cleaning has started.

[1243] Device: Displays to the user "Cleaning of the living room has begun."

[1244] Example 2: Checking the cleaning status

[1245] User: Type "Tell me how the cleaning is going" on your smartphone.

[1246] Device: Sends a status check request to the server.

[1247] Server: Transmits the request to the cleaning device.

[1248] Cleaning device: Sends current status and location information back to the server.

[1249] Server: Returns the received information to the user terminal.

[1250] Device: Shows the user "You're currently cleaning the living room."

[1251] Example 3: Adjusting cleaning behavior through emotion recognition

[1252] User: In a tired voice, types, "Clean the living room."

[1253] Terminal: Sends input to the server.

[1254] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1255] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[1256] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[1257] Server: Notifies the user device of emotional feedback from the cleaning device.

[1258] Device: Shows the user "Quietly clean the living room."

[1259] Prompt Sentence Examples

[1260] "It promptly cleans the shopping area specified by the user, reports battery status and current location information in real time, and makes appropriate adjustments based on the user's voice and facial expressions."

[1261] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[1262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1263] Step 1:

[1264] User command sending

[1265] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[1266] Input: Cleaning commands entered by the user.

[1267] Output: A cleaning command is sent from the user terminal to the server.

[1268] In this step, commands are entered through the user terminal interface and sent to the server as internal API requests.

[1269] Step 2:

[1270] Receiving and parsing commands

[1271] The server receives and analyzes the command sent from the user terminal.

[1272] Input: Cleaning command sent from user terminal.

[1273] Output: Analyzed cleaning area and action instructions.

[1274] The server analyzes the command and determines the corresponding cleaning action. This analysis is done using a natural language processing model (e.g., a generative AI model). The analysis result is then sent to the next step.

[1275] Step 3:

[1276] Sending commands to cleaning devices

[1277] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[1278] Input: Analyzed cleaning area and action instructions.

[1279] Output: Operation instructions to the cleaning device.

[1280] The server sends an HTTP request to the cleaning device, informing it of the required cleaning instructions, and the cleaning device begins to operate according to the received instructions.

[1281] Step 4:

[1282] Emotional awareness and regulation

[1283] The cleaning device performs processing to recognize the user's emotions.

[1284] Input: User's voice and facial expressions.

[1285] Output: Adjustment of cleaning behavior based on recognized emotions.

[1286] The cleaning device uses a built-in microphone and camera to recognize the user's emotional state, and the recognized emotional data is processed by an emotion engine to determine an appropriate cleaning plan.

[1287] Step 5:

[1288] Checking and reporting status

[1289] When a user wants to check the current cleaning status, the user sends a request from the user terminal saying "Tell me the cleaning status."

[1290] Input: Status check request from user terminal.

[1291] Output: Current cleaning status and location.

[1292] The server receives this request and queries the cleaning device about its current status. The cleaning device returns the current cleaning status to the server, which then transmits it to the user terminal.

[1293] Step 6:

[1294] Generating Emotional Feedback

[1295] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[1296] Input: Perceived user emotion.

[1297] Output: Emotion feedback notification to the user device.

[1298] The cleaning device generates an appropriate feedback message based on the emotion data processed by the emotion engine and transmits it to the user terminal via the server.

[1299] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[1300] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1301] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1302] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1303] [Third embodiment]

[1304] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1305] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1306] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1307] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1308] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1309] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1310] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1311] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1312] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1313] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1314] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1315] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1316] This system is based on an autonomous mobile cleaning device equipped with a generative artificial intelligence model, and is also composed of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[1317] Components

[1318] 1. Cleaning device

[1319] The cleaning device is a piece of hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[1320] 2. User Device

[1321] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[1322] 3. Server

[1323] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[1324] System Operation Overview

[1325] 1. User sends command

[1326] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[1327] 2. Receiving and parsing commands

[1328] The user terminal transmits the input command to the server.

[1329] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[1330] 3. Sending commands to the cleaning device

[1331] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[1332] Once the cleaning device receives the command, it will begin cleaning the designated area.

[1333] 4. Status Check and Reporting

[1334] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[1335] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[1336] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[1337] Specific examples

[1338] Example 1: Cleaning the living room

[1339] User: Type "Clean the living room" into their smartphone.

[1340] Terminal: Sends input to the server.

[1341] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1342] Cleaning device: Receive instructions and start cleaning the living room.

[1343] Server: Notifies the user device that cleaning has started.

[1344] Device: Displays to the user "Cleaning of the living room has begun."

[1345] Example 2: Checking the cleaning status

[1346] User: Type "Tell me how the cleaning is going" on your smartphone.

[1347] Device: Sends a status check request to the server.

[1348] Server: Transmits the request to the cleaning device.

[1349] Cleaning device: Sends current status and location information back to the server.

[1350] Server: Returns the received information to the user terminal.

[1351] Device: Shows the user "You're currently cleaning the living room."

[1352] This system allows users to operate cleaning devices flexibly and efficiently even from remote locations and check their status in real time, greatly improving the convenience of cleaning work.

[1353] The processing flow will be explained below.

[1354] Step 1:

[1355] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the kitchen."

[1356] Step 2:

[1357] The terminal receives the user's input and sends a request of the form "command: clean area kitchen" to the server.

[1358] Step 3:

[1359] The server receives the request and extracts the command part ("clean area kitchen") through the handle_user_request method.

[1360] Step 4:

[1361] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[1362] Step 5:

[1363] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[1364] Step 6:

[1365] A generative artificial intelligence model analyzes the command "clean area kitchen" and generates the response "Cleaning kitchen."

[1366] Step 7:

[1367] The cleaning device sends the generated response to the server.

[1368] Step 8:

[1369] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Kitchen cleaning has started."

[1370] Step 9:

[1371] The terminal displays the response received from the server on the screen and notifies the user.

[1372] Step 10:

[1373] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[1374] Step 11:

[1375] The device sends a request called "status" to the server.

[1376] Step 12:

[1377] The server receives the status check request and invokes the report_status method of the cleaning device.

[1378] Step 13:

[1379] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., kitchen) to the server via the report_status method.

[1380] Step 14:

[1381] The server returns the received status information to the user terminal.

[1382] Step 15:

[1383] The terminal displays the status information received from the server on the screen and notifies the user, for example, "The kitchen is currently being cleaned."

[1384] Example 1

[1385] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1386] Conventional autonomous mobile cleaning devices have had issues such as difficulty for users to control the cleaning device efficiently from a remote location, insufficient real-time confirmation of the cleaning status, insufficient status reporting, difficulty in providing optimal cleaning plans for specific stains or rooms, and difficulty in responding to questions from users in natural language.

[1387] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1388] In this invention, the server includes a means for analyzing cleaning commands received from a user-operated device and initiating the corresponding cleaning operation, a means for reporting the current cleaning status and location information in response to a status check request from the device, a means for storing the corresponding cleaning area and generating an efficient schedule, a means for incorporating a generative artificial intelligence model and responding to questions from the user in natural language, and a means for communicating data between the user device, the cleaning device, and the server. This allows users to efficiently control the cleaning device and check its status in real time, even from remote locations. It also provides optimal cleaning plans for specific stains or rooms and enables question and answering in natural language, greatly improving the convenience of cleaning work.

[1389] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new information and responses based on data, and is particularly used in natural language processing and image generation.

[1390] An "autonomous mobile cleaning device" is a device that has the function of cleaning while autonomously moving around a specific area, and performs various operations based on instructions from a user.

[1391] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC, and is a means for communicating with a cleaning device or a server.

[1392] The "server" is a central control device that mediates communication between the user terminal and the cleaning device, and processes and analyzes various requests.

[1393] A "cleaning command" is an instruction that a user inputs from a terminal and sends to the cleaning device, and includes instructions to start cleaning or to clean a specific area.

[1394] A "status confirmation request" is a request sent from a terminal by a user to check the current cleaning status and location information of the cleaning device.

[1395] An "efficient schedule" is an optimal cleaning plan generated based on the area to be cleaned and the user's schedule, and is designed to maximize the efficiency and effectiveness of cleaning.

[1396] The "function to answer questions in natural language" is a function that uses a generative artificial intelligence model to generate appropriate responses to questions or instructions from the user in natural language and responds.

[1397] "Data communication" is the process of exchanging information between different devices over a communication network.

[1398] The present invention is a system that mainly comprises an autonomous mobile cleaning device equipped with a generative artificial intelligence model, a user terminal, and a server. The details of each component and the operation of the entire system are explained below.

[1399] Components

[1400] 1. Autonomous mobile cleaning device

[1401] The cleaning device is hardware that can move around autonomously and clean. Using a built-in generative artificial intelligence model (e.g., OpenAI GPT-3), it can respond to cleaning commands and questions from the user in natural language. It also uses sensors (e.g., LIDAR and infrared sensors) to understand the surrounding environment and plan an efficient cleaning route.

[1402] 2. User Device

[1403] The user terminal is a device operated by the user, such as a smartphone, tablet, or PC. The user can issue cleaning commands and check the status through a dedicated application or web interface. Communication is via Wi-Fi or LTE.

[1404] 3. Server

[1405] The server is a central control device that mediates communication between the user device and the cleaning device and processes various requests. It analyzes the received cleaning commands and determines and instructs the specific cleaning operations. The server also reports the current status and location information of the cleaning device to the user device.

[1406] System Operation Overview

[1407] Entering and sending commands

[1408] Users use a device such as a smartphone to input and send cleaning commands, such as "Clean the living room," into a dedicated app or web interface.

[1409] Receiving and parsing commands

[1410] The user terminal transmits the input command to the server.

[1411] The server analyzes the received command using a natural language processing engine (e.g., Google Cloud Natural Language API), inputs specific instructions such as "clean the living room" as a prompt sentence to the generative AI model, and obtains the generated detailed cleaning plan.

[1412] Sending commands to cleaning devices

[1413] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[1414] The cleaning device receives instructions and begins cleaning the designated area (e.g., living room), using built-in generative AI models and sensors to plan and execute an efficient cleaning route.

[1415] Checking and reporting status

[1416] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[1417] The user terminal sends this request to the server.

[1418] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[1419] The cleaning device sends its current status and location information back to the server.

[1420] The server returns the received information to the user terminal.

[1421] The user terminal displays to the user, "Currently cleaning the living room."

[1422] Specific operation example

[1423] Example 1: Cleaning the living room

[1424] The user types "clean the living room" into their smartphone.

[1425] The terminal sends this command to the server.

[1426] The server receives the command, analyzes it, and inputs the "clean the living room" instruction into the generative AI model. Example prompt: "What is the best route to efficiently clean the living room?"

[1427] The server transmits specific instructions to the cleaning device based on the generated cleaning plan.

[1428] The vacuum cleaner receives instructions and begins cleaning the living room, using sensors and generative AI models to clean efficiently.

[1429] Example 2: Checking the cleaning status

[1430] The user types "Tell me about the cleaning status" on their smartphone.

[1431] The terminal sends a status confirmation request to the server.

[1432] The server communicates the request to the cleaning device.

[1433] The cleaning device sends its current status and location information back to the server.

[1434] The server returns the information to the user terminal.

[1435] The device displays to the user, "Currently cleaning the living room."

[1436] With this mechanism, the system of the present invention significantly improves user convenience and enables efficient and flexible cleaning work.

[1437] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1438] Specific processing steps of the program

[1439] Step 1:

[1440] The user enters a command

[1441] Users use a dedicated app on their smartphone or tablet to input cleaning commands such as "clean the living room."

[1442] Input: User's cleaning command (e.g., "Clean the living room")

[1443] Output: The entered command is recorded on the user's terminal and prepared for transmission.

[1444] Step 2:

[1445] Sending commands from the user terminal to the server

[1446] The device sends the entered cleaning command to a server via the internet, using Wi-Fi or LTE for communication.

[1447] Input: Cleaning command entered by the user

[1448] Output: Cleaning command is sent to the server

[1449] Step 3:

[1450] The server receives the command

[1451] The server receives a cleaning command sent from a user terminal.

[1452] Input: Cleaning command sent from user terminal

[1453] Output: The received cleaning command is queued in the server.

[1454] Step 4:

[1455] The server parses the command

[1456] The server analyzes the received cleaning command using a natural language processing engine (e.g., Google Cloud Natural Language API).

[1457] Input: Cleaning command received

[1458] Data processing: A natural language processing engine is used to analyze the meaning of commands and convert them into specific cleaning instructions.

[1459] Output: Specific cleaning instructions as a result of the analysis (e.g., "Start cleaning the living room")

[1460] Step 5:

[1461] The server sends a prompt to the generative AI model

[1462] Based on the analysis results, the server sends prompts to a generative AI model (e.g., OpenAI GPT-3) to generate a detailed cleaning plan.

[1463] Input: Specific cleaning instructions as analysis results

[1464] Data Computation: Generative AI model generates optimal cleaning route and plan based on prompts

[1465] Output: Generated cleaning route and plan

[1466] Step 6:

[1467] The server sends commands to the cleaning device.

[1468] The server instructs the cleaning device on specific cleaning areas and operations based on the generated cleaning route and plan.

[1469] Input: Generated cleaning route and plan

[1470] Output: Specific cleaning instructions are sent to the cleaning device

[1471] Step 7:

[1472] The cleaning device starts cleaning

[1473] Based on the instructions received, the cleaning device begins cleaning the designated area (e.g., the living room), planning and executing an efficient cleaning route using built-in sensors and generative AI models.

[1474] Input: Specific cleaning instructions from the server

[1475] Data calculation: Cleaning is performed while dynamically correcting the actual cleaning route based on sensor information.

[1476] Output: Dust and dirt in the area is removed

[1477] Step 8:

[1478] User submits a status check request

[1479] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[1480] Input: Status Check Request

[1481] Output: The request is logged on the user's terminal and ready to be sent.

[1482] Step 9:

[1483] The user device sends a request to the server

[1484] The terminal sends this request to the server.

[1485] Input: User status check request

[1486] Output: The request is transmitted to the server

[1487] Step 10:

[1488] Server requests status

[1489] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[1490] Input: User status check request

[1491] Output: The current cleaning status and location information request are sent to the cleaning device.

[1492] Step 11:

[1493] The cleaning device sends status information back to the server

[1494] The cleaning device sends its current status and location information back to the server.

[1495] Input: Status information request from the server

[1496] Data calculation: Collects current cleaning progress and location information

[1497] Output: Sending collected status and location information

[1498] Step 12:

[1499] The server sends status information to the user terminal.

[1500] The server returns the received information to the user terminal.

[1501] Input: Status and location information from the cleaning device

[1502] Output: Information is transmitted to the user terminal

[1503] Step 13:

[1504] The user terminal displays the status

[1505] The terminal displays to the user, "Currently cleaning the living room."

[1506] Input: Status and location information from the server

[1507] Output: Status information is displayed to the user

[1508] This allows the user to operate the cleaning device efficiently and in real time even from a remote location and check the current cleaning status.

[1509] (Application example 1)

[1510] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1511] Modern self-driving vehicles have advanced autonomous driving capabilities, but cleaning the interior and exterior of the vehicle and performing regular maintenance require manual labor, which is a hassle for users. Furthermore, diagnosing the vehicle's condition and identifying repair areas requires specialized knowledge, which is inconvenient for users. There is a need to solve these problems and provide a system that efficiently automates the cleaning and maintenance of self-driving vehicles.

[1512] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1513] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status confirmation request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, means for automatically cleaning the interior and exterior of the vehicle and diagnosing the vehicle status, and means for scheduling cleaning maintenance and automatically identifying and reporting areas requiring repairs. This automates vehicle cleaning and maintenance work, allowing users to flexibly manage these tasks from a remote location.

[1514] A "generative artificial intelligence model" is an artificial intelligence technique that can generate new information and content from data.

[1515] An "autonomous mobile cleaning device" is a hardware device that can clean while moving around on its own.

[1516] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[1517] A "cleaning command" is an instruction or command given by a user to a cleaning device.

[1518] A "status confirmation request" is a request made by the user to confirm the current status and location information of the cleaning device.

[1519] A "cleaning zone" is a particular area that a cleaning device is to clean.

[1520] The "means for generating a schedule" is a function that determines the time and order for the cleaning device to perform cleaning work efficiently.

[1521] "Automatic cleaning inside and outside the vehicle" is a function that automatically cleans the inside and outside of the vehicle.

[1522] "Means for diagnosing the condition of the vehicle" refers to a function that checks the condition of each part and system of the vehicle and detects abnormalities.

[1523] "Cleaning and maintenance scheduling" is a function that allows you to create a schedule for systematically cleaning and maintaining your vehicle.

[1524] The "means for automatically identifying and reporting parts requiring repair" is a function for automatically detecting parts of a vehicle that are faulty and reporting the same to the user.

[1525] This invention is a system that automates cleaning and maintenance work for self-driving vehicles using an autonomous mobile cleaning device equipped with a generative artificial intelligence model. The system is composed of a user terminal, a server, and the autonomous mobile cleaning device.

[1526] Components

[1527] 1. Autonomous mobile cleaning device

[1528] The cleaning device is a hardware device that moves autonomously and cleans the inside and outside of a vehicle. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[1529] 2. User Device

[1530] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. Using this terminal, the user can remotely issue cleaning commands and check the status.

[1531] 3. Server

[1532] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[1533] System Operation Overview

[1534] 1. User sends command

[1535] Users input and send cleaning commands such as "clean the interior of the car" or "perform a maintenance check" using smart glasses or a smartphone.

[1536] 2. Receiving and parsing commands

[1537] The server receives commands sent from the user terminal, analyzes the contents of the commands, and then identifies cleaning areas and diagnostic points inside and outside the vehicle.

[1538] 3. Sending commands to the cleaning device

[1539] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving the instructions, the cleaning device begins cleaning the designated area. At the same time, it also performs a vehicle condition diagnosis.

[1540] 4. Status Check and Reporting

[1541] When a user wants to check the current status of the cleaning device, the user sends a command such as "Tell me the current situation" from the user terminal. The server transmits this request to the cleaning device, collects the current cleaning status and location information, and reports it to the user terminal.

[1542] Hardware and software used

[1543] Hardware:

[1544] Autonomous mobile cleaning device mounted on a self-driving vehicle

[1545] Vehicle condition diagnostic sensors

[1546] software:

[1547] Python

[1548] Flask (server-side request processing)

[1549] Generative AI model

[1550] Specific examples

[1551] 1. Specific scenario

[1552] Sending cleaning commands:

[1553] A user types into a smartphone app, "Start cleaning the car interior at 9:00 tomorrow morning."

[1554] Request a maintenance check:

[1555] When getting into the vehicle, enter the command "Tell me where there is a risk of breakdown" to obtain the necessary information.

[1556] 2. Example prompts for generative AI models

[1557] "Schedule my car to be cleaned tomorrow at 9am"

[1558] "Perform a maintenance check and report any abnormalities."

[1559] This system enables users to efficiently manage cleaning and maintenance work on autonomous vehicles from remote locations, significantly reducing the burden on users.

[1560] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1561] Step 1:

[1562] The user uses a user terminal such as a smartphone to input and send cleaning commands and maintenance check instructions.

[1563] Input: The user enters a prompt statement such as "Start cleaning the car tomorrow morning at 9am."

[1564] Output: Commands are sent from the user terminal to the server.

[1565] Step 2:

[1566] The server analyzes the commands received from the user terminal and determines the corresponding cleaning actions and maintenance checks.

[1567] Input: Cleaning command data sent from the user terminal.

[1568] Data processing / calculation: The server uses a generative artificial intelligence model to analyze and interpret commands in natural language.

[1569] Output: The interpreted commands are directed to specific cleaning actions or maintenance checks.

[1570] Step 3:

[1571] Based on the analysis results, the server instructs the autonomous mobile cleaning device on specific cleaning areas and maintenance operations.

[1572] Input: Parsed cleaning actions and maintenance check instructions.

[1573] Output: Specific cleaning area and operation instructions for the cleaning device.

[1574] Step 4:

[1575] The autonomous mobile cleaning device begins cleaning in the designated area and diagnoses the condition of the inside and outside of the vehicle.

[1576] Input: Cleaning and diagnostic instructions sent from the server.

[1577] Specific operation: The cleaning device locates itself and starts cleaning, while at the same time checking for any abnormalities in the vehicle using a diagnostic sensor.

[1578] Output: Cleaning progress and vehicle status data.

[1579] Step 5:

[1580] The cleaning device and diagnostic sensors transmit collected data to a server.

[1581] Input: Cleaning progress and vehicle condition data.

[1582] Data processing / calculation: The server analyzes these data and generates information to report to the user.

[1583] Output: Parsed cleaning and diagnostic result data.

[1584] Step 6:

[1585] The server reports the current status and diagnostic results of the cleaning device to the user terminal.

[1586] Input: Cleaning progress and vehicle condition data.

[1587] Specific operation: The server organizes these data and sends them to the user terminal.

[1588] Output: Reports to the user device such as "Cleaning is complete" or "There are areas at risk of failure."

[1589] Step 7:

[1590] The user checks the report results on a smartphone or smart glasses and gives additional instructions if necessary.

[1591] Input: Report message from the server.

[1592] Specific operation: The user checks the report results and inputs new cleaning commands or maintenance instructions.

[1593] Output: New cleaning commands or maintenance instructions are sent to the server again.

[1594] Through these steps, users can efficiently and automatically manage the cleaning and maintenance of their autonomous vehicles.

[1595] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1596] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[1597] Components

[1598] 1. Cleaning device

[1599] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly.

[1600] 2. User Device

[1601] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[1602] 3. Server

[1603] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[1604] System Operation Overview

[1605] 1. User sends command

[1606] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[1607] 2. Receiving and parsing commands

[1608] The user terminal transmits the input command to the server.

[1609] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[1610] 3. Sending commands to the cleaning device

[1611] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[1612] Once the cleaning device receives the command, it will begin cleaning the designated area.

[1613] 4. Status Check and Reporting

[1614] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[1615] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[1616] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[1617] Overview of Emotion Engine Operation

[1618] 1. User Emotion Recognition

[1619] The cleaning device is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions.

[1620] For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[1621] 2. Emotion-based adjustment of cleaning behavior

[1622] The cleaning device adjusts its cleaning schedule and frequency based on the emotions it recognizes.

[1623] For example, if the user is frustrated, the cleaning device may change its operation to finish cleaning quietly and quickly.

[1624] 3. Generating Emotional Feedback

[1625] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[1626] For example, a notification such as "The user is tired, so the living room will be cleaned quietly" is sent to the user terminal.

[1627] Specific examples

[1628] Example 1: Cleaning the living room

[1629] User: Type "Clean the living room" into their smartphone.

[1630] Terminal: Sends input to the server.

[1631] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1632] Cleaning device: Receive instructions and start cleaning the living room.

[1633] Server: Notifies the user device that cleaning has started.

[1634] Device: Displays to the user "Cleaning of the living room has begun."

[1635] Example 2: Checking the cleaning status

[1636] User: Type "Tell me how the cleaning is going" on your smartphone.

[1637] Device: Sends a status check request to the server.

[1638] Server: Transmits the request to the cleaning device.

[1639] Cleaning device: Sends current status and location information back to the server.

[1640] Server: Returns the received information to the user terminal.

[1641] Device: Shows the user "You're currently cleaning the living room."

[1642] Example 3: Adjusting cleaning behavior through emotion recognition

[1643] User: In a tired voice, types, "Clean the living room."

[1644] Terminal: Sends input to the server.

[1645] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1646] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[1647] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[1648] Server: Notifies the user device of emotional feedback from the cleaning device.

[1649] Device: Shows the user "Quietly clean the living room."

[1650] This system allows users to operate the cleaning device flexibly and efficiently even from a remote location, and check the status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning operations according to the user's emotions, thereby improving user satisfaction.

[1651] The processing flow will be explained below.

[1652] Step 1:

[1653] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the living room."

[1654] Step 2:

[1655] The terminal receives the user's input and sends a request of the form "command: clean area living room" to the server.

[1656] Step 3:

[1657] The server receives the request and extracts the command part ("clean area living room") through the handle_user_request method.

[1658] Step 4:

[1659] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[1660] Step 5:

[1661] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[1662] Step 6:

[1663] A generative artificial intelligence model analyzes the command and generates the response "Cleaning living room."

[1664] Step 7:

[1665] The cleaning device sends the generated response to the server.

[1666] Step 8:

[1667] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Cleaning of the living room has started."

[1668] Step 9:

[1669] The terminal displays the response received from the server on the screen and notifies the user.

[1670] Step 10:

[1671] The cleaning device uses a built-in emotion engine to recognize emotions from the user's voice and facial expressions. For example, if the user says "Clean up" in a tired voice, the emotion engine will recognize this as "fatigue."

[1672] Step 11:

[1673] The cleaning device automatically adjusts its cleaning plan based on the emotions it recognizes, for example, recognizing fatigue and changing settings to clean with reduced noise.

[1674] Step 12:

[1675] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[1676] Step 13:

[1677] The device sends a request called "status" to the server.

[1678] Step 14:

[1679] The server receives the status check request and invokes the report_status method of the cleaning device.

[1680] Step 15:

[1681] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., living room) to the server via the report_status method.

[1682] Step 16:

[1683] The server returns the received status information to the user terminal.

[1684] Step 17:

[1685] The device displays the status information received from the server on the screen and notifies the user, for example, "The living room is currently being cleaned."

[1686] Step 18:

[1687] The cleaning device generates feedback from the emotion engine and notifies the user terminal via the server. For example, a message such as "Because the user is tired, the living room will be cleaned quietly" is displayed.

[1688] Example 2

[1689] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1690] Conventional autonomous mobile cleaning devices simply receive and execute cleaning commands from the user, making it difficult to respond flexibly to emotions and situations. Furthermore, they often fail to efficiently check the status of the cleaning device or address specific stains in the room. Furthermore, because the cleaning device lacks the ability to understand the user's emotions and adapt its behavior based on those emotions, it is unable to provide a satisfactory service to some users.

[1691] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1692] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, and means including an emotion engine for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, thereby enabling real-time response to commands from the user and flexible cleaning operation according to the user's emotions.

[1693] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new content or results from data, and is applied to tasks such as natural language processing and image generation.

[1694] An "autonomous mobile cleaning device" is a cleaning device that has the ability to clean while moving around on its own, and is equipped with sensors, cameras, and mapping functions inside.

[1695] A "user terminal" is a device operated by a user, and includes smartphones, tablets, personal computers, etc.

[1696] The "server" is a central control unit that handles communication and data processing between the user terminal and the cleaning device, and has functions such as command analysis, instruction transmission, and status management.

[1697] An "emotion engine" refers to an algorithm or system that recognizes emotions from a user's voice and facial expressions and adjusts the device's behavior based on those emotions.

[1698] A "status check request" is a request sent by a user to check the current operating status and location information of the cleaning device.

[1699] A "cleaning command" is an instruction that a user sends to a cleaning device to instruct the device to perform a specific cleaning operation.

[1700] "Cleaning area" refers to the particular location or room in which the cleaning device performs cleaning.

[1701] An "efficient schedule" refers to a work plan generated to allow the cleaning device to clean as efficiently as possible, minimizing waste of time and energy.

[1702] The present invention is a system that mainly consists of an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and is composed of a user terminal and a server. The details of each component and the operation of the entire system are explained below.

[1703] Components

[1704] 1. Autonomous mobile cleaning device

[1705] The cleaning device is a piece of hardware that has a built-in generative artificial intelligence model and emotion engine, and moves around autonomously to clean. The device is equipped with sensors, cameras, and mapping functions, and operates efficiently within a designated cleaning area.

[1706] 2. User Device

[1707] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, which allows the user to send cleaning commands and check the status remotely.

[1708] 3. Server

[1709] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes data. The server analyzes commands from the user and sends specific operational instructions to the cleaning device. It also monitors the operating status and location information of the cleaning device.

[1710] System Operation

[1711] User command sending

[1712] A user inputs and sends a cleaning command such as "Clean the living room" from their smartphone. The user device then sends the command to the server.

[1713] Receiving and parsing commands

[1714] The server receives commands sent from the user terminal and analyzes the content using a text analysis engine. For example, if the command "clean the living room" is received, the server generates a cleaning area and an action plan based on the content.

[1715] Sending commands to cleaning devices

[1716] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving these instructions, the cleaning device begins cleaning the specified area.

[1717] Checking and reporting status

[1718] When a user wants to check the current status of the cleaning device, they send a status check request such as "Tell me the cleaning status" from the user terminal. The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The server then returns the received information to the user terminal.

[1719] Emotion Engine Operation

[1720] User Emotion Recognition

[1721] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[1722] Emotion-based adjustment of cleaning behavior

[1723] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is irritated, the cleaning device will change its behavior to clean quietly and quickly.

[1724] Generating Emotional Feedback

[1725] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[1726] Specific examples

[1727] Example 1: Cleaning the living room

[1728] The user types "clean the living room" into their smartphone.

[1729] The terminal sends this command to the server.

[1730] The server parses the command and generates the instruction "Clean the living room."

[1731] The cleaning device receives the parsed instructions and begins cleaning the living room.

[1732] The server notifies the user terminal of the status from the cleaning device and displays to the user, "Cleaning of the living room has started."

[1733] Example 2: Checking the cleaning status

[1734] The user types "Tell me about the cleaning status" on their smartphone.

[1735] The terminal sends a status confirmation request to the server.

[1736] The server communicates a request to the cleaning device to obtain the current status and location of the cleaning device.

[1737] The server returns the received information to the user terminal and displays to the user, "Currently cleaning the living room."

[1738] Example 3: Adjusting cleaning behavior through emotion recognition

[1739] The user types in a tired voice, "Clean the living room."

[1740] The terminal sends this input to the server.

[1741] The server analyzes the command and sends an instruction to "clean the living room" to the cleaning device.

[1742] The cleaning device receives instructions and the emotion engine recognizes the user's fatigue.

[1743] The cleaning device is adjusted to complete the cleaning operation quietly and quickly.

[1744] The server notifies the user terminal of the emotion feedback from the cleaning device and displays to the user, "Quietly cleans the living room."

[1745] With the above operations, users can operate the cleaning device flexibly and efficiently even from a remote location and check the status in real time. In addition, by incorporating an emotion engine, the device can provide appropriate cleaning operations according to the user's emotions, improving user satisfaction.

[1746] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1747] Program processing steps

[1748] Step 1: User command input

[1749] explanation

[1750] A user uses a user terminal such as a smartphone or tablet to input specific cleaning commands to the cleaning device.

[1751] input

[1752] The user enters a prompt sentence such as "Clean the living room."

[1753] Data processing and calculation

[1754] The terminal receives voice or text input from the user and converts the voice data to text data as needed.

[1755] output

[1756] A textual cleaning command is generated.

[1757] Specific actions

[1758] The user starts the application on the terminal and inputs cleaning commands using voice recognition or keyboard input.

[1759] Step 2: Send the command

[1760] explanation

[1761] The user terminal transmits the input command to the server.

[1762] input

[1763] Textual cleaning commands.

[1764] Data processing and calculation

[1765] The device converts the generated cleaning command into JSON format data and sends it to the server using an HTTP request.

[1766] output

[1767] The request data in JSON format is sent to the server.

[1768] Specific actions

[1769] The device sends an HTTP POST request to the server's API endpoint using Wi-Fi or mobile data.

[1770] Step 3: Command Analysis

[1771] explanation

[1772] The server analyzes the received command and generates specific operating instructions for the cleaning device.

[1773] input

[1774] Cleaning commands in JSON format.

[1775] Data processing and calculation

[1776] The server analyzes the command using a natural language processing engine and generates a specific cleaning area and cleaning route. For example, from the command "Clean the living room," it extracts the coordinate data of the living room and the cleaning route.

[1777] output

[1778] Specific operating instructions for the cleaning device (cleaning area and route information).

[1779] Specific actions

[1780] The server uses an internal text analysis engine to parse the received commands and calculate the appropriate cleaning route.

[1781] Step 4: Sending commands to the cleaning device

[1782] explanation

[1783] Based on the analysis results, the server sends specific cleaning areas and operation instructions to the cleaning device.

[1784] input

[1785] Specific operating instructions for the cleaning device.

[1786] Data processing and calculation

[1787] The server converts the generated operation instructions into a protocol that can be understood by the cleaning device.

[1788] output

[1789] Cleaning instruction data is transmitted in a format that can be understood by the cleaning device.

[1790] Specific actions

[1791] The server uses an HTTP request to send instruction data to the communication module of the cleaning device.

[1792] Step 5: Start cleaning

[1793] explanation

[1794] The cleaning device receives instructions from the server and begins cleaning the designated area.

[1795] input

[1796] Cleaning instruction data.

[1797] Data processing and calculation

[1798] Based on the instructions it receives, the cleaning device works in conjunction with an internal mapping sensor to check its own position while cleaning.

[1799] output

[1800] The cleaning operation begins.

[1801] Specific actions

[1802] The cleaning device activates the operation module and performs cleaning while scanning the designated area.

[1803] Step 6: Check and report status

[1804] explanation

[1805] When a user wants to check the current status of the cleaning device, he sends a request to the server, which retrieves the status information from the cleaning device and returns it to the user terminal.

[1806] input

[1807] Status requests (e.g., "How's the cleaning going?").

[1808] Data processing and calculation

[1809] The server passes the request to the cleaning device, which then returns its current status and location information to the server.

[1810] output

[1811] The current status and location of the cleaning device is displayed on the user terminal.

[1812] Specific actions

[1813] The server receives a request from the user terminal, sends a "status check" instruction to the cleaning device, and returns a response from the cleaning device to the user terminal.

[1814] Step 7: Emotion Recognition

[1815] explanation

[1816] The cleaning device uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[1817] input

[1818] User voice and facial expression data.

[1819] Data processing and calculation

[1820] The cleaning device uses an emotion engine to extract emotions from the user's voice and facial expressions and assign appropriate emotion labels.

[1821] output

[1822] Recognized emotion data (e.g., "fatigue").

[1823] Specific actions

[1824] The cleaning device uses a built-in microphone and camera to analyze the user's emotions and stores the results in its internal memory.

[1825] Step 8: Emotionally Based Behavior Adjustments

[1826] explanation

[1827] The cleaning device adjusts its cleaning behavior based on the recognized emotion.

[1828] input

[1829] Emotional data (e.g., "fatigue").

[1830] Data processing and calculation

[1831] The cleaning device optimizes its operating parameters (volume, speed, etc.) based on emotion data.

[1832] output

[1833] The cleaning operation is performed based on the adjusted operation parameters.

[1834] Specific actions

[1835] The cleaning device switches to a quiet mode and adjusts the cleaning speed to perform operations that are considerate of the user's feelings.

[1836] The above are the specific processing steps and details of the system.

[1837] (Application example 2)

[1838] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1839] Conventional autonomous mobile cleaning devices have problems such as inefficient cleaning instructions and status checks, and inability to adapt to user emotions and store conditions. In particular, cleaning activities in brick-and-mortar stores require frequent cleaning to maintain cleanliness, and streamlining cleaning work is required to reduce the burden on store staff. Furthermore, a system that can appropriately adjust cleaning operations according to the store's congestion status and staff stress levels is desired.

[1840] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, and means for analyzing the command input from the user and reporting the operating status and battery status of the cleaning robot in real time. This allows users to flexibly and efficiently operate the cleaning device from a remote location and check the status in real time. In particular, in physical stores, appropriate cleaning operations can be provided according to the store's congestion status and the staff's stress level, thereby improving the efficiency of cleaning work and reducing the burden on store staff.

[1841] A "generative artificial intelligence model" is an artificial intelligence model that has the ability to generate new data and information based on given data.

[1842] An "autonomous mobile cleaning device" is a device that moves autonomously in response to instructions from a user and cleans a designated area.

[1843] A "user terminal" is a remote control device operated by a user, such as a smartphone, tablet, or PC.

[1844] A "cleaning command" is a command that a user inputs to instruct the autonomously mobile cleaning device to perform cleaning.

[1845] A "status confirmation request" is a request sent by a user to find out the current cleaning status and location information of the autonomously mobile cleaning device.

[1846] The "corresponding cleaning area" is a specific area that the autonomously mobile cleaning device has memorized as a cleaning target.

[1847] An "efficient schedule" is a time and motion plan designed to optimize cleaning tasks for an autonomous mobile cleaning device.

[1848] The "emotion engine" is an engine that has the function of recognizing the user's emotional state from their voice, facial expression, etc., and adjusting cleaning operations according to that state.

[1849] "Battery status" is information indicating the current remaining battery power and operating time of the autonomously mobile cleaning device.

[1850] "Real-time" is a term that refers to information being transmitted and processed almost immediately.

[1851] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[1852] Components

[1853] 1. Cleaning device

[1854] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly. The main hardware used includes a LiDAR sensor, camera, and microphone.

[1855] 2. User Device

[1856] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. This allows the user to remotely issue cleaning commands and check the status. Commands and requests are sent from the user terminal to the server.

[1857] 3. Server

[1858] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes various requests. The server analyzes commands from the cleaning device and reports their status. The main software used includes Node.js and Java. MySQL and PostgreSQL are used as databases.

[1859] System Operation Overview

[1860] 1. User sends command

[1861] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[1862] 2. Receiving and parsing commands

[1863] The user terminal sends the input command to the server, which receives the command and analyzes it to determine the corresponding cleaning action.

[1864] 3. Sending commands to the cleaning device

[1865] Based on the analysis results, the server instructs the cleaning device on the specific cleaning area and operation. Upon receiving the instruction, the cleaning device begins cleaning the specified area.

[1866] 4. Status Check and Reporting

[1867] When a user wants to check the current status of the cleaning device, the user terminal sends a status check request such as "Tell me the cleaning status." The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The cleaning device returns its current status and location information to the server, which then transmits it to the user terminal.

[1868] Overview of Emotion Engine Operation

[1869] 1. User Emotion Recognition

[1870] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[1871] 2. Emotion-based adjustment of cleaning behavior

[1872] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is tired, the cleaning device will change its behavior to clean quietly and quickly.

[1873] 3. Generating Emotional Feedback

[1874] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[1875] Specific examples

[1876] Example 1: Cleaning the living room

[1877] User: Type "Clean the living room" into their smartphone.

[1878] Terminal: Sends input to the server.

[1879] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1880] Cleaning device: Receive instructions and start cleaning the living room.

[1881] Server: Notifies the user device that cleaning has started.

[1882] Device: Displays to the user "Cleaning of the living room has begun."

[1883] Example 2: Checking the cleaning status

[1884] User: Type "Tell me how the cleaning is going" on your smartphone.

[1885] Device: Sends a status check request to the server.

[1886] Server: Transmits the request to the cleaning device.

[1887] Cleaning device: Sends current status and location information back to the server.

[1888] Server: Returns the received information to the user terminal.

[1889] Device: Shows the user "You're currently cleaning the living room."

[1890] Example 3: Adjusting cleaning behavior through emotion recognition

[1891] User: In a tired voice, types, "Clean the living room."

[1892] Terminal: Sends input to the server.

[1893] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1894] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[1895] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[1896] Server: Notifies the user device of emotional feedback from the cleaning device.

[1897] Device: Shows the user "Quietly clean the living room."

[1898] Prompt Sentence Examples

[1899] "It promptly cleans the shopping area specified by the user, reports battery status and current location information in real time, and makes appropriate adjustments based on the user's voice and facial expressions."

[1900] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[1901] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1902] Step 1:

[1903] User command sending

[1904] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[1905] Input: Cleaning commands entered by the user.

[1906] Output: A cleaning command is sent from the user terminal to the server.

[1907] In this step, commands are entered through the user terminal interface and sent to the server as internal API requests.

[1908] Step 2:

[1909] Receiving and parsing commands

[1910] The server receives and analyzes the command sent from the user terminal.

[1911] Input: Cleaning command sent from user terminal.

[1912] Output: Analyzed cleaning area and action instructions.

[1913] The server analyzes the command and determines the corresponding cleaning action. This analysis is done using a natural language processing model (e.g., a generative AI model). The analysis result is then sent to the next step.

[1914] Step 3:

[1915] Sending commands to cleaning devices

[1916] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[1917] Input: Analyzed cleaning area and action instructions.

[1918] Output: Operation instructions to the cleaning device.

[1919] The server sends an HTTP request to the cleaning device, informing it of the required cleaning instructions, and the cleaning device begins to operate according to the received instructions.

[1920] Step 4:

[1921] Emotional awareness and regulation

[1922] The cleaning device performs processing to recognize the user's emotions.

[1923] Input: User's voice and facial expressions.

[1924] Output: Adjustment of cleaning behavior based on recognized emotions.

[1925] The cleaning device uses a built-in microphone and camera to recognize the user's emotional state, and the recognized emotional data is processed by an emotion engine to determine an appropriate cleaning plan.

[1926] Step 5:

[1927] Checking and reporting status

[1928] When a user wants to check the current cleaning status, the user sends a request from the user terminal saying "Tell me the cleaning status."

[1929] Input: Status check request from user terminal.

[1930] Output: Current cleaning status and location.

[1931] The server receives this request and queries the cleaning device about its current status. The cleaning device returns the current cleaning status to the server, which then transmits it to the user terminal.

[1932] Step 6:

[1933] Generating Emotional Feedback

[1934] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[1935] Input: Perceived user emotion.

[1936] Output: Emotion feedback notification to the user device.

[1937] The cleaning device generates an appropriate feedback message based on the emotion data processed by the emotion engine and transmits it to the user terminal via the server.

[1938] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[1939] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1940] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1941] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1942] [Fourth embodiment]

[1943] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1944] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1945] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1946] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1947] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1948] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1949] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1950] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1951] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1952] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1953] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1954] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1955] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1956] This system is based on an autonomous mobile cleaning device equipped with a generative artificial intelligence model, and is also composed of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[1957] Components

[1958] 1. Cleaning device

[1959] The cleaning device is a piece of hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[1960] 2. User Device

[1961] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[1962] 3. Server

[1963] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[1964] System Operation Overview

[1965] 1. User sends command

[1966] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[1967] 2. Receiving and parsing commands

[1968] The user terminal transmits the input command to the server.

[1969] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[1970] 3. Sending commands to the cleaning device

[1971] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[1972] Once the cleaning device receives the command, it will begin cleaning the designated area.

[1973] 4. Status Check and Reporting

[1974] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[1975] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[1976] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[1977] Specific examples

[1978] Example 1: Cleaning the living room

[1979] User: Type "Clean the living room" into their smartphone.

[1980] Terminal: Sends input to the server.

[1981] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[1982] Cleaning device: Receive instructions and start cleaning the living room.

[1983] Server: Notifies the user device that cleaning has started.

[1984] Device: Displays to the user "Cleaning of the living room has begun."

[1985] Example 2: Checking the cleaning status

[1986] User: Type "Tell me how the cleaning is going" on your smartphone.

[1987] Device: Sends a status check request to the server.

[1988] Server: Transmits the request to the cleaning device.

[1989] Cleaning device: Sends current status and location information back to the server.

[1990] Server: Returns the received information to the user terminal.

[1991] Device: Shows the user "You're currently cleaning the living room."

[1992] This system allows users to operate cleaning devices flexibly and efficiently even from remote locations and check their status in real time, greatly improving the convenience of cleaning work.

[1993] The processing flow will be explained below.

[1994] Step 1:

[1995] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the kitchen."

[1996] Step 2:

[1997] The terminal receives the user's input and sends a request of the form "command: clean area kitchen" to the server.

[1998] Step 3:

[1999] The server receives the request and extracts the command part ("clean area kitchen") through the handle_user_request method.

[2000] Step 4:

[2001] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[2002] Step 5:

[2003] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[2004] Step 6:

[2005] A generative artificial intelligence model analyzes the command "clean area kitchen" and generates the response "Cleaning kitchen."

[2006] Step 7:

[2007] The cleaning device sends the generated response to the server.

[2008] Step 8:

[2009] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Kitchen cleaning has started."

[2010] Step 9:

[2011] The terminal displays the response received from the server on the screen and notifies the user.

[2012] Step 10:

[2013] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[2014] Step 11:

[2015] The device sends a request called "status" to the server.

[2016] Step 12:

[2017] The server receives the status check request and invokes the report_status method of the cleaning device.

[2018] Step 13:

[2019] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., kitchen) to the server via the report_status method.

[2020] Step 14:

[2021] The server returns the received status information to the user terminal.

[2022] Step 15:

[2023] The terminal displays the status information received from the server on the screen and notifies the user, for example, "The kitchen is currently being cleaned."

[2024] Example 1

[2025] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2026] Conventional autonomous mobile cleaning devices have had issues such as difficulty for users to control the cleaning device efficiently from a remote location, insufficient real-time confirmation of the cleaning status, insufficient status reporting, difficulty in providing optimal cleaning plans for specific stains or rooms, and difficulty in responding to questions from users in natural language.

[2027] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2028] In this invention, the server includes a means for analyzing cleaning commands received from a user-operated device and initiating the corresponding cleaning operation, a means for reporting the current cleaning status and location information in response to a status check request from the device, a means for storing the corresponding cleaning area and generating an efficient schedule, a means for incorporating a generative artificial intelligence model and responding to questions from the user in natural language, and a means for communicating data between the user device, the cleaning device, and the server. This allows users to efficiently control the cleaning device and check its status in real time, even from remote locations. It also provides optimal cleaning plans for specific stains or rooms and enables question and answering in natural language, greatly improving the convenience of cleaning work.

[2029] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new information and responses based on data, and is particularly used in natural language processing and image generation.

[2030] An "autonomous mobile cleaning device" is a device that has the function of cleaning while autonomously moving around a specific area, and performs various operations based on instructions from a user.

[2031] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC, and is a means for communicating with a cleaning device or a server.

[2032] The "server" is a central control device that mediates communication between the user terminal and the cleaning device, and processes and analyzes various requests.

[2033] A "cleaning command" is an instruction that a user inputs from a terminal and sends to the cleaning device, and includes instructions to start cleaning or to clean a specific area.

[2034] A "status confirmation request" is a request sent from a terminal by a user to check the current cleaning status and location information of the cleaning device.

[2035] An "efficient schedule" is an optimal cleaning plan generated based on the area to be cleaned and the user's schedule, and is designed to maximize the efficiency and effectiveness of cleaning.

[2036] The "function to answer questions in natural language" is a function that uses a generative artificial intelligence model to generate appropriate responses to questions or instructions from the user in natural language and responds.

[2037] "Data communication" is the process of exchanging information between different devices over a communication network.

[2038] The present invention is a system that mainly comprises an autonomous mobile cleaning device equipped with a generative artificial intelligence model, a user terminal, and a server. The details of each component and the operation of the entire system are explained below.

[2039] Components

[2040] 1. Autonomous mobile cleaning device

[2041] The cleaning device is hardware that can move around autonomously and clean. Using a built-in generative artificial intelligence model (e.g., OpenAI GPT-3), it can respond to cleaning commands and questions from the user in natural language. It also uses sensors (e.g., LIDAR and infrared sensors) to understand the surrounding environment and plan an efficient cleaning route.

[2042] 2. User Device

[2043] The user terminal is a device operated by the user, such as a smartphone, tablet, or PC. The user can issue cleaning commands and check the status through a dedicated application or web interface. Communication is via Wi-Fi or LTE.

[2044] 3. Server

[2045] The server is a central control device that mediates communication between the user device and the cleaning device and processes various requests. It analyzes the received cleaning commands and determines and instructs the specific cleaning operations. The server also reports the current status and location information of the cleaning device to the user device.

[2046] System Operation Overview

[2047] Entering and sending commands

[2048] Users use a device such as a smartphone to input and send cleaning commands, such as "Clean the living room," into a dedicated app or web interface.

[2049] Receiving and parsing commands

[2050] The user terminal transmits the input command to the server.

[2051] The server analyzes the received command using a natural language processing engine (e.g., Google Cloud Natural Language API), inputs specific instructions such as "clean the living room" as a prompt sentence to the generative AI model, and obtains the generated detailed cleaning plan.

[2052] Sending commands to cleaning devices

[2053] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[2054] The cleaning device receives instructions and begins cleaning the designated area (e.g., living room), using built-in generative AI models and sensors to plan and execute an efficient cleaning route.

[2055] Checking and reporting status

[2056] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[2057] The user terminal sends this request to the server.

[2058] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[2059] The cleaning device sends its current status and location information back to the server.

[2060] The server returns the received information to the user terminal.

[2061] The user terminal displays to the user, "Currently cleaning the living room."

[2062] Specific operation example

[2063] Example 1: Cleaning the living room

[2064] The user types "clean the living room" into their smartphone.

[2065] The terminal sends this command to the server.

[2066] The server receives the command, analyzes it, and inputs the "clean the living room" instruction into the generative AI model. Example prompt: "What is the best route to efficiently clean the living room?"

[2067] The server transmits specific instructions to the cleaning device based on the generated cleaning plan.

[2068] The vacuum cleaner receives instructions and begins cleaning the living room, using sensors and generative AI models to clean efficiently.

[2069] Example 2: Checking the cleaning status

[2070] The user types "Tell me about the cleaning status" on their smartphone.

[2071] The terminal sends a status confirmation request to the server.

[2072] The server communicates the request to the cleaning device.

[2073] The cleaning device sends its current status and location information back to the server.

[2074] The server returns the information to the user terminal.

[2075] The device displays to the user, "Currently cleaning the living room."

[2076] With this mechanism, the system of the present invention significantly improves user convenience and enables efficient and flexible cleaning work.

[2077] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2078] Specific processing steps of the program

[2079] Step 1:

[2080] The user enters a command

[2081] Users use a dedicated app on their smartphone or tablet to input cleaning commands such as "clean the living room."

[2082] Input: User's cleaning command (e.g., "Clean the living room")

[2083] Output: The entered command is recorded on the user's terminal and prepared for transmission.

[2084] Step 2:

[2085] Sending commands from the user terminal to the server

[2086] The device sends the entered cleaning command to a server via the internet, using Wi-Fi or LTE for communication.

[2087] Input: Cleaning command entered by the user

[2088] Output: Cleaning command is sent to the server

[2089] Step 3:

[2090] The server receives the command

[2091] The server receives a cleaning command sent from a user terminal.

[2092] Input: Cleaning command sent from user terminal

[2093] Output: The received cleaning command is queued in the server.

[2094] Step 4:

[2095] The server parses the command

[2096] The server analyzes the received cleaning command using a natural language processing engine (e.g., Google Cloud Natural Language API).

[2097] Input: Cleaning command received

[2098] Data processing: A natural language processing engine is used to analyze the meaning of commands and convert them into specific cleaning instructions.

[2099] Output: Specific cleaning instructions as a result of the analysis (e.g., "Start cleaning the living room")

[2100] Step 5:

[2101] The server sends a prompt to the generative AI model

[2102] Based on the analysis results, the server sends prompts to a generative AI model (e.g., OpenAI GPT-3) to generate a detailed cleaning plan.

[2103] Input: Specific cleaning instructions as analysis results

[2104] Data Computation: Generative AI model generates optimal cleaning route and plan based on prompts

[2105] Output: Generated cleaning route and plan

[2106] Step 6:

[2107] The server sends commands to the cleaning device.

[2108] The server instructs the cleaning device on specific cleaning areas and operations based on the generated cleaning route and plan.

[2109] Input: Generated cleaning route and plan

[2110] Output: Specific cleaning instructions are sent to the cleaning device

[2111] Step 7:

[2112] The cleaning device starts cleaning

[2113] Based on the instructions received, the cleaning device begins cleaning the designated area (e.g., the living room), planning and executing an efficient cleaning route using built-in sensors and generative AI models.

[2114] Input: Specific cleaning instructions from the server

[2115] Data calculation: Cleaning is performed while dynamically correcting the actual cleaning route based on sensor information.

[2116] Output: Dust and dirt in the area is removed

[2117] Step 8:

[2118] User submits a status check request

[2119] The user sends a status confirmation request such as "Tell me the current cleaning status" from the terminal.

[2120] Input: Status Check Request

[2121] Output: The request is logged on the user's terminal and ready to be sent.

[2122] Step 9:

[2123] The user device sends a request to the server

[2124] The terminal sends this request to the server.

[2125] Input: User status check request

[2126] Output: The request is transmitted to the server

[2127] Step 10:

[2128] Server requests status

[2129] The server transmits the request to the cleaning device and obtains the current cleaning status and location information.

[2130] Input: User status check request

[2131] Output: The current cleaning status and location information request are sent to the cleaning device.

[2132] Step 11:

[2133] The cleaning device sends status information back to the server

[2134] The cleaning device sends its current status and location information back to the server.

[2135] Input: Status information request from the server

[2136] Data calculation: Collects current cleaning progress and location information

[2137] Output: Sending collected status and location information

[2138] Step 12:

[2139] The server sends status information to the user terminal.

[2140] The server returns the received information to the user terminal.

[2141] Input: Status and location information from the cleaning device

[2142] Output: Information is transmitted to the user terminal

[2143] Step 13:

[2144] The user terminal displays the status

[2145] The terminal displays to the user, "Currently cleaning the living room."

[2146] Input: Status and location information from the server

[2147] Output: Status information is displayed to the user

[2148] This allows the user to operate the cleaning device efficiently and in real time even from a remote location and check the current cleaning status.

[2149] (Application example 1)

[2150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2151] Modern self-driving vehicles have advanced autonomous driving capabilities, but cleaning the interior and exterior of the vehicle and performing regular maintenance require manual labor, which is a hassle for users. Furthermore, diagnosing the vehicle's condition and identifying repair areas requires specialized knowledge, which is inconvenient for users. There is a need to solve these problems and provide a system that efficiently automates the cleaning and maintenance of self-driving vehicles.

[2152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2153] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status confirmation request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, means for automatically cleaning the interior and exterior of the vehicle and diagnosing the vehicle status, and means for scheduling cleaning maintenance and automatically identifying and reporting areas requiring repairs. This automates vehicle cleaning and maintenance work, allowing users to flexibly manage these tasks from a remote location.

[2154] A "generative artificial intelligence model" is an artificial intelligence technique that can generate new information and content from data.

[2155] An "autonomous mobile cleaning device" is a hardware device that can clean while moving around on its own.

[2156] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[2157] A "cleaning command" is an instruction or command given by a user to a cleaning device.

[2158] A "status confirmation request" is a request made by the user to confirm the current status and location information of the cleaning device.

[2159] A "cleaning zone" is a particular area that a cleaning device is to clean.

[2160] The "means for generating a schedule" is a function that determines the time and order for the cleaning device to perform cleaning work efficiently.

[2161] "Automatic cleaning inside and outside the vehicle" is a function that automatically cleans the inside and outside of the vehicle.

[2162] "Means for diagnosing the condition of the vehicle" refers to a function that checks the condition of each part and system of the vehicle and detects abnormalities.

[2163] "Cleaning and maintenance scheduling" is a function that allows you to create a schedule for systematically cleaning and maintaining your vehicle.

[2164] The "means for automatically identifying and reporting parts requiring repair" is a function for automatically detecting parts of a vehicle that are faulty and reporting the same to the user.

[2165] This invention is a system that automates cleaning and maintenance work for self-driving vehicles using an autonomous mobile cleaning device equipped with a generative artificial intelligence model. The system is composed of a user terminal, a server, and the autonomous mobile cleaning device.

[2166] Components

[2167] 1. Autonomous mobile cleaning device

[2168] The cleaning device is a hardware device that moves autonomously and cleans the inside and outside of a vehicle. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language.

[2169] 2. User Device

[2170] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. Using this terminal, the user can remotely issue cleaning commands and check the status.

[2171] 3. Server

[2172] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[2173] System Operation Overview

[2174] 1. User sends command

[2175] Users input and send cleaning commands such as "clean the interior of the car" or "perform a maintenance check" using smart glasses or a smartphone.

[2176] 2. Receiving and parsing commands

[2177] The server receives commands sent from the user terminal, analyzes the contents of the commands, and then identifies cleaning areas and diagnostic points inside and outside the vehicle.

[2178] 3. Sending commands to the cleaning device

[2179] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving the instructions, the cleaning device begins cleaning the designated area. At the same time, it also performs a vehicle condition diagnosis.

[2180] 4. Status Check and Reporting

[2181] When a user wants to check the current status of the cleaning device, the user sends a command such as "Tell me the current situation" from the user terminal. The server transmits this request to the cleaning device, collects the current cleaning status and location information, and reports it to the user terminal.

[2182] Hardware and software used

[2183] Hardware:

[2184] Autonomous mobile cleaning device mounted on a self-driving vehicle

[2185] Vehicle condition diagnostic sensors

[2186] software:

[2187] Python

[2188] Flask (server-side request processing)

[2189] Generative AI model

[2190] Specific examples

[2191] 1. Specific scenario

[2192] Sending cleaning commands:

[2193] A user types into a smartphone app, "Start cleaning the car interior at 9:00 tomorrow morning."

[2194] Request a maintenance check:

[2195] When getting into the vehicle, enter the command "Tell me where there is a risk of breakdown" to obtain the necessary information.

[2196] 2. Example prompts for generative AI models

[2197] "Schedule my car to be cleaned tomorrow at 9am"

[2198] "Perform a maintenance check and report any abnormalities."

[2199] This system enables users to efficiently manage cleaning and maintenance work on autonomous vehicles from remote locations, significantly reducing the burden on users.

[2200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2201] Step 1:

[2202] The user uses a user terminal such as a smartphone to input and send cleaning commands and maintenance check instructions.

[2203] Input: The user enters a prompt statement such as "Start cleaning the car tomorrow morning at 9am."

[2204] Output: Commands are sent from the user terminal to the server.

[2205] Step 2:

[2206] The server analyzes the commands received from the user terminal and determines the corresponding cleaning actions and maintenance checks.

[2207] Input: Cleaning command data sent from the user terminal.

[2208] Data processing / calculation: The server uses a generative artificial intelligence model to analyze and interpret commands in natural language.

[2209] Output: The interpreted commands are directed to specific cleaning actions or maintenance checks.

[2210] Step 3:

[2211] Based on the analysis results, the server instructs the autonomous mobile cleaning device on specific cleaning areas and maintenance operations.

[2212] Input: Parsed cleaning actions and maintenance check instructions.

[2213] Output: Specific cleaning area and operation instructions for the cleaning device.

[2214] Step 4:

[2215] The autonomous mobile cleaning device begins cleaning in the designated area and diagnoses the condition of the inside and outside of the vehicle.

[2216] Input: Cleaning and diagnostic instructions sent from the server.

[2217] Specific operation: The cleaning device locates itself and starts cleaning, while at the same time checking for any abnormalities in the vehicle using a diagnostic sensor.

[2218] Output: Cleaning progress and vehicle status data.

[2219] Step 5:

[2220] The cleaning device and diagnostic sensors transmit collected data to a server.

[2221] Input: Cleaning progress and vehicle condition data.

[2222] Data processing / calculation: The server analyzes these data and generates information to report to the user.

[2223] Output: Parsed cleaning and diagnostic result data.

[2224] Step 6:

[2225] The server reports the current status and diagnostic results of the cleaning device to the user terminal.

[2226] Input: Cleaning progress and vehicle condition data.

[2227] Specific operation: The server organizes these data and sends them to the user terminal.

[2228] Output: Reports to the user device such as "Cleaning is complete" or "There are areas at risk of failure."

[2229] Step 7:

[2230] The user checks the report results on a smartphone or smart glasses and gives additional instructions if necessary.

[2231] Input: Report message from the server.

[2232] Specific operation: The user checks the report results and inputs new cleaning commands or maintenance instructions.

[2233] Output: New cleaning commands or maintenance instructions are sent to the server again.

[2234] Through these steps, users can efficiently and automatically manage the cleaning and maintenance of their autonomous vehicles.

[2235] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2236] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[2237] Components

[2238] 1. Cleaning device

[2239] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly.

[2240] 2. User Device

[2241] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, allowing the user to issue cleaning commands and check the status remotely.

[2242] 3. Server

[2243] The server is a central control unit that mediates communication between the user terminal and the cleaning device, processes various requests, analyzes commands from the cleaning device, and reports the status.

[2244] System Operation Overview

[2245] 1. User sends command

[2246] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[2247] 2. Receiving and parsing commands

[2248] The user terminal transmits the input command to the server.

[2249] The server receives this command, analyzes the command content, and determines the corresponding cleaning action.

[2250] 3. Sending commands to the cleaning device

[2251] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[2252] Once the cleaning device receives the command, it will begin cleaning the designated area.

[2253] 4. Status Check and Reporting

[2254] When a user wants to check the current status of the cleaning device, he sends a command such as "status" from the user terminal.

[2255] The server transmits this request to the cleaning device and obtains the current cleaning status and location information.

[2256] The cleaning device returns its current status and location information to the server, which then relays it to the user terminal.

[2257] Overview of Emotion Engine Operation

[2258] 1. User Emotion Recognition

[2259] The cleaning device is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions.

[2260] For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[2261] 2. Emotion-based adjustment of cleaning behavior

[2262] The cleaning device adjusts its cleaning schedule and frequency based on the emotions it recognizes.

[2263] For example, if the user is frustrated, the cleaning device may change its operation to finish cleaning quietly and quickly.

[2264] 3. Generating Emotional Feedback

[2265] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[2266] For example, a notification such as "The user is tired, so the living room will be cleaned quietly" is sent to the user terminal.

[2267] Specific examples

[2268] Example 1: Cleaning the living room

[2269] User: Type "Clean the living room" into their smartphone.

[2270] Terminal: Sends input to the server.

[2271] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[2272] Cleaning device: Receive instructions and start cleaning the living room.

[2273] Server: Notifies the user device that cleaning has started.

[2274] Device: Displays to the user "Cleaning of the living room has begun."

[2275] Example 2: Checking the cleaning status

[2276] User: Type "Tell me how the cleaning is going" on your smartphone.

[2277] Device: Sends a status check request to the server.

[2278] Server: Transmits the request to the cleaning device.

[2279] Cleaning device: Sends current status and location information back to the server.

[2280] Server: Returns the received information to the user terminal.

[2281] Device: Shows the user "You're currently cleaning the living room."

[2282] Example 3: Adjusting cleaning behavior through emotion recognition

[2283] User: In a tired voice, types, "Clean the living room."

[2284] Terminal: Sends input to the server.

[2285] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[2286] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[2287] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[2288] Server: Notifies the user device of emotional feedback from the cleaning device.

[2289] Device: Shows the user "Quietly clean the living room."

[2290] This system allows users to operate the cleaning device flexibly and efficiently even from a remote location, and check the status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning operations according to the user's emotions, thereby improving user satisfaction.

[2291] The processing flow will be explained below.

[2292] Step 1:

[2293] The user inputs and sends a cleaning command from their smartphone (user device). For example, they input "Clean the living room."

[2294] Step 2:

[2295] The terminal receives the user's input and sends a request of the form "command: clean area living room" to the server.

[2296] Step 3:

[2297] The server receives the request and extracts the command part ("clean area living room") through the handle_user_request method.

[2298] Step 4:

[2299] The server passes the extracted command to the receive_command method of the cleaning device to send the instruction.

[2300] Step 5:

[2301] The cleaning device analyzes the command through the receive_command method and calls the process_command method of the built-in generative artificial intelligence model.

[2302] Step 6:

[2303] A generative artificial intelligence model analyzes the command and generates the response "Cleaning living room."

[2304] Step 7:

[2305] The cleaning device sends the generated response to the server.

[2306] Step 8:

[2307] The server returns the response received from the cleaning device to the user terminal, for example, a message saying "Cleaning of the living room has started."

[2308] Step 9:

[2309] The terminal displays the response received from the server on the screen and notifies the user.

[2310] Step 10:

[2311] The cleaning device uses a built-in emotion engine to recognize emotions from the user's voice and facial expressions. For example, if the user says "Clean up" in a tired voice, the emotion engine will recognize this as "fatigue."

[2312] Step 11:

[2313] The cleaning device automatically adjusts its cleaning plan based on the emotions it recognizes, for example, recognizing fatigue and changing settings to clean with reduced noise.

[2314] Step 12:

[2315] The user sends a "status" request from their smartphone to check the current status of the cleaning device.

[2316] Step 13:

[2317] The device sends a request called "status" to the server.

[2318] Step 14:

[2319] The server receives the status check request and invokes the report_status method of the cleaning device.

[2320] Step 15:

[2321] The cleaning device returns its current status (e.g., cleaning) and location information (e.g., living room) to the server via the report_status method.

[2322] Step 16:

[2323] The server returns the received status information to the user terminal.

[2324] Step 17:

[2325] The device displays the status information received from the server on the screen and notifies the user, for example, "The living room is currently being cleaned."

[2326] Step 18:

[2327] The cleaning device generates feedback from the emotion engine and notifies the user terminal via the server. For example, a message such as "Because the user is tired, the living room will be cleaned quietly" is displayed.

[2328] Example 2

[2329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2330] Conventional autonomous mobile cleaning devices simply receive and execute cleaning commands from the user, making it difficult to respond flexibly to emotions and situations. Furthermore, they often fail to efficiently check the status of the cleaning device or address specific stains in the room. Furthermore, because the cleaning device lacks the ability to understand the user's emotions and adapt its behavior based on those emotions, it is unable to provide a satisfactory service to some users.

[2331] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2332] In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for storing the corresponding cleaning area and generating an efficient schedule, and means including an emotion engine for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, thereby enabling real-time response to commands from the user and flexible cleaning operation according to the user's emotions.

[2333] A "generative artificial intelligence model" is a type of artificial intelligence that has the ability to generate new content or results from data, and is applied to tasks such as natural language processing and image generation.

[2334] An "autonomous mobile cleaning device" is a cleaning device that has the ability to clean while moving around on its own, and is equipped with sensors, cameras, and mapping functions inside.

[2335] A "user terminal" is a device operated by a user, and includes smartphones, tablets, personal computers, etc.

[2336] The "server" is a central control unit that handles communication and data processing between the user terminal and the cleaning device, and has functions such as command analysis, instruction transmission, and status management.

[2337] An "emotion engine" refers to an algorithm or system that recognizes emotions from a user's voice and facial expressions and adjusts the device's behavior based on those emotions.

[2338] A "status check request" is a request sent by a user to check the current operating status and location information of the cleaning device.

[2339] A "cleaning command" is an instruction that a user sends to a cleaning device to instruct the device to perform a specific cleaning operation.

[2340] "Cleaning area" refers to the particular location or room in which the cleaning device performs cleaning.

[2341] An "efficient schedule" refers to a work plan generated to allow the cleaning device to clean as efficiently as possible, minimizing waste of time and energy.

[2342] The present invention is a system that mainly consists of an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and is composed of a user terminal and a server. The details of each component and the operation of the entire system are explained below.

[2343] Components

[2344] 1. Autonomous mobile cleaning device

[2345] The cleaning device is a piece of hardware that has a built-in generative artificial intelligence model and emotion engine, and moves around autonomously to clean. The device is equipped with sensors, cameras, and mapping functions, and operates efficiently within a designated cleaning area.

[2346] 2. User Device

[2347] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC, which allows the user to send cleaning commands and check the status remotely.

[2348] 3. Server

[2349] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes data. The server analyzes commands from the user and sends specific operational instructions to the cleaning device. It also monitors the operating status and location information of the cleaning device.

[2350] System Operation

[2351] User command sending

[2352] A user inputs and sends a cleaning command such as "Clean the living room" from their smartphone. The user device then sends the command to the server.

[2353] Receiving and parsing commands

[2354] The server receives commands sent from the user terminal and analyzes the content using a text analysis engine. For example, if the command "clean the living room" is received, the server generates a cleaning area and an action plan based on the content.

[2355] Sending commands to cleaning devices

[2356] Based on the analysis results, the server sends specific cleaning area and operation instructions to the cleaning device. Upon receiving these instructions, the cleaning device begins cleaning the specified area.

[2357] Checking and reporting status

[2358] When a user wants to check the current status of the cleaning device, they send a status check request such as "Tell me the cleaning status" from the user terminal. The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The server then returns the received information to the user terminal.

[2359] Emotion Engine Operation

[2360] User Emotion Recognition

[2361] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[2362] Emotion-based adjustment of cleaning behavior

[2363] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is irritated, the cleaning device will change its behavior to clean quietly and quickly.

[2364] Generating Emotional Feedback

[2365] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[2366] Specific examples

[2367] Example 1: Cleaning the living room

[2368] The user types "clean the living room" into their smartphone.

[2369] The terminal sends this command to the server.

[2370] The server parses the command and generates the instruction "Clean the living room."

[2371] The cleaning device receives the parsed instructions and begins cleaning the living room.

[2372] The server notifies the user terminal of the status from the cleaning device and displays to the user, "Cleaning of the living room has started."

[2373] Example 2: Checking the cleaning status

[2374] The user types "Tell me about the cleaning status" on their smartphone.

[2375] The terminal sends a status confirmation request to the server.

[2376] The server communicates a request to the cleaning device to obtain the current status and location of the cleaning device.

[2377] The server returns the received information to the user terminal and displays to the user, "Currently cleaning the living room."

[2378] Example 3: Adjusting cleaning behavior through emotion recognition

[2379] The user types in a tired voice, "Clean the living room."

[2380] The terminal sends this input to the server.

[2381] The server analyzes the command and sends an instruction to "clean the living room" to the cleaning device.

[2382] The cleaning device receives instructions and the emotion engine recognizes the user's fatigue.

[2383] The cleaning device is adjusted to complete the cleaning operation quietly and quickly.

[2384] The server notifies the user terminal of the emotion feedback from the cleaning device and displays to the user, "Quietly cleans the living room."

[2385] With the above operations, users can operate the cleaning device flexibly and efficiently even from a remote location and check the status in real time. In addition, by incorporating an emotion engine, the device can provide appropriate cleaning operations according to the user's emotions, improving user satisfaction.

[2386] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2387] Program processing steps

[2388] Step 1: User command input

[2389] explanation

[2390] A user uses a user terminal such as a smartphone or tablet to input specific cleaning commands to the cleaning device.

[2391] input

[2392] The user enters a prompt sentence such as "Clean the living room."

[2393] Data processing and calculation

[2394] The terminal receives voice or text input from the user and converts the voice data to text data as needed.

[2395] output

[2396] A textual cleaning command is generated.

[2397] Specific actions

[2398] The user starts the application on the terminal and inputs cleaning commands using voice recognition or keyboard input.

[2399] Step 2: Send the command

[2400] explanation

[2401] The user terminal transmits the input command to the server.

[2402] input

[2403] Textual cleaning commands.

[2404] Data processing and calculation

[2405] The device converts the generated cleaning command into JSON format data and sends it to the server using an HTTP request.

[2406] output

[2407] The request data in JSON format is sent to the server.

[2408] Specific actions

[2409] The device sends an HTTP POST request to the server's API endpoint using Wi-Fi or mobile data.

[2410] Step 3: Command Analysis

[2411] explanation

[2412] The server analyzes the received command and generates specific operating instructions for the cleaning device.

[2413] input

[2414] Cleaning commands in JSON format.

[2415] Data processing and calculation

[2416] The server analyzes the command using a natural language processing engine and generates a specific cleaning area and cleaning route. For example, from the command "Clean the living room," it extracts the coordinate data of the living room and the cleaning route.

[2417] output

[2418] Specific operating instructions for the cleaning device (cleaning area and route information).

[2419] Specific actions

[2420] The server uses an internal text analysis engine to parse the received commands and calculate the appropriate cleaning route.

[2421] Step 4: Sending commands to the cleaning device

[2422] explanation

[2423] Based on the analysis results, the server sends specific cleaning areas and operation instructions to the cleaning device.

[2424] input

[2425] Specific operating instructions for the cleaning device.

[2426] Data processing and calculation

[2427] The server converts the generated operation instructions into a protocol that can be understood by the cleaning device.

[2428] output

[2429] Cleaning instruction data is transmitted in a format that can be understood by the cleaning device.

[2430] Specific actions

[2431] The server uses an HTTP request to send instruction data to the communication module of the cleaning device.

[2432] Step 5: Start cleaning

[2433] explanation

[2434] The cleaning device receives instructions from the server and begins cleaning the designated area.

[2435] input

[2436] Cleaning instruction data.

[2437] Data processing and calculation

[2438] Based on the instructions it receives, the cleaning device works in conjunction with an internal mapping sensor to check its own position while cleaning.

[2439] output

[2440] The cleaning operation begins.

[2441] Specific actions

[2442] The cleaning device activates the operation module and performs cleaning while scanning the designated area.

[2443] Step 6: Check and report status

[2444] explanation

[2445] When a user wants to check the current status of the cleaning device, he sends a request to the server, which retrieves the status information from the cleaning device and returns it to the user terminal.

[2446] input

[2447] Status requests (e.g., "How's the cleaning going?").

[2448] Data processing and calculation

[2449] The server passes the request to the cleaning device, which then returns its current status and location information to the server.

[2450] output

[2451] The current status and location of the cleaning device is displayed on the user terminal.

[2452] Specific actions

[2453] The server receives a request from the user terminal, sends a "status check" instruction to the cleaning device, and returns a response from the cleaning device to the user terminal.

[2454] Step 7: Emotion Recognition

[2455] explanation

[2456] The cleaning device uses an emotion engine to recognize emotions from the user's voice and facial expressions.

[2457] input

[2458] User voice and facial expression data.

[2459] Data processing and calculation

[2460] The cleaning device uses an emotion engine to extract emotions from the user's voice and facial expressions and assign appropriate emotion labels.

[2461] output

[2462] Recognized emotion data (e.g., "fatigue").

[2463] Specific actions

[2464] The cleaning device uses a built-in microphone and camera to analyze the user's emotions and stores the results in its internal memory.

[2465] Step 8: Emotionally Based Behavior Adjustments

[2466] explanation

[2467] The cleaning device adjusts its cleaning behavior based on the recognized emotion.

[2468] input

[2469] Emotional data (e.g., "fatigue").

[2470] Data processing and calculation

[2471] The cleaning device optimizes its operating parameters (volume, speed, etc.) based on emotion data.

[2472] output

[2473] The cleaning operation is performed based on the adjusted operation parameters.

[2474] Specific actions

[2475] The cleaning device switches to a quiet mode and adjusts the cleaning speed to perform operations that are considerate of the user's feelings.

[2476] The above are the specific processing steps and details of the system.

[2477] (Application example 2)

[2478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2479] Conventional autonomous mobile cleaning devices have problems such as inefficient cleaning instructions and status checks, and inability to adapt to user emotions and store conditions. In particular, cleaning activities in brick-and-mortar stores require frequent cleaning to maintain cleanliness, and streamlining cleaning work is required to reduce the burden on store staff. Furthermore, a system that can appropriately adjust cleaning operations according to the store's congestion status and staff stress levels is desired.

[2480] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing cleaning commands received from a user terminal and starting a corresponding cleaning operation, means for reporting the current cleaning status and location information in response to a status check request from the user terminal, means for recognizing the user's emotions and adjusting the cleaning operation based on the emotions, and means for analyzing the command input from the user and reporting the operating status and battery status of the cleaning robot in real time. This allows users to flexibly and efficiently operate the cleaning device from a remote location and check the status in real time. In particular, in physical stores, appropriate cleaning operations can be provided according to the store's congestion status and the staff's stress level, thereby improving the efficiency of cleaning work and reducing the burden on store staff.

[2481] A "generative artificial intelligence model" is an artificial intelligence model that has the ability to generate new data and information based on given data.

[2482] An "autonomous mobile cleaning device" is a device that moves autonomously in response to instructions from a user and cleans a designated area.

[2483] A "user terminal" is a remote control device operated by a user, such as a smartphone, tablet, or PC.

[2484] A "cleaning command" is a command that a user inputs to instruct the autonomously mobile cleaning device to perform cleaning.

[2485] A "status confirmation request" is a request sent by a user to find out the current cleaning status and location information of the autonomously mobile cleaning device.

[2486] The "corresponding cleaning area" is a specific area that the autonomously mobile cleaning device has memorized as a cleaning target.

[2487] An "efficient schedule" is a time and motion plan designed to optimize cleaning tasks for an autonomous mobile cleaning device.

[2488] The "emotion engine" is an engine that has the function of recognizing the user's emotional state from their voice, facial expression, etc., and adjusting cleaning operations according to that state.

[2489] "Battery status" is information indicating the current remaining battery power and operating time of the autonomously mobile cleaning device.

[2490] "Real-time" is a term that refers to information being transmitted and processed almost immediately.

[2491] This system is based on an autonomous mobile cleaning device equipped with a generative AI model and an emotion engine, and also consists of a user terminal and a server. Below, we will explain the details of each component and the operation of the entire system.

[2492] Components

[2493] 1. Cleaning device

[2494] The cleaning device is hardware that can move around autonomously and clean. It has a built-in generative artificial intelligence model and can respond to cleaning commands and questions from the user in natural language. It also has an emotion engine that can recognize the user's emotions and adjust its cleaning behavior accordingly. The main hardware used includes a LiDAR sensor, camera, and microphone.

[2495] 2. User Device

[2496] A user terminal is a device operated by a user, such as a smartphone, tablet, or PC. This allows the user to remotely issue cleaning commands and check the status. Commands and requests are sent from the user terminal to the server.

[2497] 3. Server

[2498] The server is a central control unit that mediates communication between the user terminal and the cleaning device and processes various requests. The server analyzes commands from the cleaning device and reports their status. The main software used includes Node.js and Java. MySQL and PostgreSQL are used as databases.

[2499] System Operation Overview

[2500] 1. User sends command

[2501] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[2502] 2. Receiving and parsing commands

[2503] The user terminal sends the input command to the server, which receives the command and analyzes it to determine the corresponding cleaning action.

[2504] 3. Sending commands to the cleaning device

[2505] Based on the analysis results, the server instructs the cleaning device on the specific cleaning area and operation. Upon receiving the instruction, the cleaning device begins cleaning the specified area.

[2506] 4. Status Check and Reporting

[2507] When a user wants to check the current status of the cleaning device, the user terminal sends a status check request such as "Tell me the cleaning status." The server transmits this request to the cleaning device and obtains the current cleaning status and location information. The cleaning device returns its current status and location information to the server, which then transmits it to the user terminal.

[2508] Overview of Emotion Engine Operation

[2509] 1. User Emotion Recognition

[2510] The vacuum cleaner is equipped with an emotion engine that recognizes emotions from the user's voice and facial expressions. For example, if the user is tired, the emotion engine will recognize this as "fatigue."

[2511] 2. Emotion-based adjustment of cleaning behavior

[2512] The cleaning device will adjust its cleaning schedule and frequency based on the emotions it senses, for example, if the user is tired, the cleaning device will change its behavior to clean quietly and quickly.

[2513] 3. Generating Emotional Feedback

[2514] The cleaning device generates feedback based on the user's emotions and notifies the user terminal, for example, by sending a notification such as "Because you are tired, we will quietly clean the living room."

[2515] Specific examples

[2516] Example 1: Cleaning the living room

[2517] User: Type "Clean the living room" into their smartphone.

[2518] Terminal: Sends input to the server.

[2519] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[2520] Cleaning device: Receive instructions and start cleaning the living room.

[2521] Server: Notifies the user device that cleaning has started.

[2522] Device: Displays to the user "Cleaning of the living room has begun."

[2523] Example 2: Checking the cleaning status

[2524] User: Type "Tell me how the cleaning is going" on your smartphone.

[2525] Device: Sends a status check request to the server.

[2526] Server: Transmits the request to the cleaning device.

[2527] Cleaning device: Sends current status and location information back to the server.

[2528] Server: Returns the received information to the user terminal.

[2529] Device: Shows the user "You're currently cleaning the living room."

[2530] Example 3: Adjusting cleaning behavior through emotion recognition

[2531] User: In a tired voice, types, "Clean the living room."

[2532] Terminal: Sends input to the server.

[2533] Server: Analyzes the command and sends the instruction "Clean the living room" to the cleaning device.

[2534] Cleaning device: Receives instructions, but the emotion engine recognizes user fatigue.

[2535] Cleaning device: Adjusted to complete cleaning operations quietly and quickly.

[2536] Server: Notifies the user device of emotional feedback from the cleaning device.

[2537] Device: Shows the user "Quietly clean the living room."

[2538] Prompt Sentence Examples

[2539] "It promptly cleans the shopping area specified by the user, reports battery status and current location information in real time, and makes appropriate adjustments based on the user's voice and facial expressions."

[2540] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[2541] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2542] Step 1:

[2543] User command sending

[2544] The user inputs and transmits a cleaning command such as "clean the living room" using the user terminal.

[2545] Input: Cleaning commands entered by the user.

[2546] Output: A cleaning command is sent from the user terminal to the server.

[2547] In this step, commands are entered through the user terminal interface and sent to the server as internal API requests.

[2548] Step 2:

[2549] Receiving and parsing commands

[2550] The server receives and analyzes the command sent from the user terminal.

[2551] Input: Cleaning command sent from user terminal.

[2552] Output: Analyzed cleaning area and action instructions.

[2553] The server analyzes the command and determines the corresponding cleaning action. This analysis is done using a natural language processing model (e.g., a generative AI model). The analysis result is then sent to the next step.

[2554] Step 3:

[2555] Sending commands to cleaning devices

[2556] Based on the analysis results, the server instructs the cleaning device on specific cleaning areas and operations.

[2557] Input: Analyzed cleaning area and action instructions.

[2558] Output: Operation instructions to the cleaning device.

[2559] The server sends an HTTP request to the cleaning device, informing it of the required cleaning instructions, and the cleaning device begins to operate according to the received instructions.

[2560] Step 4:

[2561] Emotional awareness and regulation

[2562] The cleaning device performs processing to recognize the user's emotions.

[2563] Input: User's voice and facial expressions.

[2564] Output: Adjustment of cleaning behavior based on recognized emotions.

[2565] The cleaning device uses a built-in microphone and camera to recognize the user's emotional state, and the recognized emotional data is processed by an emotion engine to determine an appropriate cleaning plan.

[2566] Step 5:

[2567] Checking and reporting status

[2568] When a user wants to check the current cleaning status, the user sends a request from the user terminal saying "Tell me the cleaning status."

[2569] Input: Status check request from user terminal.

[2570] Output: Current cleaning status and location.

[2571] The server receives this request and queries the cleaning device about its current status. The cleaning device returns the current cleaning status to the server, which then transmits it to the user terminal.

[2572] Step 6:

[2573] Generating Emotional Feedback

[2574] The cleaning device generates feedback based on the user's emotions and notifies the user terminal.

[2575] Input: Perceived user emotion.

[2576] Output: Emotion feedback notification to the user device.

[2577] The cleaning device generates an appropriate feedback message based on the emotion data processed by the emotion engine and transmits it to the user terminal via the server.

[2578] This system allows users to flexibly and efficiently operate cleaning devices even from remote locations and check their status in real time. Furthermore, by incorporating an emotion engine, the system can provide appropriate cleaning actions according to the user's emotions, thereby improving user satisfaction. In physical stores, this system enables efficient cleaning work that reduces the burden on staff while maintaining store cleanliness.

[2579] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2580] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2581] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2582] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2583] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2584] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situa...

Claims

1. An autonomous mobile cleaning device equipped with a generative artificial intelligence model and remotely controlled by a user, means for analyzing a cleaning command received from a user terminal and initiating a corresponding cleaning operation; means for reporting a current cleaning status and location information in response to a status confirmation request from a user terminal; means for storing corresponding cleaning areas and generating an efficient schedule; A system including:

2. The system of claim 1 , which provides an optimal cleaning plan based on a user's schedule to address specific stains or rooms.

3. 10. The system of claim 1, further comprising the capability of answering questions in natural language using a generative artificial intelligence model.

Images