system

Abstract

Provide a system. A method for capturing an environment using a night vision camera; means for processing the captured video to extract features and update location information; A means of autonomously navigating and creating maps; means for detecting contamination using optical recognition technology; means for performing a cleaning operation to clean the detected soiling; A means to send real-time alerts after cleaning, and A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional cleaning robots lack sufficient night vision capabilities, making them unable to efficiently clean at night or in dark places. Furthermore, they often lack accuracy in detecting dirt, resulting in incomplete cleaning. Additionally, they lack sufficient autonomous navigation capabilities, making it difficult for them to effectively map and clean complex environments. The present invention aims to provide a solution to these issues and realize efficient and accurate late-night cleaning. [Means for solving the problem]

[0005] The present invention is a system that includes the following means: a means for capturing the environment using a night vision camera, a means for processing the captured image to extract features and update location information, a means for autonomously navigating and creating a map, a means for detecting dirt using optical recognition technology, a means for performing cleaning operations to clean the detected dirt, and a means for sending a real-time alert after cleaning. It also includes a means for optimizing cleaning activities using a learning algorithm to improve future cleaning routes, and a means for sending a cleaning log to a server and storing it in a database. This enables efficient and accurate cleaning during the night while also contributing to environmental conservation.

[0006] A "night vision camera" is a camera device for acquiring visual information in low-light environments.

[0007] "Capturing the environment" means acquiring visual information about the surroundings using a night vision camera.

[0008] "Feature extraction" means identifying and extracting specific patterns or information from captured video.

[0009] "Updating location information" means continually determining the current location on the map based on the extracted features.

[0010] "Autonomous navigation" means that the system itself determines its direction of travel and route without external instructions.

[0011] "Mapping" means generating a map showing the structure and characteristics of the entire area based on information from each point in the environment.

[0012] "Optical recognition technology" is a technology for identifying objects and patterns using visual information from cameras and other sources.

[0013] "Detecting dirt" refers to using optical recognition technology to locate dirt or foreign objects in the environment.

[0014] "Performing a cleaning action" means taking physical action to remove the detected dirt.

[0015] "Real-time alerts" means immediate notification of cleaning results and other important information.

[0016] A "learning algorithm" is a computational method for optimizing the next action or method to be taken based on past data and experience.

[0017] "Optimizing cleaning activities" means using learning algorithms to improve the efficiency of cleaning routes and methods.

[0018] "Cleaning log" refers to recorded data such as the location and time of cleaning.

[0019] "Sending to a server" means transferring data from a terminal to a central computing device.

[0020] "Saving in a database" means storing the transmitted data in a specialized storage device. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention relates to a cleaning robot system using a night vision camera, and specific embodiments thereof will be described below. Details of program processing will be explained with specific examples, divided into the subjects of the server, terminal, and user.

[0043] overview

[0044] The cleaning robot system first captures the environment using a night vision camera, analyzes the footage, and extracts features. It then creates a map using an autonomous navigation system and detects dirt using optical recognition technology. When dirt is detected, the robot performs cleaning operations and sends the results as an alert in real time. It also uses a learning algorithm to optimize cleaning activities, and sends cleaning logs to a server for storage in a database. This ensures efficient and accurate late-night cleaning.

[0045] Server Roles and Operations

[0046] The server receives data sent from the robot and updates the integrated database. Specifically, it stores information such as navigation data, map data, and cleaning history, and uses it as a reference for the robot to perform cleaning activities efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[0047] example:

[0048] The server receives the map data sent from each robot and stores it in an integrated database.

[0049] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[0050] Roles and actions of the terminal (robot)

[0051] The terminal (robot) operates autonomously and performs cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins processing. Next, optical recognition technology is used to extract features from the image and update the map of the current location. After that, it uses an autonomous navigation system to calculate the next position to go to, and if dirt is detected, it performs cleaning operations.

[0052] example:

[0053] The device uses a night vision camera to capture and map the environment at night.

[0054] The device will start cleaning when it detects dirt in a specific location.

[0055] User Roles and Actions

[0056] Users can remotely monitor the robot's status and issue instructions as needed. Users receive real-time alerts to see the cleaning robot's activity and can also give instructions to re-clean specific designated areas.

[0057] example:

[0058] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[0059] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[0060] The learning and optimization process

[0061] The robot learns using cleaning activity data and map data, optimizing future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[0062] example:

[0063] The device sends data on cleaning activities to a server, which stores it in a database.

[0064] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[0068] Step 2:

[0069] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale to facilitate further processing.

[0070] Step 3:

[0071] The device extracts features from the captured video and updates a map of the device's current location based on the extracted features, accurately determining the device's location.

[0072] Step 4:

[0073] The device uses an autonomous navigation system to calculate and navigate its next location, allowing the device to efficiently cover the entire area.

[0074] Step 5:

[0075] The device detects dirt using optical recognition technology, capturing the environment with a night vision camera and analyzing the footage to identify dirt and foreign objects.

[0076] Step 6:

[0077] The device will perform cleaning operations at the location where the dirt is detected, and once the cleaning operation is complete, the device will send the results to the server in real time and also send an alert that the cleaning is complete.

[0078] Step 7:

[0079] The server stores the cleaning log received from the device in a database, analyzes the cleaning history and map data, derives efficient cleaning patterns, updates the learning algorithm, and provides feedback to the device on an optimized navigation system.

[0080] Step 8:

[0081] Users remotely monitor the device's location and cleaning status in real time, and can send instructions to the device to re-clean specific areas as needed.

[0082] Step 9:

[0083] The device receives updated information from the server and instructions from the user to continue cleaning, and then initiates the next cleaning activity accordingly. Through this cyclical process, the device continuously improves its cleaning efficiency and contributes to environmental conservation.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Current cleaning robot systems have difficulty performing efficient and accurate cleaning at night, and there are many challenges with dirt detection and map creation, especially in environments that use night vision. This reduces cleaning accuracy and efficiency, and increases time and costs. Furthermore, few systems utilize cleaning activity logs to further improve efficiency and optimization. To solve these problems, a more accurate and efficient cleaning robot system is needed.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for capturing the environment using a night vision camera, means for converting the captured image to grayscale to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, and means for sending cleaning data to the server and storing it in an integrated database. This makes it possible to detect dirt with high accuracy even in nighttime environments, perform efficient cleaning, and optimize future cleanings by utilizing cleaning activity log data.

[0089] A "night vision camera" is a special imaging device that can capture images even in darkness or low-light environments.

[0090] The "means for capturing the environment" is a method for capturing an image of the surroundings using a night vision camera.

[0091] "Means for converting to grayscale" refers to a method for converting the acquired image into a format that is expressed in shades of black and white.

[0092] The "means for extracting features and updating position information" is a method for detecting specific patterns or objects from the image converted to grayscale and using that information to update the robot's position.

[0093] "Means for autonomous navigation and map creation" refers to a method in which a robot determines its own route and generates a map based on information about its surroundings.

[0094] "Optical recognition technology" is a technology that analyzes video data acquired by a camera and recognizes objects and features.

[0095] The "means for detecting dirt" is a method for identifying dirt from captured images using optical recognition technology.

[0096] "Means for performing cleaning actions" refers to a series of actions that the robot takes to remove the detected soiling.

[0097] "Means for sending real-time alerts" refers to a method for sending immediate notification to users or systems after cleaning is completed.

[0098] "Means for transmitting cleaning data to a server and storing it in an integrated database" refers to a method for transmitting records of cleaning activities to a server via the Internet or the like and storing them there.

[0099] The present invention relates to an autonomous cleaning robot system using a night vision camera, and specific embodiments thereof will be described below.

[0100] Server Roles and Operations

[0101] The server receives data sent from the robot (navigation data, map data, cleaning history, etc.) and stores it in an integrated database. The server analyzes the information stored in the database and identifies areas that are frequently soiled. This allows for the creation of an efficient and effective cleaning plan. The server also analyzes the cleaning history, generates new cleaning algorithms, and distributes them to the terminal (robot).

[0102] Examples:

[0103] The server receives the map data sent from each robot and stores it in an integrated database.

[0104] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[0105] Roles and actions of the terminal (robot)

[0106] The terminal (robot) operates autonomously to carry out cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins image processing. Optical recognition technology is used to extract features from the grayscale image and update the robot's current location on a map. Next, an autonomous navigation system is used to calculate an appropriate route, and if dirt is detected, it immediately begins cleaning.

[0107] Examples:

[0108] The device uses a night vision camera to capture and map the environment at night.

[0109] The device will start cleaning when it detects dirt in a specific location.

[0110] User Roles and Actions

[0111] Users can remotely monitor the robot's status. They can check the robot's location and cleaning status in real time using a smartphone or other device. If necessary, users can also issue instructions to re-clean a specific area. If an alert occurs, they can quickly check the details and take appropriate action.

[0112] Examples:

[0113] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[0114] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[0115] The learning and optimization process

[0116] The device (robot) learns from cleaning activity data and map data and optimizes future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[0117] Examples:

[0118] The device sends data on cleaning activities to a server, which stores it in a database.

[0119] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[0120] Example prompt for a generative AI model:

[0121] "Please explain how to save map data of a cleaning robot acquired at night by a night vision camera to a server."

[0122] "Explain how you can use a night vision camera to capture video, convert it to grayscale, detect dirt, and update the map."

[0123] "Explain how you can monitor the status of your cleaning robot on your smartphone and receive alerts to re-clean specific areas."

[0124] "Explain how data collected from a cleaning robot can be used to optimize the next cleaning session."

[0125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0126] Step 1:

[0127] The terminal (robot) activates the night vision camera and starts capturing the environment. The night vision camera acquires images of the environment and sends the data to the computer vision system. The input is the raw image captured by the night vision camera. The output is the image data itself.

[0128] Specific behavior:

[0129] The device switches to night setting mode.

[0130] Activate the night vision camera and capture video continuously.

[0131] Step 2:

[0132] The device converts the captured video to grayscale. The input is color video data acquired from a night vision camera, and the output is grayscale video data. This conversion makes the video data easier to analyze.

[0133] Specific behavior:

[0134] The terminal activates an image processing module to convert the color image into grayscale format.

[0135] The converted grayscale image is stored in a temporary memory.

[0136] Step 3:

[0137] The device uses optical recognition technology to extract features from the grayscale image and update the location information. The input is the grayscale image data obtained in step 2, and the output is feature data and updated location information. The feature data indicates important parts of the image, such as dirt or obstacles.

[0138] Specific behavior:

[0139] The device runs a feature extraction algorithm to recognize dirt and obstacles from the video data.

[0140] The current location information is updated based on the extracted features.

[0141] Step 4:

[0142] The device updates the map using the autonomous navigation system. The input is the feature data and location information obtained in step 3, and the output is updated map data. This map is modified by adding new data based on the known range.

[0143] Specific behavior:

[0144] The device runs a navigation algorithm and updates the map to reflect the current location and feature data.

[0145] The updated map is stored in memory and prepared for the next action.

[0146] Step 5:

[0147] The device performs cleaning operation when dirt is detected. The input is the map data updated in step 4 and the location information of the dirt, and the output is the cleaning completion data.

[0148] Specific behavior:

[0149] An autonomous navigation system is used to navigate to the location where the contamination is detected.

[0150] Activates the cleaning module and cleans the designated area.

[0151] Step 6:

[0152] After cleaning, the terminal sends a real-time alert to the server. The input is the cleaning completion data, and the output is the alert notification to the server.

[0153] Specific behavior:

[0154] The terminal activates the communication module and transmits cleaning completion data to the server.

[0155] An alert is sent to the server and the user is notified.

[0156] Step 7:

[0157] The server receives the cleaning data sent from the terminal and stores it in an integrated database. The input is the cleaning data sent from the terminal, and the output is the history data stored in the database.

[0158] Specific behavior:

[0159] The server starts the communication module and receives data from the terminal.

[0160] The received data is analyzed and stored in an integrated database.

[0161] Step 8:

[0162] The server analyzes the received cleaning data and generates a new cleaning algorithm. The input is the stored cleaning history data, and the output is the updated cleaning algorithm.

[0163] Specific behavior:

[0164] The server launches a data analysis module and calculates an efficient cleaning pattern based on past cleaning history.

[0165] Generate new algorithms and prepare them for distribution to the robots.

[0166] Step 9:

[0167] The user remotely monitors the robot's status and issues instructions for re-cleaning as necessary. The input is alert notifications from the server and the robot's location information, and the output is instructions for re-cleaning.

[0168] Specific behavior:

[0169] The user launches the smartphone app and checks the robot's status.

[0170] If necessary, instructions to re-clean specific areas will be sent via the app.

[0171] (Application example 1)

[0172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] In large facilities such as logistics centers, efficient cleaning is difficult in environments with poor visibility at night. Furthermore, to efficiently clean large areas, it is necessary to manage the robot's real-time location information, detect dirt, and respond immediately. However, current cleaning systems lack the means to effectively solve these issues. There is also a need for a method to optimize the efficiency of cleaning activities and improve future cleaning routes based on past cleaning data.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0175] In this invention, the server includes: means for capturing the environment using a night vision camera; means for processing the captured image to extract features and update location information; means for autonomously navigating and creating maps; means for detecting dirt using optical recognition technology; means for performing cleaning operations to clean the detected dirt; means for sending real-time alerts after cleaning; means including a smartphone application for displaying the robot's real-time location information; means for viewing the environmental image captured by the night vision camera; means for issuing alerts when dirt is detected; means for remotely operating the robot; and means for displaying cleaning history and checking learning data. This enables efficient and accurate cleaning even in environments with poor visibility, even at night, and enables real-time status monitoring and immediate response. Furthermore, it enables optimization of cleaning activities and improvement of cleaning routes based on past data.

[0176] A "night vision camera" is a camera used to ensure visibility in the dark, and is a device that uses infrared light to capture the environment.

[0177] The "means for capturing the environment" is a mechanism that has the function of acquiring the surrounding situation as video data using a night vision camera.

[0178] The "means for extracting features and updating location information" is a technology that detects specific patterns or objects from captured video data and updates the current location information based on that.

[0179] "Means for autonomous navigation and map creation" refers to a system that automatically moves based on acquired location information, aggregates information from each location, and generates a map.

[0180] "Optical recognition technology" is a technology that uses optical devices such as cameras to detect specific objects or features from image data.

[0181] The "means for detecting dirt" is a system that uses optical recognition technology to identify dirt from video data and pinpoint its location.

[0182] "Means for performing cleaning actions" refers to a series of operations that the robot performs to remove the detected dirt.

[0183] "Means for sending alerts in real time" refers to a system that has a communication function to immediately notify users of detected dirt and the cleaning status after cleaning.

[0184] The "smartphone application for displaying real-time location information of a robot" is an application that displays the current location of a cleaning robot on a smartphone in real time.

[0185] "Means for viewing environmental images captured by a night vision camera" refers to a mechanism for displaying images captured by a night vision camera on a smartphone or other device.

[0186] "Means for issuing an alert notification when dirt is detected" refers to a mechanism that has the function of sending an immediate notification when dirt is confirmed.

[0187] "Means for remotely operating a robot" refers to remote control technology for remotely controlling the operation of a cleaning robot.

[0188] "Means for displaying cleaning history and checking learning data" refers to a mechanism that displays the history of past cleaning activities and learns from the acquired data to improve the cleaning algorithm.

[0189] This section describes a specific embodiment of a cleaning robot system using a night vision camera. This invention utilizes an autonomously operating cleaning robot to achieve efficient cleaning, particularly in large-scale facilities such as logistics centers.

[0190] System Configuration

[0191] The cleaning robot system of the present invention consists of the following main components:

[0192] 1. Autonomous mobile robot equipped with a night vision camera

[0193] 2. Smartphone Applications

[0194] 3. Server

[0195] 4. Communications Infrastructure

[0196] Server Roles

[0197] The server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning routes, and means for sending a cleaning log to the server and storing it in a database.

[0198] Specifically, the server processes the map data and cleaning history sent from the robot, saving them in an integrated database, and analyzing this data to calculate efficient cleaning patterns.The main software used is PostgreSQL for database management and Python libraries for analysis.

[0199] The role of the terminal (robot)

[0200] The terminal (robot) captures the environment with a night vision camera, converts the image to grayscale, and detects dirt using optical recognition technology. It then performs cleaning operations on the detected dirt and updates the map. The hardware used is a night vision camera and an autonomous mobile robot, and the software uses ROS (Robot Operating System). This enables autonomous navigation and cleaning.

[0201] User Roles

[0202] Users can monitor the robot's status through a smartphone application and remotely control it as needed. The application displays the robot's real-time location information, allows viewing of environmental images from a night vision camera, sends alerts when dirt is detected, displays cleaning history, and checks learning data. The application was developed using ANDROID (registered trademark) Studio.

[0203] A specific example is the use of a smartphone app to check the cleaning status of warehouse area A in real time. For example, when the robot detects dirt and begins cleaning, a notification is sent to the user, who can then view footage of the site on their smartphone. If necessary, the user can remotely instruct the robot to re-clean a specific area.

[0204] Prompt Sentence Examples

[0205] Enter the following prompt into the generative AI model to generate an alert message:

[0206] "A cleaning robot using a night vision camera in a distribution center has detected dirt in a specific area. Cleaning is required as an emergency response. Please generate a message for an alert notification."

[0207] Example: "Urgent Alert: Soil detected in Warehouse Area A of the Distribution Center. Immediate management attention required."

[0208] Thus, the present invention is a cleaning robot system using a night vision camera, and provides a specific method for effectively cleaning environments such as logistics centers.

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Program processing flow and detailed explanations for each step

[0211] Step 1: Capture your environment

[0212] Subject: Terminal

[0213] The terminal (robot) captures the environment using a night vision camera. The night vision camera acquires video data of the surroundings and converts the video data into grayscale. This allows for highly accurate video data to be obtained even in dark places. The acquired video data is also temporarily stored in local storage.

[0214] Input: Surrounding video data

[0215] Output: Grayscale converted video data

[0216] Step 2: Feature extraction and location update

[0217] Subject: Terminal

[0218] The video data converted to grayscale is analyzed and features are extracted using optical recognition technology. Image analysis algorithms (such as the OpenCV library) are used for feature extraction. The current location information is updated based on the feature information obtained from this analysis. This allows the robot's position to be accurately displayed on a map.

[0219] Input: Grayscale converted video data

[0220] Output: extracted feature information, updated location information

[0221] Step 3: Create a map

[0222] Subject: Terminal

[0223] Based on the extracted feature information and location information, the system performs autonomous navigation and creates a map of the surrounding area using the SLAM (Simultaneous Localization and Mapping) algorithm. This map data is temporarily stored in the device's local memory.

[0224] Input: extracted feature information, updated location information

[0225] Output: Created map data

[0226] Step 4: Detecting Contamination

[0227] Subject: Terminal

[0228] The device detects dirt from map data created using optical recognition technology, a process that uses algorithms to identify specific colors and shapes, and when a dirt is detected, its location is recorded in a log.

[0229] Input: Created map data

[0230] Output: Specific location information of stains

[0231] Step 5: Perform cleaning actions

[0232] Subject: Terminal

[0233] When dirt is detected, the device initiates a cleaning operation, the robot moves to a specific location, performs the cleaning function, and when the cleaning is complete, the information is recorded in a log.

[0234] Input: Specific location of stain

[0235] Output: Cleaning completion log

[0236] Step 6: Sending an alert

[0237] Subject: Terminal

[0238] Once the cleaning is complete, the device sends a real-time alert that includes information about the cleaning completion and the specific location of the cleaning. The alert is sent to the smartphone application and server.

[0239] Input: Cleanup completion log

[0240] Output: Alert notification

[0241] Step 7: Save your data

[0242] Subject: Server

[0243] The server receives the map data and cleaning logs sent from the device and stores them in an integrated database, which centrally manages past cleaning history and map information.

[0244] Input: Map data, cleaning log

[0245] Output: Data stored in the integrated database

[0246] Step 8: Learn and optimize

[0247] Subject: Server

[0248] The server analyzes the stored data and uses a learning algorithm to optimize cleaning activities. This process includes identifying frequently soiled areas and calculating efficient cleaning patterns. The analysis results are reflected in the cleaning algorithm and used for future cleaning activities.

[0249] Input: Data from the integrated database

[0250] Output: Optimized cleaning algorithm

[0251] In this way, the present invention realizes efficient cleaning in large-scale facilities such as logistics centers through specific processing steps of a cleaning robot system using a night vision camera.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and achieves efficient cleaning through optical recognition technology and autonomous navigation. This system has the ability to recognize the user's emotions and adjust its behavior based on those emotions. Below, we will explain the details of the program processing, with concrete examples, divided into the subjects of the server, terminal, and user.

[0254] Server Roles and Operations

[0255] The server receives data sent from the robot and updates the integrated database. It stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[0256] example:

[0257] The server receives the map data sent from each robot and stores it in an integrated database.

[0258] The server analyzes cleaning history and emotional data to derive efficient cleaning patterns and areas for improvement.

[0259] Roles and actions of the terminal (robot)

[0260] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera and converts the image to grayscale for processing. It extracts features from the image using optical recognition technology and updates a map of the current location. It uses an autonomous navigation system to calculate the next location to go to and navigate. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[0261] example:

[0262] The device uses a night vision camera to capture and map the environment.

[0263] Cleaning is performed at the location where dirt is detected, and the cleaning status is reported to the server.

[0264] The terminal recognizes the user's emotions using an emotion engine, and operates quietly, for example, when the user is feeling stressed.

[0265] User Roles and Actions

[0266] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[0267] example:

[0268] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[0269] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[0270] The learning and optimization process

[0271] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[0272] example:

[0273] The device transmits cleaning activity and emotion data to a server and stores it in a database.

[0274] The server analyzes the data and distributes new cleaning algorithms to the robots.

[0275] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[0276] As described above, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[0277] The processing flow will be explained below.

[0278] Step 1:

[0279] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[0280] Step 2:

[0281] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale for easier further processing.

[0282] Step 3:

[0283] The device extracts features from the captured video, including the patterns on walls, furniture, and floors, and uses these features to create a map and update its current location.

[0284] Step 4:

[0285] The device uses an autonomous navigation system to calculate the next location to go to and navigate, moving while efficiently covering the designated area.

[0286] Step 5:

[0287] The device uses optical recognition technology to detect dirt and debris on the floor, analyzing images from a night vision camera and recognizing distinctive dirt patterns.

[0288] Step 6:

[0289] The device performs cleaning operations at the location where dirt is detected, including sweeping, vacuuming, wiping with water, etc. Once the cleaning is complete, the results are recorded.

[0290] Step 7:

[0291] The device sends information about the location where cleaning was completed and the cleaning results to the server in real time. The server receives this information and stores it in a database. The device also sends an alert to the user when cleaning is complete.

[0292] Step 8:

[0293] Users can check the robot's location and cleaning status in real time on their smartphone or PC, and can also instruct it to clean a specific area again.

[0294] Step 9:

[0295] The device uses an emotion engine to recognize the user's emotions. For this purpose, the device is equipped with a camera and microphone, and analyzes emotions from the user's facial expressions and voice.

[0296] Step 10:

[0297] The device adjusts its cleaning behavior based on the user's emotions, for example, if the user is feeling stressed, the device will operate in quiet mode to maintain a relaxing environment for the user.

[0298] Step 11:

[0299] The device sends cleaning activity and emotion data to the server, which stores it in a database. A learning algorithm optimizes future cleaning activities.

[0300] Step 12:

[0301] The server analyzes the data, updates the cleaning algorithm and emotion-based cleaning patterns, and feeds them back to the device, continuously improving cleaning accuracy and user satisfaction.

[0302] Example 2

[0303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] Conventional cleaning robot systems were able to accurately recognize their environment and autonomously clean efficiently, but they were unable to adjust their behavior based on the user's emotions, making it difficult to clean in a way that took the user's comfort into consideration. Furthermore, they lacked real-time cleaning status reports and adequate optimization of cleaning activities, creating challenges in improving cleaning efficiency.

[0305] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting the cleaning status to the server in real time, a means for recognizing the user's emotion using an emotion recognition engine, and a means for adjusting the cleaning mode based on the user's emotion. This enables the terminal to operate in accordance with the user's emotion, providing comfortable cleaning and monitoring the cleaning status in real time, thereby realizing efficient cleaning activities.

[0306] A "night vision camera" is a camera that can capture clear images even in dark environments.

[0307] The "means for capturing the environment" is a method for capturing an image of the surrounding environment using a night vision camera.

[0308] The "means for processing the image to extract features and update the position information" is a method for extracting important data from the captured image and calculating and updating the current position of the robot.

[0309] "Means for autonomous navigation and mapping" refers to a method that uses optical recognition technology and sensors to determine one's own position, plan a route, and generate a map of the surrounding area.

[0310] "Optical recognition technology" is a technology that uses image processing technology to identify objects and features.

[0311] A "means for detecting soiling" is a method for identifying soiling on a surface using optical recognition technology.

[0312] The "means for performing a cleaning action" refers to an action that the robot takes to remove the detected dirt, such as suction or brushing.

[0313] "Means for transmitting cleaning status to a server in real time" refers to a method for transmitting data on the progress and status of cleaning to a server in real time.

[0314] An "emotion recognition engine" is software or hardware that identifies the emotional state of a user from their voice and facial expressions.

[0315] The "means for recognizing the user's emotions" is a method for determining the user's emotional state using an emotion recognition engine.

[0316] The "means for adjusting the cleaning mode" is a method for changing the operation mode of the cleaning robot based on the emotional state of the user.

[0317] A "learning algorithm" is an artificial intelligence technology that uses past data and patterns to optimize future actions.

[0318] "Means to optimize cleaning activities and improve future cleaning routes" refers to a method that uses a learning algorithm to calculate efficient cleaning patterns and routes.

[0319] "Means for customizing cleaning patterns using user emotional data" refers to a method for designing an optimal cleaning plan for a user based on collected emotional data.

[0320] A "cleaning log" is a record of cleaning activities performed by a robot.

[0321] A "database" is a system for systematically organizing and storing information.

[0322] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and uses optical recognition technology and autonomous navigation to achieve efficient cleaning. The system has the ability to recognize the user's emotions and adjust its behavior based on those emotions.

[0323] Server Roles and Operations

[0324] The server receives data sent from the robot and updates the integrated database. For example, it stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[0325] Specific behavior:

[0326] The server receives the map data sent by each robot and stores it in an integrated database, accumulating information such as the overall image of the room and the cleaning frequency of each area.

[0327] The server analyzes cleaning history and emotion data to derive efficient cleaning patterns and areas for improvement. For example, if it finds that a particular area gets dirty frequently during a certain time period, it will create a cleaning plan to focus on that area.

[0328] Roles and actions of the terminal (robot)

[0329] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera, converts the image to grayscale, and processes it. It uses optical recognition technology to extract features from the image and updates a map of its current location. It uses an autonomous navigation system to calculate and navigate to the next location. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[0330] Specific behavior:

[0331] The device uses a night vision camera to capture and map the environment, for example, accurately detecting furniture placement and obstacles even in the dark and generating a movement map.

[0332] The device performs cleaning at the location where the dirt is detected and reports the cleaning status to the server. For example, it detects a drink spilled on a carpet and soaks it up on the spot.

[0333] The device uses an emotion engine to recognize the user's emotions and, for example, operates quietly if the user is feeling stressed. For example, if the user wants to relax at night, the device will clean in quiet mode with reduced noise.

[0334] User Roles and Actions

[0335] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[0336] Specific behavior:

[0337] Users can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. For example, they can check using the smartphone app to see if a particular room has been cleaned.

[0338] The user receives emotional feedback from the robot and can instruct it to change its operating mode as needed, for example, cleaning in normal mode during the day and switching to quiet mode at night.

[0339] The learning and optimization process

[0340] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[0341] Specific behavior:

[0342] The device sends cleaning activity and emotion data to the server and stores it in a database. For example, after cleaning is completed, the device uploads the cleaning activity and emotion data of the user for that day.

[0343] The server analyzes the data and distributes new cleaning algorithms to the robot, such as calculating and directing efficient cleaning routes for specific times of the day or days of the week.

[0344] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[0345] Examples of prompts for generative AI models

[0346] Example prompt:

[0347] "Please explain how the cleaning robot captures its environment with a night vision camera. Please also explain the basic image processing flow and autonomous navigation."

[0348] Please explain, with specific algorithms and examples, how a cleaning robot can recognize a user's emotions and adjust its behavior accordingly.

[0349] In this way, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[0350] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0351] Step 1: Initialization and Data Collection

[0352] When the system is powered on, the terminal (robot) performs an initialization process. It starts up the night vision camera, sensors, navigation module, and emotion engine, and collects initial data on the working area. Specifically, it uses the night vision camera to capture the surrounding environment, converts the image to grayscale, and processes it. At the same time, it obtains location information and obstacle distance data from various sensors (such as LIDAR and infrared sensors).

[0353] input:

[0354] Powering on the system

[0355] Environmental data from night vision cameras and sensors

[0356] Data processing / data calculation:

[0357] Converting an image to grayscale

[0358] Sensor information acquisition and integration

[0359] output:

[0360] Initialization complete message

[0361] Environmental Datasets

[0362] Specific behavior:

[0363] The device captures an initial image with its night vision camera and records the location of each obstacle.

[0364] Step 2: Data processing and mapping

[0365] The device processes the collected data and creates a map of the room. Specifically, it uses optical recognition technology to extract feature points from grayscale images and generate a map based on each piece of data. An autonomous navigation system then updates the device's position and surrounding terrain in real time.

[0366] input:

[0367] Environmental Datasets

[0368] Grayscale images

[0369] Data processing / data calculation:

[0370] Extracting feature points from images

[0371] Running the map generation algorithm

[0372] output:

[0373] Generated map data

[0374] Current location information

[0375] Specific behavior:

[0376] The device applies edge detection algorithms to identify the location of furniture and walls.

[0377] The terminal generates a map and passes it to the navigation algorithm.

[0378] Step 3: Navigation and Cleaning Plan

[0379] The device uses map data and navigation algorithms to plan the optimal cleaning route, with the navigation system calculating the next location and guiding the robot.

[0380] input:

[0381] Map data

[0382] Current location information

[0383] Data processing / data calculation:

[0384] Execution of the navigation algorithm

[0385] Cleaning route calculation

[0386] output:

[0387] Cleaning route planning

[0388] Next Movement Point

[0389] Specific behavior:

[0390] The device uses algorithms such as the A algorithm to determine an efficient cleaning route while avoiding obstacles.

[0391] The terminal sets the cleaning pattern and indicates the cleaning sequence.

[0392] Step 4: Cleaning execution and real-time monitoring

[0393] The device moves along a planned cleaning route, performs cleaning actions when dirt is detected, and transmits the cleaning status to the server in real time.

[0394] input:

[0395] Cleaning route planning

[0396] Dirt detection data

[0397] Data processing / data calculation:

[0398] Running the dirt detection algorithm

[0399] Cleaning trigger

[0400] output:

[0401] Cleaning progress data

[0402] Real-time upload to server

[0403] Specific behavior:

[0404] The device uses suction or a brush to clean the area where dirt is detected.

[0405] The terminal reports cleaning progress to the server in real time.

[0406] Step 5: Emotion recognition and behavior regulation

[0407] The device uses an emotion engine to recognize the user's emotions and adjusts its behavior based on that information.

[0408] input:

[0409] User voice and facial expression data

[0410] Analysis results from the emotion engine

[0411] Data processing / data calculation:

[0412] Applying speech and facial expression recognition algorithms

[0413] Emotional state estimation

[0414] output:

[0415] Emotion analysis results

[0416] Operation adjustment command

[0417] Specific behavior:

[0418] If the user is in a stressful state, the terminal switches to a silent mode that reduces the operating noise.

[0419] Step 6: Data analysis and feedback

[0420] The server analyzes the cleaning data and emotion data sent from the device, updates the efficient cleaning algorithm, and provides feedback to the device on a new cleaning plan based on the analysis results.

[0421] input:

[0422] Cleaning Data

[0423] Emotional Data

[0424] Data processing / data calculation:

[0425] Executing data analysis algorithms

[0426] Optimized cleaning algorithm

[0427] output:

[0428] Updated cleaning algorithm

[0429] Feedback data to the device

[0430] Specific behavior:

[0431] The server analyzes cleaning frequency and effective cleaning order and sends the results to the terminal.

[0432] Step 7: User monitoring and guidance

[0433] Users can use a smartphone or tablet to monitor the robot's operating status and issue instructions to the robot as needed.

[0434] input:

[0435] Instructions from a smartphone app

[0436] Real-time Monitoring Data

[0437] Data processing / data calculation:

[0438] Analysis of instruction data

[0439] output:

[0440] Commands and instructions for robots

[0441] Situation Report Data

[0442] Specific behavior:

[0443] Users send specific instructions to the robot through the app, such as "prioritize cleaning the living room."

[0444] (Application example 2)

[0445] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0446] Efficient cleaning activities are important in modern factories, but the effectiveness of automated cleaning systems is limited in environments with low visibility at night. Furthermore, while there is a need to adjust operations based on worker emotions, conventional systems lack emotion recognition capabilities and are therefore unable to meet this requirement. Furthermore, it is difficult to integrate cleaning activity data with employee emotion data to provide optimized cleaning patterns.

[0447] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for detecting dirt using optical recognition technology, means for autonomously navigating and creating a map, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for adjusting operations based on the user's emotions with an emotion recognition function, means for optimizing cleaning activities using a learning algorithm and improving subsequent cleaning routes, and means for transmitting cleaning logs and emotion data to the server and storing them in a database. This enables effective cleaning operations even in environments with low visibility at night, realizes operation adjustments based on the user's emotions, and provides optimized cleaning patterns.

[0448] A "night vision camera" is a camera device that can capture clear images of objects even in low-light environments.

[0449] "Capturing" refers to the process of taking in video or images.

[0450] "Feature extraction" means finding important information or patterns in captured video or images.

[0451] "Updating location information" means recording the current location and coordinate data and replacing it with the latest information.

[0452] "Autonomous navigation" means determining the direction of travel and moving independently without external instructions.

[0453] "Creating a map" means recording the surrounding environment and the route you are taking and generating visualized data.

[0454] "Optical recognition technology" is a technology that uses cameras and sensors to determine the shape and characteristics of objects.

[0455] "Detecting dirt" means finding dirt or abnormalities on the surface.

[0456] "Performing a cleaning action" means taking action to remove the detected dirt.

[0457] "Real-time alerting" means being notified immediately of an event or situation that occurs.

[0458] The "emotion recognition function" is a function that determines a person's emotions from video and audio data.

[0459] "Adjusting behavior" means changing the way equipment or systems behave depending on the situation or conditions.

[0460] A "learning algorithm" is a computational method for analyzing accumulated data and discovering new knowledge and patterns.

[0461] "Optimizing cleaning activities" means finding ways to make cleaning tasks more efficient and effective.

[0462] "Improving the cleaning route from next time onwards" means improving the route for the next cleaning work based on past data.

[0463] "Cleaning log and emotional data" refers to a record of cleaning activities and data indicating the user's emotional state.

[0464] "Send to server" means sending data over the network to a central management system.

[0465] "Storing in a database" means organizing and storing the acquired data.

[0466] Server Roles and Operations

[0467] The server captures the environment using a night vision camera and receives data from the device that processes the image and extracts features. This process includes converting the captured image to grayscale and detecting dirt using optical recognition technology. The server also stores navigation data, map data, cleaning history, and emotion data from emotion recognition in an integrated database to support efficient cleaning activities. Furthermore, the server analyzes data sent from each robot and identifies frequently soiled areas to provide an optimal cleaning plan for future cleanings.

[0468] Examples:

[0469] The server stores the map data and cleaning history sent from the factory's nighttime cleaning robots in an integrated database.

[0470] The server analyzes the cleaning logs and emotional data to customize and optimize the robot's cleaning patterns.

[0471] Roles and actions of the terminal (robot)

[0472] The terminal (robot) autonomously moves around the factory to perform cleaning activities. The robot is equipped with a night vision camera, captures the environment, and converts the images to grayscale to detect dirt. It uses an autonomous navigation system to move efficiently while updating a map of its current location. It performs cleaning operations at points where dirt is detected and sends the results to a server in real time after cleaning is completed. It also uses emotion recognition functionality to recognize the emotions of nearby workers and adjust its behavior accordingly. For example, if a worker is feeling stressed, the robot will operate in silent mode.

[0473] Examples:

[0474] The robot uses night vision cameras to capture and map the environment in the factory at night.

[0475] It performs cleaning wherever dirt is detected and reports the results to the server in real time.

[0476] Emotion recognition allows the robot to switch to silent mode if the worker is feeling stressed.

[0477] User Roles and Actions

[0478] Users can remotely monitor the robot's status and issue instructions as needed. Using a smartphone or other device, users can monitor the robot's location, cleaning status, and emotional data in real time. Furthermore, the operating mode can be switched based on the emotional feedback received from the robot. For example, the robot can be set to operate in quiet mode when family members or redundant staff are relaxing. In this way, users can perform optimal cleaning operations that are tailored to the factory environment and the emotions of employees.

[0479] Examples:

[0480] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[0481] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[0482] The learning and optimization process

[0483] The device (robot) sends cleaning activity and emotion recognition data to a server, which analyzes the data. Based on the analysis results, a learning algorithm is used to optimize cleaning activities and improve cleaning routes and patterns for future cleanings. This improves the accuracy and efficiency of cleaning activities, even in environments with low visibility at night. In addition, the pattern of each cleaning activity can be customized based on the user's emotion data, allowing for even more detailed optimization.

[0484] Examples:

[0485] The terminal (robot) transmits cleaning activity and emotion data to the server and stores it in a database.

[0486] The server analyzes the data sent and distributes the cleaning algorithm to the robot for the next cleaning cycle.

[0487] Based on the user's emotional data, it creates a customized cleaning pattern and optimizes cleaning activities.

[0488] Prompt Sentence Examples

[0489] In factories, cleaning robots use night vision cameras to capture the environment and detect dirt. They use emotion recognition to analyze the facial expressions of workers while they work. If the robot detects "emotion: stress," for example, it switches to silent mode. Cleaning status and emotion data are sent to a server in real time, allowing it to learn optimal cleaning patterns.

[0490] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0491] Step 1:

[0492] The device uses a night vision camera to capture the environment. The input is the image captured by the camera, and the output is the video data, which is the basis for capturing detailed information, especially in low-light environments.

[0493] Step 2:

[0494] The device converts the captured video to grayscale and extracts features. This conversion process removes unnecessary color information, making it easier to detect dirt and obstacles. The input is the video data obtained in step 1, and the output is the grayscale image and extracted feature data.

[0495] Step 3:

[0496] The device detects dirt using optical recognition technology. The input is a grayscale image and feature data, and it determines whether dirt is present based on specific patterns and features. The output is the presence or absence of dirt and its location. For example, it performs noise reduction and edge detection to identify areas that appear to be dirty.

[0497] Step 4:

[0498] The device uses an autonomous navigation system to create a map and update its current location. The input is the device's current location information and feature data acquired from the environment, and the output is updated map information and location information. Based on this, the device can calculate where to go next and move efficiently.

[0499] Step 5:

[0500] The device performs cleaning operations at the location where the stain is detected. The input is the location of the stain, and the output is a cleaning completion report. The device's cleaning unit moves to the stained location and performs the specific stain removal task.

[0501] Step 6:

[0502] The device sends a real-time alert after cleaning. The input is the cleaning completion report and its location information, and the output is notification data sent to the server, allowing the server to keep track of the progress of each cleaning activity.

[0503] Step 7:

[0504] The device uses emotion recognition to recognize the user's emotions and adjusts its behavior based on those emotions. The input is camera video data, and the output is the emotion recognition results and the adjusted behavior mode. For example, if the user is feeling stressed, the device can switch to silent mode.

[0505] Step 8:

[0506] The server stores all data (cleaning logs, emotion data, and map information) in an integrated database. The input is the data sent from the device, and the output is the saved database state. The server analyzes this and uses it in an algorithm to optimize future cleaning patterns.

[0507] Step 9:

[0508] The user can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. The input is real-time data sent from the server, and the output is information displayed on the monitoring screen. The user can issue commands to switch the operating mode as needed.

[0509] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0510] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0511] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0512] [Second embodiment]

[0513] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0514] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0515] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0516] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0517] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0518] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0519] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0520] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0521] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0522] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0523] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0524] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0525] The present invention relates to a cleaning robot system using a night vision camera, and specific embodiments thereof will be described below. Details of program processing will be explained with specific examples, divided into the subjects of the server, terminal, and user.

[0526] overview

[0527] The cleaning robot system first captures the environment using a night vision camera, analyzes the footage, and extracts features. It then creates a map using an autonomous navigation system and detects dirt using optical recognition technology. When dirt is detected, the robot performs cleaning operations and sends the results as an alert in real time. It also uses a learning algorithm to optimize cleaning activities, and sends cleaning logs to a server for storage in a database. This ensures efficient and accurate late-night cleaning.

[0528] Server Roles and Operations

[0529] The server receives data sent from the robot and updates the integrated database. Specifically, it stores information such as navigation data, map data, and cleaning history, and uses it as a reference for the robot to perform cleaning activities efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[0530] example:

[0531] The server receives the map data sent from each robot and stores it in an integrated database.

[0532] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[0533] Roles and actions of the terminal (robot)

[0534] The terminal (robot) operates autonomously and performs cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins processing. Next, optical recognition technology is used to extract features from the image and update the map of the current location. After that, it uses an autonomous navigation system to calculate the next position to go to, and if dirt is detected, it performs cleaning operations.

[0535] example:

[0536] The device uses a night vision camera to capture and map the environment at night.

[0537] The device will start cleaning when it detects dirt in a specific location.

[0538] User Roles and Actions

[0539] Users can remotely monitor the robot's status and issue instructions as needed. Users receive real-time alerts to see the cleaning robot's activity and can also give instructions to re-clean specific designated areas.

[0540] example:

[0541] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[0542] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[0543] The learning and optimization process

[0544] The robot learns using cleaning activity data and map data, optimizing future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[0545] example:

[0546] The device sends data on cleaning activities to a server, which stores it in a database.

[0547] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[0548] The processing flow will be explained below.

[0549] Step 1:

[0550] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[0551] Step 2:

[0552] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale to facilitate further processing.

[0553] Step 3:

[0554] The device extracts features from the captured video and updates a map of the device's current location based on the extracted features, accurately determining the device's location.

[0555] Step 4:

[0556] The device uses an autonomous navigation system to calculate and navigate its next location, allowing the device to efficiently cover the entire area.

[0557] Step 5:

[0558] The device detects dirt using optical recognition technology, capturing the environment with a night vision camera and analyzing the footage to identify dirt and foreign objects.

[0559] Step 6:

[0560] The device will perform cleaning operations at the location where the dirt is detected, and once the cleaning operation is complete, the device will send the results to the server in real time and also send an alert that the cleaning is complete.

[0561] Step 7:

[0562] The server stores the cleaning log received from the device in a database, analyzes the cleaning history and map data, derives efficient cleaning patterns, updates the learning algorithm, and provides feedback to the device on an optimized navigation system.

[0563] Step 8:

[0564] Users remotely monitor the device's location and cleaning status in real time, and can send instructions to the device to re-clean specific areas as needed.

[0565] Step 9:

[0566] The device receives updated information from the server and instructions from the user to continue cleaning, and then initiates the next cleaning activity accordingly. Through this cyclical process, the device continuously improves its cleaning efficiency and contributes to environmental conservation.

[0567] Example 1

[0568] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0569] Current cleaning robot systems have difficulty performing efficient and accurate cleaning at night, and there are many challenges with dirt detection and map creation, especially in environments that use night vision. This reduces cleaning accuracy and efficiency, and increases time and costs. Furthermore, few systems utilize cleaning activity logs to further improve efficiency and optimization. To solve these problems, a more accurate and efficient cleaning robot system is needed.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0571] In this invention, the server includes means for capturing the environment using a night vision camera, means for converting the captured image to grayscale to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, and means for sending cleaning data to the server and storing it in an integrated database. This makes it possible to detect dirt with high accuracy even in nighttime environments, perform efficient cleaning, and optimize future cleanings by utilizing cleaning activity log data.

[0572] A "night vision camera" is a special imaging device that can capture images even in darkness or low-light environments.

[0573] The "means for capturing the environment" is a method for capturing an image of the surroundings using a night vision camera.

[0574] "Means for converting to grayscale" refers to a method for converting the acquired image into a format that is expressed in shades of black and white.

[0575] The "means for extracting features and updating position information" is a method for detecting specific patterns or objects from the image converted to grayscale and using that information to update the robot's position.

[0576] "Means for autonomous navigation and map creation" refers to a method in which a robot determines its own route and generates a map based on information about its surroundings.

[0577] "Optical recognition technology" is a technology that analyzes video data acquired by a camera and recognizes objects and features.

[0578] The "means for detecting dirt" is a method for identifying dirt from captured images using optical recognition technology.

[0579] "Means for performing cleaning actions" refers to a series of actions that the robot takes to remove the detected soiling.

[0580] "Means for sending real-time alerts" refers to a method for sending immediate notification to users or systems after cleaning is completed.

[0581] "Means for transmitting cleaning data to a server and storing it in an integrated database" refers to a method for transmitting records of cleaning activities to a server via the Internet or the like and storing them there.

[0582] The present invention relates to an autonomous cleaning robot system using a night vision camera, and specific embodiments thereof will be described below.

[0583] Server Roles and Operations

[0584] The server receives data sent from the robot (navigation data, map data, cleaning history, etc.) and stores it in an integrated database. The server analyzes the information stored in the database and identifies areas that are frequently soiled. This allows for the creation of an efficient and effective cleaning plan. The server also analyzes the cleaning history, generates new cleaning algorithms, and distributes them to the terminal (robot).

[0585] Examples:

[0586] The server receives the map data sent from each robot and stores it in an integrated database.

[0587] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[0588] Roles and actions of the terminal (robot)

[0589] The terminal (robot) operates autonomously to carry out cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins image processing. Optical recognition technology is used to extract features from the grayscale image and update the robot's current location on a map. Next, an autonomous navigation system is used to calculate an appropriate route, and if dirt is detected, it immediately begins cleaning.

[0590] Examples:

[0591] The device uses a night vision camera to capture and map the environment at night.

[0592] The device will start cleaning when it detects dirt in a specific location.

[0593] User Roles and Actions

[0594] Users can remotely monitor the robot's status. They can check the robot's location and cleaning status in real time using a smartphone or other device. If necessary, users can also issue instructions to re-clean a specific area. If an alert occurs, they can quickly check the details and take appropriate action.

[0595] Examples:

[0596] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[0597] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[0598] The learning and optimization process

[0599] The device (robot) learns from cleaning activity data and map data and optimizes future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[0600] Examples:

[0601] The device sends data on cleaning activities to a server, which stores it in a database.

[0602] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[0603] Example prompt for a generative AI model:

[0604] "Please explain how to save map data of a cleaning robot acquired at night by a night vision camera to a server."

[0605] "Explain how you can use a night vision camera to capture video, convert it to grayscale, detect dirt, and update the map."

[0606] "Explain how you can monitor the status of your cleaning robot on your smartphone and receive alerts to re-clean specific areas."

[0607] "Explain how data collected from a cleaning robot can be used to optimize the next cleaning session."

[0608] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0609] Step 1:

[0610] The terminal (robot) activates the night vision camera and starts capturing the environment. The night vision camera acquires images of the environment and sends the data to the computer vision system. The input is the raw image captured by the night vision camera. The output is the image data itself.

[0611] Specific behavior:

[0612] The device switches to night setting mode.

[0613] Activate the night vision camera and capture video continuously.

[0614] Step 2:

[0615] The device converts the captured video to grayscale. The input is color video data acquired from a night vision camera, and the output is grayscale video data. This conversion makes the video data easier to analyze.

[0616] Specific behavior:

[0617] The terminal activates an image processing module to convert the color image into grayscale format.

[0618] The converted grayscale image is stored in a temporary memory.

[0619] Step 3:

[0620] The device uses optical recognition technology to extract features from the grayscale image and update the location information. The input is the grayscale image data obtained in step 2, and the output is feature data and updated location information. The feature data indicates important parts of the image, such as dirt or obstacles.

[0621] Specific behavior:

[0622] The device runs a feature extraction algorithm to recognize dirt and obstacles from the video data.

[0623] The current location information is updated based on the extracted features.

[0624] Step 4:

[0625] The device updates the map using the autonomous navigation system. The input is the feature data and location information obtained in step 3, and the output is updated map data. This map is modified by adding new data based on the known range.

[0626] Specific behavior:

[0627] The device runs a navigation algorithm and updates the map to reflect the current location and feature data.

[0628] The updated map is stored in memory and prepared for the next action.

[0629] Step 5:

[0630] The device performs cleaning operation when dirt is detected. The input is the map data updated in step 4 and the location information of the dirt, and the output is the cleaning completion data.

[0631] Specific behavior:

[0632] An autonomous navigation system is used to navigate to the location where the contamination is detected.

[0633] Activates the cleaning module and cleans the designated area.

[0634] Step 6:

[0635] After cleaning, the terminal sends a real-time alert to the server. The input is the cleaning completion data, and the output is the alert notification to the server.

[0636] Specific behavior:

[0637] The terminal activates the communication module and transmits cleaning completion data to the server.

[0638] An alert is sent to the server and the user is notified.

[0639] Step 7:

[0640] The server receives the cleaning data sent from the terminal and stores it in an integrated database. The input is the cleaning data sent from the terminal, and the output is the history data stored in the database.

[0641] Specific behavior:

[0642] The server starts the communication module and receives data from the terminal.

[0643] The received data is analyzed and stored in an integrated database.

[0644] Step 8:

[0645] The server analyzes the received cleaning data and generates a new cleaning algorithm. The input is the stored cleaning history data, and the output is the updated cleaning algorithm.

[0646] Specific behavior:

[0647] The server launches a data analysis module and calculates an efficient cleaning pattern based on past cleaning history.

[0648] Generate new algorithms and prepare them for distribution to the robots.

[0649] Step 9:

[0650] The user remotely monitors the robot's status and issues instructions for re-cleaning as necessary. The input is alert notifications from the server and the robot's location information, and the output is instructions for re-cleaning.

[0651] Specific behavior:

[0652] The user launches the smartphone app and checks the robot's status.

[0653] If necessary, instructions to re-clean specific areas will be sent via the app.

[0654] (Application example 1)

[0655] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0656] In large facilities such as logistics centers, efficient cleaning is difficult in environments with poor visibility at night. Furthermore, to efficiently clean large areas, it is necessary to manage the robot's real-time location information, detect dirt, and respond immediately. However, current cleaning systems lack the means to effectively solve these issues. There is also a need for a method to optimize the efficiency of cleaning activities and improve future cleaning routes based on past cleaning data.

[0657] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0658] In this invention, the server includes: means for capturing the environment using a night vision camera; means for processing the captured image to extract features and update location information; means for autonomously navigating and creating maps; means for detecting dirt using optical recognition technology; means for performing cleaning operations to clean the detected dirt; means for sending real-time alerts after cleaning; means including a smartphone application for displaying the robot's real-time location information; means for viewing the environmental image captured by the night vision camera; means for issuing alerts when dirt is detected; means for remotely operating the robot; and means for displaying cleaning history and checking learning data. This enables efficient and accurate cleaning even in environments with poor visibility, even at night, and enables real-time status monitoring and immediate response. Furthermore, it enables optimization of cleaning activities and improvement of cleaning routes based on past data.

[0659] A "night vision camera" is a camera used to ensure visibility in the dark, and is a device that uses infrared light to capture the environment.

[0660] The "means for capturing the environment" is a mechanism that has the function of acquiring the surrounding situation as video data using a night vision camera.

[0661] The "means for extracting features and updating location information" is a technology that detects specific patterns or objects from captured video data and updates the current location information based on that.

[0662] "Means for autonomous navigation and map creation" refers to a system that automatically moves based on acquired location information, aggregates information from each location, and generates a map.

[0663] "Optical recognition technology" is a technology that uses optical devices such as cameras to detect specific objects or features from image data.

[0664] The "means for detecting dirt" is a system that uses optical recognition technology to identify dirt from video data and pinpoint its location.

[0665] "Means for performing cleaning actions" refers to a series of operations that the robot performs to remove the detected dirt.

[0666] "Means for sending alerts in real time" refers to a system that has a communication function to immediately notify users of detected dirt and the cleaning status after cleaning.

[0667] The "smartphone application for displaying real-time location information of a robot" is an application that displays the current location of a cleaning robot on a smartphone in real time.

[0668] "Means for viewing environmental images captured by a night vision camera" refers to a mechanism for displaying images captured by a night vision camera on a smartphone or other device.

[0669] "Means for issuing an alert notification when dirt is detected" refers to a mechanism that has the function of sending an immediate notification when dirt is confirmed.

[0670] "Means for remotely operating a robot" refers to remote control technology for remotely controlling the operation of a cleaning robot.

[0671] "Means for displaying cleaning history and checking learning data" refers to a mechanism that displays the history of past cleaning activities and learns from the acquired data to improve the cleaning algorithm.

[0672] This section describes a specific embodiment of a cleaning robot system using a night vision camera. This invention utilizes an autonomously operating cleaning robot to achieve efficient cleaning, particularly in large-scale facilities such as logistics centers.

[0673] System Configuration

[0674] The cleaning robot system of the present invention consists of the following main components:

[0675] 1. Autonomous mobile robot equipped with a night vision camera

[0676] 2. Smartphone Applications

[0677] 3. Server

[0678] 4. Communications Infrastructure

[0679] Server Roles

[0680] The server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning routes, and means for sending a cleaning log to the server and storing it in a database.

[0681] Specifically, the server processes the map data and cleaning history sent from the robot, saving them in an integrated database, and analyzing this data to calculate efficient cleaning patterns.The main software used is PostgreSQL for database management and Python libraries for analysis.

[0682] The role of the terminal (robot)

[0683] The terminal (robot) captures the environment with a night vision camera, converts the image to grayscale, and detects dirt using optical recognition technology. It then performs cleaning operations on the detected dirt and updates the map. The hardware used is a night vision camera and an autonomous mobile robot, and the software uses ROS (Robot Operating System). This enables autonomous navigation and cleaning.

[0684] User Roles

[0685] Users can monitor the robot's status through a smartphone application and remotely control it as needed. The application displays the robot's real-time location information, allows viewing of environmental images from a night vision camera, sends alerts when dirt is detected, displays cleaning history, and checks learning data. The application was developed using Android Studio.

[0686] A specific example is the use of a smartphone app to check the cleaning status of warehouse area A in real time. For example, when the robot detects dirt and begins cleaning, a notification is sent to the user, who can then view footage of the site on their smartphone. If necessary, the user can remotely instruct the robot to re-clean a specific area.

[0687] Prompt Sentence Examples

[0688] Enter the following prompt into the generative AI model to generate an alert message:

[0689] "A cleaning robot using a night vision camera in a distribution center has detected dirt in a specific area. Cleaning is required as an emergency response. Please generate a message for an alert notification."

[0690] Example: "Urgent Alert: Soil detected in Warehouse Area A of the Distribution Center. Immediate management attention required."

[0691] Thus, the present invention is a cleaning robot system using a night vision camera, and provides a specific method for effectively cleaning environments such as logistics centers.

[0692] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0693] Program processing flow and detailed explanations for each step

[0694] Step 1: Capture your environment

[0695] Subject: Terminal

[0696] The terminal (robot) captures the environment using a night vision camera. The night vision camera acquires video data of the surroundings and converts the video data into grayscale. This allows for highly accurate video data to be obtained even in dark places. The acquired video data is also temporarily stored in local storage.

[0697] Input: Surrounding video data

[0698] Output: Grayscale converted video data

[0699] Step 2: Feature extraction and location update

[0700] Subject: Terminal

[0701] The video data converted to grayscale is analyzed and features are extracted using optical recognition technology. Image analysis algorithms (such as the OpenCV library) are used for feature extraction. The current location information is updated based on the feature information obtained from this analysis. This allows the robot's position to be accurately displayed on a map.

[0702] Input: Grayscale converted video data

[0703] Output: extracted feature information, updated location information

[0704] Step 3: Create a map

[0705] Subject: Terminal

[0706] Based on the extracted feature information and location information, the system performs autonomous navigation and creates a map of the surrounding area using the SLAM (Simultaneous Localization and Mapping) algorithm. This map data is temporarily stored in the device's local memory.

[0707] Input: extracted feature information, updated location information

[0708] Output: Created map data

[0709] Step 4: Detecting Contamination

[0710] Subject: Terminal

[0711] The device detects dirt from map data created using optical recognition technology, a process that uses algorithms to identify specific colors and shapes, and when a dirt is detected, its location is recorded in a log.

[0712] Input: Created map data

[0713] Output: Specific location information of stains

[0714] Step 5: Perform cleaning actions

[0715] Subject: Terminal

[0716] When dirt is detected, the device initiates a cleaning operation, the robot moves to a specific location, performs the cleaning function, and when the cleaning is complete, the information is recorded in a log.

[0717] Input: Specific location of stain

[0718] Output: Cleaning completion log

[0719] Step 6: Sending an alert

[0720] Subject: Terminal

[0721] Once the cleaning is complete, the device sends a real-time alert that includes information about the cleaning completion and the specific location of the cleaning. The alert is sent to the smartphone application and server.

[0722] Input: Cleanup completion log

[0723] Output: Alert notification

[0724] Step 7: Save your data

[0725] Subject: Server

[0726] The server receives the map data and cleaning logs sent from the device and stores them in an integrated database, which centrally manages past cleaning history and map information.

[0727] Input: Map data, cleaning log

[0728] Output: Data stored in the integrated database

[0729] Step 8: Learn and optimize

[0730] Subject: Server

[0731] The server analyzes the stored data and uses a learning algorithm to optimize cleaning activities. This process includes identifying frequently soiled areas and calculating efficient cleaning patterns. The analysis results are reflected in the cleaning algorithm and used for future cleaning activities.

[0732] Input: Data from the integrated database

[0733] Output: Optimized cleaning algorithm

[0734] In this way, the present invention realizes efficient cleaning in large-scale facilities such as logistics centers through specific processing steps of a cleaning robot system using a night vision camera.

[0735] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0736] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and achieves efficient cleaning through optical recognition technology and autonomous navigation. This system has the ability to recognize the user's emotions and adjust its behavior based on those emotions. Below, we will explain the details of the program processing, with concrete examples, divided into the subjects of the server, terminal, and user.

[0737] Server Roles and Operations

[0738] The server receives data sent from the robot and updates the integrated database. It stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[0739] example:

[0740] The server receives the map data sent from each robot and stores it in an integrated database.

[0741] The server analyzes cleaning history and emotional data to derive efficient cleaning patterns and areas for improvement.

[0742] Roles and actions of the terminal (robot)

[0743] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera and converts the image to grayscale for processing. It extracts features from the image using optical recognition technology and updates a map of the current location. It uses an autonomous navigation system to calculate the next location to go to and navigate. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[0744] example:

[0745] The device uses a night vision camera to capture and map the environment.

[0746] Cleaning is performed at the location where dirt is detected, and the cleaning status is reported to the server.

[0747] The terminal recognizes the user's emotions using an emotion engine, and operates quietly, for example, when the user is feeling stressed.

[0748] User Roles and Actions

[0749] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[0750] example:

[0751] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[0752] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[0753] The learning and optimization process

[0754] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[0755] example:

[0756] The device transmits cleaning activity and emotion data to a server and stores it in a database.

[0757] The server analyzes the data and distributes new cleaning algorithms to the robots.

[0758] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[0759] As described above, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[0760] The processing flow will be explained below.

[0761] Step 1:

[0762] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[0763] Step 2:

[0764] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale for easier further processing.

[0765] Step 3:

[0766] The device extracts features from the captured video, including the patterns on walls, furniture, and floors, and uses these features to create a map and update its current location.

[0767] Step 4:

[0768] The device uses an autonomous navigation system to calculate the next location to go to and navigate, moving while efficiently covering the designated area.

[0769] Step 5:

[0770] The device uses optical recognition technology to detect dirt and debris on the floor, analyzing images from a night vision camera and recognizing distinctive dirt patterns.

[0771] Step 6:

[0772] The device performs cleaning operations at the location where dirt is detected, including sweeping, vacuuming, wiping with water, etc. Once the cleaning is complete, the results are recorded.

[0773] Step 7:

[0774] The device sends information about the location where cleaning was completed and the cleaning results to the server in real time. The server receives this information and stores it in a database. The device also sends an alert to the user when cleaning is complete.

[0775] Step 8:

[0776] Users can check the robot's location and cleaning status in real time on their smartphone or PC, and can also instruct it to clean a specific area again.

[0777] Step 9:

[0778] The device uses an emotion engine to recognize the user's emotions. For this purpose, the device is equipped with a camera and microphone, and analyzes emotions from the user's facial expressions and voice.

[0779] Step 10:

[0780] The device adjusts its cleaning behavior based on the user's emotions, for example, if the user is feeling stressed, the device will operate in quiet mode to maintain a relaxing environment for the user.

[0781] Step 11:

[0782] The device sends cleaning activity and emotion data to the server, which stores it in a database. A learning algorithm optimizes future cleaning activities.

[0783] Step 12:

[0784] The server analyzes the data, updates the cleaning algorithm and emotion-based cleaning patterns, and feeds them back to the device, continuously improving cleaning accuracy and user satisfaction.

[0785] Example 2

[0786] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0787] Conventional cleaning robot systems were able to accurately recognize their environment and autonomously clean efficiently, but they were unable to adjust their behavior based on the user's emotions, making it difficult to clean in a way that took the user's comfort into consideration. Furthermore, they lacked real-time cleaning status reports and adequate optimization of cleaning activities, creating challenges in improving cleaning efficiency.

[0788] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting the cleaning status to the server in real time, a means for recognizing the user's emotion using an emotion recognition engine, and a means for adjusting the cleaning mode based on the user's emotion. This enables the terminal to operate in accordance with the user's emotion, providing comfortable cleaning and monitoring the cleaning status in real time, thereby realizing efficient cleaning activities.

[0789] A "night vision camera" is a camera that can capture clear images even in dark environments.

[0790] The "means for capturing the environment" is a method for capturing an image of the surrounding environment using a night vision camera.

[0791] The "means for processing the image to extract features and update the position information" is a method for extracting important data from the captured image and calculating and updating the current position of the robot.

[0792] "Means for autonomous navigation and mapping" refers to a method that uses optical recognition technology and sensors to determine one's own position, plan a route, and generate a map of the surrounding area.

[0793] "Optical recognition technology" is a technology that uses image processing technology to identify objects and features.

[0794] A "means for detecting soiling" is a method for identifying soiling on a surface using optical recognition technology.

[0795] The "means for performing a cleaning action" refers to an action that the robot takes to remove the detected dirt, such as suction or brushing.

[0796] "Means for transmitting cleaning status to a server in real time" refers to a method for transmitting data on the progress and status of cleaning to a server in real time.

[0797] An "emotion recognition engine" is software or hardware that identifies the emotional state of a user from their voice and facial expressions.

[0798] The "means for recognizing the user's emotions" is a method for determining the user's emotional state using an emotion recognition engine.

[0799] The "means for adjusting the cleaning mode" is a method for changing the operation mode of the cleaning robot based on the emotional state of the user.

[0800] A "learning algorithm" is an artificial intelligence technology that uses past data and patterns to optimize future actions.

[0801] "Means to optimize cleaning activities and improve future cleaning routes" refers to a method that uses a learning algorithm to calculate efficient cleaning patterns and routes.

[0802] "Means for customizing cleaning patterns using user emotional data" refers to a method for designing an optimal cleaning plan for a user based on collected emotional data.

[0803] A "cleaning log" is a record of cleaning activities performed by a robot.

[0804] A "database" is a system for systematically organizing and storing information.

[0805] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and uses optical recognition technology and autonomous navigation to achieve efficient cleaning. The system has the ability to recognize the user's emotions and adjust its behavior based on those emotions.

[0806] Server Roles and Operations

[0807] The server receives data sent from the robot and updates the integrated database. For example, it stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[0808] Specific behavior:

[0809] The server receives the map data sent by each robot and stores it in an integrated database, accumulating information such as the overall image of the room and the cleaning frequency of each area.

[0810] The server analyzes cleaning history and emotion data to derive efficient cleaning patterns and areas for improvement. For example, if it finds that a particular area gets dirty frequently during a certain time period, it will create a cleaning plan to focus on that area.

[0811] Roles and actions of the terminal (robot)

[0812] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera, converts the image to grayscale, and processes it. It uses optical recognition technology to extract features from the image and updates a map of its current location. It uses an autonomous navigation system to calculate and navigate to the next location. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[0813] Specific behavior:

[0814] The device uses a night vision camera to capture and map the environment, for example, accurately detecting furniture placement and obstacles even in the dark and generating a movement map.

[0815] The device performs cleaning at the location where the dirt is detected and reports the cleaning status to the server. For example, it detects a drink spilled on a carpet and soaks it up on the spot.

[0816] The device uses an emotion engine to recognize the user's emotions and, for example, operates quietly if the user is feeling stressed. For example, if the user wants to relax at night, the device will clean in quiet mode with reduced noise.

[0817] User Roles and Actions

[0818] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[0819] Specific behavior:

[0820] Users can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. For example, they can check using the smartphone app to see if a particular room has been cleaned.

[0821] The user receives emotional feedback from the robot and can instruct it to change its operating mode as needed, for example, cleaning in normal mode during the day and switching to quiet mode at night.

[0822] The learning and optimization process

[0823] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[0824] Specific behavior:

[0825] The device sends cleaning activity and emotion data to the server and stores it in a database. For example, after cleaning is completed, the device uploads the cleaning activity and emotion data of the user for that day.

[0826] The server analyzes the data and distributes new cleaning algorithms to the robot, such as calculating and directing efficient cleaning routes for specific times of the day or days of the week.

[0827] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[0828] Examples of prompts for generative AI models

[0829] Example prompt:

[0830] "Please explain how the cleaning robot captures its environment with a night vision camera. Please also explain the basic image processing flow and autonomous navigation."

[0831] Please explain, with specific algorithms and examples, how a cleaning robot can recognize a user's emotions and adjust its behavior accordingly.

[0832] In this way, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[0833] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0834] Step 1: Initialization and Data Collection

[0835] When the system is powered on, the terminal (robot) performs an initialization process. It starts up the night vision camera, sensors, navigation module, and emotion engine, and collects initial data on the working area. Specifically, it uses the night vision camera to capture the surrounding environment, converts the image to grayscale, and processes it. At the same time, it obtains location information and obstacle distance data from various sensors (such as LIDAR and infrared sensors).

[0836] input:

[0837] Powering on the system

[0838] Environmental data from night vision cameras and sensors

[0839] Data processing / data calculation:

[0840] Converting an image to grayscale

[0841] Sensor information acquisition and integration

[0842] output:

[0843] Initialization complete message

[0844] Environmental Datasets

[0845] Specific behavior:

[0846] The device captures an initial image with its night vision camera and records the location of each obstacle.

[0847] Step 2: Data processing and mapping

[0848] The device processes the collected data and creates a map of the room. Specifically, it uses optical recognition technology to extract feature points from grayscale images and generate a map based on each piece of data. An autonomous navigation system then updates the device's position and surrounding terrain in real time.

[0849] input:

[0850] Environmental Datasets

[0851] Grayscale images

[0852] Data processing / data calculation:

[0853] Extracting feature points from images

[0854] Running the map generation algorithm

[0855] output:

[0856] Generated map data

[0857] Current location information

[0858] Specific behavior:

[0859] The device applies edge detection algorithms to identify the location of furniture and walls.

[0860] The terminal generates a map and passes it to the navigation algorithm.

[0861] Step 3: Navigation and Cleaning Plan

[0862] The device uses map data and navigation algorithms to plan the optimal cleaning route, with the navigation system calculating the next location and guiding the robot.

[0863] input:

[0864] Map data

[0865] Current location information

[0866] Data processing / data calculation:

[0867] Execution of the navigation algorithm

[0868] Cleaning route calculation

[0869] output:

[0870] Cleaning route planning

[0871] Next Movement Point

[0872] Specific behavior:

[0873] The device uses algorithms such as the A algorithm to determine an efficient cleaning route while avoiding obstacles.

[0874] The terminal sets the cleaning pattern and indicates the cleaning sequence.

[0875] Step 4: Cleaning execution and real-time monitoring

[0876] The device moves along a planned cleaning route, performs cleaning actions when dirt is detected, and transmits the cleaning status to the server in real time.

[0877] input:

[0878] Cleaning route planning

[0879] Dirt detection data

[0880] Data processing / data calculation:

[0881] Running the dirt detection algorithm

[0882] Cleaning trigger

[0883] output:

[0884] Cleaning progress data

[0885] Real-time upload to server

[0886] Specific behavior:

[0887] The device uses suction or a brush to clean the area where dirt is detected.

[0888] The terminal reports cleaning progress to the server in real time.

[0889] Step 5: Emotion recognition and behavior regulation

[0890] The device uses an emotion engine to recognize the user's emotions and adjusts its behavior based on that information.

[0891] input:

[0892] User voice and facial expression data

[0893] Analysis results from the emotion engine

[0894] Data processing / data calculation:

[0895] Applying speech and facial expression recognition algorithms

[0896] Emotional state estimation

[0897] output:

[0898] Emotion analysis results

[0899] Operation adjustment command

[0900] Specific behavior:

[0901] If the user is in a stressful state, the terminal switches to a silent mode that reduces the operating noise.

[0902] Step 6: Data analysis and feedback

[0903] The server analyzes the cleaning data and emotion data sent from the device, updates the efficient cleaning algorithm, and provides feedback to the device on a new cleaning plan based on the analysis results.

[0904] input:

[0905] Cleaning Data

[0906] Emotional Data

[0907] Data processing / data calculation:

[0908] Executing data analysis algorithms

[0909] Optimized cleaning algorithm

[0910] output:

[0911] Updated cleaning algorithm

[0912] Feedback data to the device

[0913] Specific behavior:

[0914] The server analyzes cleaning frequency and effective cleaning order and sends the results to the terminal.

[0915] Step 7: User monitoring and guidance

[0916] Users can use a smartphone or tablet to monitor the robot's operating status and issue instructions to the robot as needed.

[0917] input:

[0918] Instructions from a smartphone app

[0919] Real-time Monitoring Data

[0920] Data processing / data calculation:

[0921] Analysis of instruction data

[0922] output:

[0923] Commands and instructions for robots

[0924] Situation Report Data

[0925] Specific behavior:

[0926] Users send specific instructions to the robot through the app, such as "prioritize cleaning the living room."

[0927] (Application example 2)

[0928] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0929] Efficient cleaning activities are important in modern factories, but the effectiveness of automated cleaning systems is limited in environments with low visibility at night. Furthermore, while there is a need to adjust operations based on worker emotions, conventional systems lack emotion recognition capabilities and are therefore unable to meet this requirement. Furthermore, it is difficult to integrate cleaning activity data with employee emotion data to provide optimized cleaning patterns.

[0930] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for detecting dirt using optical recognition technology, means for autonomously navigating and creating a map, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for adjusting operations based on the user's emotions with an emotion recognition function, means for optimizing cleaning activities using a learning algorithm and improving subsequent cleaning routes, and means for transmitting cleaning logs and emotion data to the server and storing them in a database. This enables effective cleaning operations even in environments with low visibility at night, realizes operation adjustments based on the user's emotions, and provides optimized cleaning patterns.

[0931] A "night vision camera" is a camera device that can capture clear images of objects even in low-light environments.

[0932] "Capturing" refers to the process of taking in video or images.

[0933] "Feature extraction" means finding important information or patterns in captured video or images.

[0934] "Updating location information" means recording the current location and coordinate data and replacing it with the latest information.

[0935] "Autonomous navigation" means determining the direction of travel and moving independently without external instructions.

[0936] "Creating a map" means recording the surrounding environment and the route you are taking and generating visualized data.

[0937] "Optical recognition technology" is a technology that uses cameras and sensors to determine the shape and characteristics of objects.

[0938] "Detecting dirt" means finding dirt or abnormalities on the surface.

[0939] "Performing a cleaning action" means taking action to remove the detected dirt.

[0940] "Real-time alerting" means being notified immediately of an event or situation that occurs.

[0941] The "emotion recognition function" is a function that determines a person's emotions from video and audio data.

[0942] "Adjusting behavior" means changing the way equipment or systems behave depending on the situation or conditions.

[0943] A "learning algorithm" is a computational method for analyzing accumulated data and discovering new knowledge and patterns.

[0944] "Optimizing cleaning activities" means finding ways to make cleaning tasks more efficient and effective.

[0945] "Improving the cleaning route from next time onwards" means improving the route for the next cleaning work based on past data.

[0946] "Cleaning log and emotional data" refers to a record of cleaning activities and data indicating the user's emotional state.

[0947] "Send to server" means sending data over the network to a central management system.

[0948] "Storing in a database" means organizing and storing the acquired data.

[0949] Server Roles and Operations

[0950] The server captures the environment using a night vision camera and receives data from the device that processes the image and extracts features. This process includes converting the captured image to grayscale and detecting dirt using optical recognition technology. The server also stores navigation data, map data, cleaning history, and emotion data from emotion recognition in an integrated database to support efficient cleaning activities. Furthermore, the server analyzes data sent from each robot and identifies frequently soiled areas to provide an optimal cleaning plan for future cleanings.

[0951] Examples:

[0952] The server stores the map data and cleaning history sent from the factory's nighttime cleaning robots in an integrated database.

[0953] The server analyzes the cleaning logs and emotional data to customize and optimize the robot's cleaning patterns.

[0954] Roles and actions of the terminal (robot)

[0955] The terminal (robot) autonomously moves around the factory to perform cleaning activities. The robot is equipped with a night vision camera, captures the environment, and converts the images to grayscale to detect dirt. It uses an autonomous navigation system to move efficiently while updating a map of its current location. It performs cleaning operations at points where dirt is detected and sends the results to a server in real time after cleaning is completed. It also uses emotion recognition functionality to recognize the emotions of nearby workers and adjust its behavior accordingly. For example, if a worker is feeling stressed, the robot will operate in silent mode.

[0956] Examples:

[0957] The robot uses night vision cameras to capture and map the environment in the factory at night.

[0958] It performs cleaning wherever dirt is detected and reports the results to the server in real time.

[0959] Emotion recognition allows the robot to switch to silent mode if the worker is feeling stressed.

[0960] User Roles and Actions

[0961] Users can remotely monitor the robot's status and issue instructions as needed. Using a smartphone or other device, users can monitor the robot's location, cleaning status, and emotional data in real time. Furthermore, the operating mode can be switched based on the emotional feedback received from the robot. For example, the robot can be set to operate in quiet mode when family members or redundant staff are relaxing. In this way, users can perform optimal cleaning operations that are tailored to the factory environment and the emotions of employees.

[0962] Examples:

[0963] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[0964] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[0965] The learning and optimization process

[0966] The device (robot) sends cleaning activity and emotion recognition data to a server, which analyzes the data. Based on the analysis results, a learning algorithm is used to optimize cleaning activities and improve cleaning routes and patterns for future cleanings. This improves the accuracy and efficiency of cleaning activities, even in environments with low visibility at night. In addition, the pattern of each cleaning activity can be customized based on the user's emotion data, allowing for even more detailed optimization.

[0967] Examples:

[0968] The terminal (robot) transmits cleaning activity and emotion data to the server and stores it in a database.

[0969] The server analyzes the data sent and distributes the cleaning algorithm to the robot for the next cleaning cycle.

[0970] Based on the user's emotional data, it creates a customized cleaning pattern and optimizes cleaning activities.

[0971] Prompt Sentence Examples

[0972] In factories, cleaning robots use night vision cameras to capture the environment and detect dirt. They use emotion recognition to analyze the facial expressions of workers while they work. If the robot detects "emotion: stress," for example, it switches to silent mode. Cleaning status and emotion data are sent to a server in real time, allowing it to learn optimal cleaning patterns.

[0973] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0974] Step 1:

[0975] The device uses a night vision camera to capture the environment. The input is the image captured by the camera, and the output is the video data, which is the basis for capturing detailed information, especially in low-light environments.

[0976] Step 2:

[0977] The device converts the captured video to grayscale and extracts features. This conversion process removes unnecessary color information, making it easier to detect dirt and obstacles. The input is the video data obtained in step 1, and the output is the grayscale image and extracted feature data.

[0978] Step 3:

[0979] The device detects dirt using optical recognition technology. The input is a grayscale image and feature data, and it determines whether dirt is present based on specific patterns and features. The output is the presence or absence of dirt and its location. For example, it performs noise reduction and edge detection to identify areas that appear to be dirty.

[0980] Step 4:

[0981] The device uses an autonomous navigation system to create a map and update its current location. The input is the device's current location information and feature data acquired from the environment, and the output is updated map information and location information. Based on this, the device can calculate where to go next and move efficiently.

[0982] Step 5:

[0983] The device performs cleaning operations at the location where the stain is detected. The input is the location of the stain, and the output is a cleaning completion report. The device's cleaning unit moves to the stained location and performs the specific stain removal task.

[0984] Step 6:

[0985] The device sends a real-time alert after cleaning. The input is the cleaning completion report and its location information, and the output is notification data sent to the server, allowing the server to keep track of the progress of each cleaning activity.

[0986] Step 7:

[0987] The device uses emotion recognition to recognize the user's emotions and adjusts its behavior based on those emotions. The input is camera video data, and the output is the emotion recognition results and the adjusted behavior mode. For example, if the user is feeling stressed, the device can switch to silent mode.

[0988] Step 8:

[0989] The server stores all data (cleaning logs, emotion data, and map information) in an integrated database. The input is the data sent from the device, and the output is the saved database state. The server analyzes this and uses it in an algorithm to optimize future cleaning patterns.

[0990] Step 9:

[0991] The user can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. The input is real-time data sent from the server, and the output is information displayed on the monitoring screen. The user can issue commands to switch the operating mode as needed.

[0992] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0993] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0994] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0995] [Third embodiment]

[0996] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0997] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0998] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0999] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1000] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1001] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1002] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1003] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1004] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1005] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1006] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1007] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1008] The present invention relates to a cleaning robot system using a night vision camera, and specific embodiments thereof will be described below. Details of program processing will be explained with specific examples, divided into the subjects of the server, terminal, and user.

[1009] overview

[1010] The cleaning robot system first captures the environment using a night vision camera, analyzes the footage, and extracts features. It then creates a map using an autonomous navigation system and detects dirt using optical recognition technology. When dirt is detected, the robot performs cleaning operations and sends the results as an alert in real time. It also uses a learning algorithm to optimize cleaning activities, and sends cleaning logs to a server for storage in a database. This ensures efficient and accurate late-night cleaning.

[1011] Server Roles and Operations

[1012] The server receives data sent from the robot and updates the integrated database. Specifically, it stores information such as navigation data, map data, and cleaning history, and uses it as a reference for the robot to perform cleaning activities efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[1013] example:

[1014] The server receives the map data sent from each robot and stores it in an integrated database.

[1015] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[1016] Roles and actions of the terminal (robot)

[1017] The terminal (robot) operates autonomously and performs cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins processing. Next, optical recognition technology is used to extract features from the image and update the map of the current location. After that, it uses an autonomous navigation system to calculate the next position to go to, and if dirt is detected, it performs cleaning operations.

[1018] example:

[1019] The device uses a night vision camera to capture and map the environment at night.

[1020] The device will start cleaning when it detects dirt in a specific location.

[1021] User Roles and Actions

[1022] Users can remotely monitor the robot's status and issue instructions as needed. Users receive real-time alerts to see the cleaning robot's activity and can also give instructions to re-clean specific designated areas.

[1023] example:

[1024] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[1025] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[1026] The learning and optimization process

[1027] The robot learns using cleaning activity data and map data, optimizing future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[1028] example:

[1029] The device sends data on cleaning activities to a server, which stores it in a database.

[1030] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[1031] The processing flow will be explained below.

[1032] Step 1:

[1033] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[1034] Step 2:

[1035] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale to facilitate further processing.

[1036] Step 3:

[1037] The device extracts features from the captured video and updates a map of the device's current location based on the extracted features, accurately determining the device's location.

[1038] Step 4:

[1039] The device uses an autonomous navigation system to calculate and navigate its next location, allowing the device to efficiently cover the entire area.

[1040] Step 5:

[1041] The device detects dirt using optical recognition technology, capturing the environment with a night vision camera and analyzing the footage to identify dirt and foreign objects.

[1042] Step 6:

[1043] The device will perform cleaning operations at the location where the dirt is detected, and once the cleaning operation is complete, the device will send the results to the server in real time and also send an alert that the cleaning is complete.

[1044] Step 7:

[1045] The server stores the cleaning log received from the device in a database, analyzes the cleaning history and map data, derives efficient cleaning patterns, updates the learning algorithm, and provides feedback to the device on an optimized navigation system.

[1046] Step 8:

[1047] Users remotely monitor the device's location and cleaning status in real time, and can send instructions to the device to re-clean specific areas as needed.

[1048] Step 9:

[1049] The device receives updated information from the server and instructions from the user to continue cleaning, and then initiates the next cleaning activity accordingly. Through this cyclical process, the device continuously improves its cleaning efficiency and contributes to environmental conservation.

[1050] Example 1

[1051] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1052] Current cleaning robot systems have difficulty performing efficient and accurate cleaning at night, and there are many challenges with dirt detection and map creation, especially in environments that use night vision. This reduces cleaning accuracy and efficiency, and increases time and costs. Furthermore, few systems utilize cleaning activity logs to further improve efficiency and optimization. To solve these problems, a more accurate and efficient cleaning robot system is needed.

[1053] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1054] In this invention, the server includes means for capturing the environment using a night vision camera, means for converting the captured image to grayscale to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, and means for sending cleaning data to the server and storing it in an integrated database. This makes it possible to detect dirt with high accuracy even in nighttime environments, perform efficient cleaning, and optimize future cleanings by utilizing cleaning activity log data.

[1055] A "night vision camera" is a special imaging device that can capture images even in darkness or low-light environments.

[1056] The "means for capturing the environment" is a method for capturing an image of the surroundings using a night vision camera.

[1057] "Means for converting to grayscale" refers to a method for converting the acquired image into a format that is expressed in shades of black and white.

[1058] The "means for extracting features and updating position information" is a method for detecting specific patterns or objects from the image converted to grayscale and using that information to update the robot's position.

[1059] "Means for autonomous navigation and map creation" refers to a method in which a robot determines its own route and generates a map based on information about its surroundings.

[1060] "Optical recognition technology" is a technology that analyzes video data acquired by a camera and recognizes objects and features.

[1061] The "means for detecting dirt" is a method for identifying dirt from captured images using optical recognition technology.

[1062] "Means for performing cleaning actions" refers to a series of actions that the robot takes to remove the detected soiling.

[1063] "Means for sending real-time alerts" refers to a method for sending immediate notification to users or systems after cleaning is completed.

[1064] "Means for transmitting cleaning data to a server and storing it in an integrated database" refers to a method for transmitting records of cleaning activities to a server via the Internet or the like and storing them there.

[1065] The present invention relates to an autonomous cleaning robot system using a night vision camera, and specific embodiments thereof will be described below.

[1066] Server Roles and Operations

[1067] The server receives data sent from the robot (navigation data, map data, cleaning history, etc.) and stores it in an integrated database. The server analyzes the information stored in the database and identifies areas that are frequently soiled. This allows for the creation of an efficient and effective cleaning plan. The server also analyzes the cleaning history, generates new cleaning algorithms, and distributes them to the terminal (robot).

[1068] Examples:

[1069] The server receives the map data sent from each robot and stores it in an integrated database.

[1070] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[1071] Roles and actions of the terminal (robot)

[1072] The terminal (robot) operates autonomously to carry out cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins image processing. Optical recognition technology is used to extract features from the grayscale image and update the robot's current location on a map. Next, an autonomous navigation system is used to calculate an appropriate route, and if dirt is detected, it immediately begins cleaning.

[1073] Examples:

[1074] The device uses a night vision camera to capture and map the environment at night.

[1075] The device will start cleaning when it detects dirt in a specific location.

[1076] User Roles and Actions

[1077] Users can remotely monitor the robot's status. They can check the robot's location and cleaning status in real time using a smartphone or other device. If necessary, users can also issue instructions to re-clean a specific area. If an alert occurs, they can quickly check the details and take appropriate action.

[1078] Examples:

[1079] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[1080] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[1081] The learning and optimization process

[1082] The device (robot) learns from cleaning activity data and map data and optimizes future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[1083] Examples:

[1084] The device sends data on cleaning activities to a server, which stores it in a database.

[1085] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[1086] Example prompt for a generative AI model:

[1087] "Please explain how to save map data of a cleaning robot acquired at night by a night vision camera to a server."

[1088] "Explain how you can use a night vision camera to capture video, convert it to grayscale, detect dirt, and update the map."

[1089] "Explain how you can monitor the status of your cleaning robot on your smartphone and receive alerts to re-clean specific areas."

[1090] "Explain how data collected from a cleaning robot can be used to optimize the next cleaning session."

[1091] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1092] Step 1:

[1093] The terminal (robot) activates the night vision camera and starts capturing the environment. The night vision camera acquires images of the environment and sends the data to the computer vision system. The input is the raw image captured by the night vision camera. The output is the image data itself.

[1094] Specific behavior:

[1095] The device switches to night setting mode.

[1096] Activate the night vision camera and capture video continuously.

[1097] Step 2:

[1098] The device converts the captured video to grayscale. The input is color video data acquired from a night vision camera, and the output is grayscale video data. This conversion makes the video data easier to analyze.

[1099] Specific behavior:

[1100] The terminal activates an image processing module to convert the color image into grayscale format.

[1101] The converted grayscale image is stored in a temporary memory.

[1102] Step 3:

[1103] The device uses optical recognition technology to extract features from the grayscale image and update the location information. The input is the grayscale image data obtained in step 2, and the output is feature data and updated location information. The feature data indicates important parts of the image, such as dirt or obstacles.

[1104] Specific behavior:

[1105] The device runs a feature extraction algorithm to recognize dirt and obstacles from the video data.

[1106] The current location information is updated based on the extracted features.

[1107] Step 4:

[1108] The device updates the map using the autonomous navigation system. The input is the feature data and location information obtained in step 3, and the output is updated map data. This map is modified by adding new data based on the known range.

[1109] Specific behavior:

[1110] The device runs a navigation algorithm and updates the map to reflect the current location and feature data.

[1111] The updated map is stored in memory and prepared for the next action.

[1112] Step 5:

[1113] The device performs cleaning operation when dirt is detected. The input is the map data updated in step 4 and the location information of the dirt, and the output is the cleaning completion data.

[1114] Specific behavior:

[1115] An autonomous navigation system is used to navigate to the location where the contamination is detected.

[1116] Activates the cleaning module and cleans the designated area.

[1117] Step 6:

[1118] After cleaning, the terminal sends a real-time alert to the server. The input is the cleaning completion data, and the output is the alert notification to the server.

[1119] Specific behavior:

[1120] The terminal activates the communication module and transmits cleaning completion data to the server.

[1121] An alert is sent to the server and the user is notified.

[1122] Step 7:

[1123] The server receives the cleaning data sent from the terminal and stores it in an integrated database. The input is the cleaning data sent from the terminal, and the output is the history data stored in the database.

[1124] Specific behavior:

[1125] The server starts the communication module and receives data from the terminal.

[1126] The received data is analyzed and stored in an integrated database.

[1127] Step 8:

[1128] The server analyzes the received cleaning data and generates a new cleaning algorithm. The input is the stored cleaning history data, and the output is the updated cleaning algorithm.

[1129] Specific behavior:

[1130] The server launches a data analysis module and calculates an efficient cleaning pattern based on past cleaning history.

[1131] Generate new algorithms and prepare them for distribution to the robots.

[1132] Step 9:

[1133] The user remotely monitors the robot's status and issues instructions for re-cleaning as necessary. The input is alert notifications from the server and the robot's location information, and the output is instructions for re-cleaning.

[1134] Specific behavior:

[1135] The user launches the smartphone app and checks the robot's status.

[1136] If necessary, instructions to re-clean specific areas will be sent via the app.

[1137] (Application example 1)

[1138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1139] In large facilities such as logistics centers, efficient cleaning is difficult in environments with poor visibility at night. Furthermore, to efficiently clean large areas, it is necessary to manage the robot's real-time location information, detect dirt, and respond immediately. However, current cleaning systems lack the means to effectively solve these issues. There is also a need for a method to optimize the efficiency of cleaning activities and improve future cleaning routes based on past cleaning data.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1141] In this invention, the server includes: means for capturing the environment using a night vision camera; means for processing the captured image to extract features and update location information; means for autonomously navigating and creating maps; means for detecting dirt using optical recognition technology; means for performing cleaning operations to clean the detected dirt; means for sending real-time alerts after cleaning; means including a smartphone application for displaying the robot's real-time location information; means for viewing the environmental image captured by the night vision camera; means for issuing alerts when dirt is detected; means for remotely operating the robot; and means for displaying cleaning history and checking learning data. This enables efficient and accurate cleaning even in environments with poor visibility, even at night, and enables real-time status monitoring and immediate response. Furthermore, it enables optimization of cleaning activities and improvement of cleaning routes based on past data.

[1142] A "night vision camera" is a camera used to ensure visibility in the dark, and is a device that uses infrared light to capture the environment.

[1143] The "means for capturing the environment" is a mechanism that has the function of acquiring the surrounding situation as video data using a night vision camera.

[1144] The "means for extracting features and updating location information" is a technology that detects specific patterns or objects from captured video data and updates the current location information based on that.

[1145] "Means for autonomous navigation and map creation" refers to a system that automatically moves based on acquired location information, aggregates information from each location, and generates a map.

[1146] "Optical recognition technology" is a technology that uses optical devices such as cameras to detect specific objects or features from image data.

[1147] The "means for detecting dirt" is a system that uses optical recognition technology to identify dirt from video data and pinpoint its location.

[1148] "Means for performing cleaning actions" refers to a series of operations that the robot performs to remove the detected dirt.

[1149] "Means for sending alerts in real time" refers to a system that has a communication function to immediately notify users of detected dirt and the cleaning status after cleaning.

[1150] The "smartphone application for displaying real-time location information of a robot" is an application that displays the current location of a cleaning robot on a smartphone in real time.

[1151] "Means for viewing environmental images captured by a night vision camera" refers to a mechanism for displaying images captured by a night vision camera on a smartphone or other device.

[1152] "Means for issuing an alert notification when dirt is detected" refers to a mechanism that has the function of sending an immediate notification when dirt is confirmed.

[1153] "Means for remotely operating a robot" refers to remote control technology for remotely controlling the operation of a cleaning robot.

[1154] "Means for displaying cleaning history and checking learning data" refers to a mechanism that displays the history of past cleaning activities and learns from the acquired data to improve the cleaning algorithm.

[1155] This section describes a specific embodiment of a cleaning robot system using a night vision camera. This invention utilizes an autonomously operating cleaning robot to achieve efficient cleaning, particularly in large-scale facilities such as logistics centers.

[1156] System Configuration

[1157] The cleaning robot system of the present invention consists of the following main components:

[1158] 1. Autonomous mobile robot equipped with a night vision camera

[1159] 2. Smartphone Applications

[1160] 3. Server

[1161] 4. Communications Infrastructure

[1162] Server Roles

[1163] The server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning routes, and means for sending a cleaning log to the server and storing it in a database.

[1164] Specifically, the server processes the map data and cleaning history sent from the robot, saving them in an integrated database, and analyzing this data to calculate efficient cleaning patterns.The main software used is PostgreSQL for database management and Python libraries for analysis.

[1165] The role of the terminal (robot)

[1166] The terminal (robot) captures the environment with a night vision camera, converts the image to grayscale, and detects dirt using optical recognition technology. It then performs cleaning operations on the detected dirt and updates the map. The hardware used is a night vision camera and an autonomous mobile robot, and the software uses ROS (Robot Operating System). This enables autonomous navigation and cleaning.

[1167] User Roles

[1168] Users can monitor the robot's status through a smartphone application and remotely control it as needed. The application displays the robot's real-time location information, allows viewing of environmental images from a night vision camera, sends alerts when dirt is detected, displays cleaning history, and checks learning data. The application was developed using Android Studio.

[1169] A specific example is the use of a smartphone app to check the cleaning status of warehouse area A in real time. For example, when the robot detects dirt and begins cleaning, a notification is sent to the user, who can then view footage of the site on their smartphone. If necessary, the user can remotely instruct the robot to re-clean a specific area.

[1170] Prompt Sentence Examples

[1171] Enter the following prompt into the generative AI model to generate an alert message:

[1172] "A cleaning robot using a night vision camera in a distribution center has detected dirt in a specific area. Cleaning is required as an emergency response. Please generate a message for an alert notification."

[1173] Example: "Urgent Alert: Soil detected in Warehouse Area A of the Distribution Center. Immediate management attention required."

[1174] Thus, the present invention is a cleaning robot system using a night vision camera, and provides a specific method for effectively cleaning environments such as logistics centers.

[1175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1176] Program processing flow and detailed explanations for each step

[1177] Step 1: Capture your environment

[1178] Subject: Terminal

[1179] The terminal (robot) captures the environment using a night vision camera. The night vision camera acquires video data of the surroundings and converts the video data into grayscale. This allows for highly accurate video data to be obtained even in dark places. The acquired video data is also temporarily stored in local storage.

[1180] Input: Surrounding video data

[1181] Output: Grayscale converted video data

[1182] Step 2: Feature extraction and location update

[1183] Subject: Terminal

[1184] The video data converted to grayscale is analyzed and features are extracted using optical recognition technology. Image analysis algorithms (such as the OpenCV library) are used for feature extraction. The current location information is updated based on the feature information obtained from this analysis. This allows the robot's position to be accurately displayed on a map.

[1185] Input: Grayscale converted video data

[1186] Output: extracted feature information, updated location information

[1187] Step 3: Create a map

[1188] Subject: Terminal

[1189] Based on the extracted feature information and location information, the system performs autonomous navigation and creates a map of the surrounding area using the SLAM (Simultaneous Localization and Mapping) algorithm. This map data is temporarily stored in the device's local memory.

[1190] Input: extracted feature information, updated location information

[1191] Output: Created map data

[1192] Step 4: Detecting Contamination

[1193] Subject: Terminal

[1194] The device detects dirt from map data created using optical recognition technology, a process that uses algorithms to identify specific colors and shapes, and when a dirt is detected, its location is recorded in a log.

[1195] Input: Created map data

[1196] Output: Specific location information of stains

[1197] Step 5: Perform cleaning actions

[1198] Subject: Terminal

[1199] When dirt is detected, the device initiates a cleaning operation, the robot moves to a specific location, performs the cleaning function, and when the cleaning is complete, the information is recorded in a log.

[1200] Input: Specific location of stain

[1201] Output: Cleaning completion log

[1202] Step 6: Sending an alert

[1203] Subject: Terminal

[1204] Once the cleaning is complete, the device sends a real-time alert that includes information about the cleaning completion and the specific location of the cleaning. The alert is sent to the smartphone application and server.

[1205] Input: Cleanup completion log

[1206] Output: Alert notification

[1207] Step 7: Save your data

[1208] Subject: Server

[1209] The server receives the map data and cleaning logs sent from the device and stores them in an integrated database, which centrally manages past cleaning history and map information.

[1210] Input: Map data, cleaning log

[1211] Output: Data stored in the integrated database

[1212] Step 8: Learn and optimize

[1213] Subject: Server

[1214] The server analyzes the stored data and uses a learning algorithm to optimize cleaning activities. This process includes identifying frequently soiled areas and calculating efficient cleaning patterns. The analysis results are reflected in the cleaning algorithm and used for future cleaning activities.

[1215] Input: Data from the integrated database

[1216] Output: Optimized cleaning algorithm

[1217] In this way, the present invention realizes efficient cleaning in large-scale facilities such as logistics centers through specific processing steps of a cleaning robot system using a night vision camera.

[1218] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1219] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and achieves efficient cleaning through optical recognition technology and autonomous navigation. This system has the ability to recognize the user's emotions and adjust its behavior based on those emotions. Below, we will explain the details of the program processing, with concrete examples, divided into the subjects of the server, terminal, and user.

[1220] Server Roles and Operations

[1221] The server receives data sent from the robot and updates the integrated database. It stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[1222] example:

[1223] The server receives the map data sent from each robot and stores it in an integrated database.

[1224] The server analyzes cleaning history and emotional data to derive efficient cleaning patterns and areas for improvement.

[1225] Roles and actions of the terminal (robot)

[1226] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera and converts the image to grayscale for processing. It extracts features from the image using optical recognition technology and updates a map of the current location. It uses an autonomous navigation system to calculate the next location to go to and navigate. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[1227] example:

[1228] The device uses a night vision camera to capture and map the environment.

[1229] Cleaning is performed at the location where dirt is detected, and the cleaning status is reported to the server.

[1230] The terminal recognizes the user's emotions using an emotion engine, and operates quietly, for example, when the user is feeling stressed.

[1231] User Roles and Actions

[1232] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[1233] example:

[1234] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[1235] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[1236] The learning and optimization process

[1237] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[1238] example:

[1239] The device transmits cleaning activity and emotion data to a server and stores it in a database.

[1240] The server analyzes the data and distributes new cleaning algorithms to the robots.

[1241] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[1242] As described above, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[1243] The processing flow will be explained below.

[1244] Step 1:

[1245] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[1246] Step 2:

[1247] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale for easier further processing.

[1248] Step 3:

[1249] The device extracts features from the captured video, including the patterns on walls, furniture, and floors, and uses these features to create a map and update its current location.

[1250] Step 4:

[1251] The device uses an autonomous navigation system to calculate the next location to go to and navigate, moving while efficiently covering the designated area.

[1252] Step 5:

[1253] The device uses optical recognition technology to detect dirt and debris on the floor, analyzing images from a night vision camera and recognizing distinctive dirt patterns.

[1254] Step 6:

[1255] The device performs cleaning operations at the location where dirt is detected, including sweeping, vacuuming, wiping with water, etc. Once the cleaning is complete, the results are recorded.

[1256] Step 7:

[1257] The device sends information about the location where cleaning was completed and the cleaning results to the server in real time. The server receives this information and stores it in a database. The device also sends an alert to the user when cleaning is complete.

[1258] Step 8:

[1259] Users can check the robot's location and cleaning status in real time on their smartphone or PC, and can also instruct it to clean a specific area again.

[1260] Step 9:

[1261] The device uses an emotion engine to recognize the user's emotions. For this purpose, the device is equipped with a camera and microphone, and analyzes emotions from the user's facial expressions and voice.

[1262] Step 10:

[1263] The device adjusts its cleaning behavior based on the user's emotions, for example, if the user is feeling stressed, the device will operate in quiet mode to maintain a relaxing environment for the user.

[1264] Step 11:

[1265] The device sends cleaning activity and emotion data to the server, which stores it in a database. A learning algorithm optimizes future cleaning activities.

[1266] Step 12:

[1267] The server analyzes the data, updates the cleaning algorithm and emotion-based cleaning patterns, and feeds them back to the device, continuously improving cleaning accuracy and user satisfaction.

[1268] Example 2

[1269] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1270] Conventional cleaning robot systems were able to accurately recognize their environment and autonomously clean efficiently, but they were unable to adjust their behavior based on the user's emotions, making it difficult to clean in a way that took the user's comfort into consideration. Furthermore, they lacked real-time cleaning status reports and adequate optimization of cleaning activities, creating challenges in improving cleaning efficiency.

[1271] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting the cleaning status to the server in real time, a means for recognizing the user's emotion using an emotion recognition engine, and a means for adjusting the cleaning mode based on the user's emotion. This enables the terminal to operate in accordance with the user's emotion, providing comfortable cleaning and monitoring the cleaning status in real time, thereby realizing efficient cleaning activities.

[1272] A "night vision camera" is a camera that can capture clear images even in dark environments.

[1273] The "means for capturing the environment" is a method for capturing an image of the surrounding environment using a night vision camera.

[1274] The "means for processing the image to extract features and update the position information" is a method for extracting important data from the captured image and calculating and updating the current position of the robot.

[1275] "Means for autonomous navigation and mapping" refers to a method that uses optical recognition technology and sensors to determine one's own position, plan a route, and generate a map of the surrounding area.

[1276] "Optical recognition technology" is a technology that uses image processing technology to identify objects and features.

[1277] A "means for detecting soiling" is a method for identifying soiling on a surface using optical recognition technology.

[1278] The "means for performing a cleaning action" refers to an action that the robot takes to remove the detected dirt, such as suction or brushing.

[1279] "Means for transmitting cleaning status to a server in real time" refers to a method for transmitting data on the progress and status of cleaning to a server in real time.

[1280] An "emotion recognition engine" is software or hardware that identifies the emotional state of a user from their voice and facial expressions.

[1281] The "means for recognizing the user's emotions" is a method for determining the user's emotional state using an emotion recognition engine.

[1282] The "means for adjusting the cleaning mode" is a method for changing the operation mode of the cleaning robot based on the emotional state of the user.

[1283] A "learning algorithm" is an artificial intelligence technology that uses past data and patterns to optimize future actions.

[1284] "Means to optimize cleaning activities and improve future cleaning routes" refers to a method that uses a learning algorithm to calculate efficient cleaning patterns and routes.

[1285] "Means for customizing cleaning patterns using user emotional data" refers to a method for designing an optimal cleaning plan for a user based on collected emotional data.

[1286] A "cleaning log" is a record of cleaning activities performed by a robot.

[1287] A "database" is a system for systematically organizing and storing information.

[1288] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and uses optical recognition technology and autonomous navigation to achieve efficient cleaning. The system has the ability to recognize the user's emotions and adjust its behavior based on those emotions.

[1289] Server Roles and Operations

[1290] The server receives data sent from the robot and updates the integrated database. For example, it stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[1291] Specific behavior:

[1292] The server receives the map data sent by each robot and stores it in an integrated database, accumulating information such as the overall image of the room and the cleaning frequency of each area.

[1293] The server analyzes cleaning history and emotion data to derive efficient cleaning patterns and areas for improvement. For example, if it finds that a particular area gets dirty frequently during a certain time period, it will create a cleaning plan to focus on that area.

[1294] Roles and actions of the terminal (robot)

[1295] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera, converts the image to grayscale, and processes it. It uses optical recognition technology to extract features from the image and updates a map of its current location. It uses an autonomous navigation system to calculate and navigate to the next location. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[1296] Specific behavior:

[1297] The device uses a night vision camera to capture and map the environment, for example, accurately detecting furniture placement and obstacles even in the dark and generating a movement map.

[1298] The device performs cleaning at the location where the dirt is detected and reports the cleaning status to the server. For example, it detects a drink spilled on a carpet and soaks it up on the spot.

[1299] The device uses an emotion engine to recognize the user's emotions and, for example, operates quietly if the user is feeling stressed. For example, if the user wants to relax at night, the device will clean in quiet mode with reduced noise.

[1300] User Roles and Actions

[1301] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[1302] Specific behavior:

[1303] Users can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. For example, they can check using the smartphone app to see if a particular room has been cleaned.

[1304] The user receives emotional feedback from the robot and can instruct it to change its operating mode as needed, for example, cleaning in normal mode during the day and switching to quiet mode at night.

[1305] The learning and optimization process

[1306] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[1307] Specific behavior:

[1308] The device sends cleaning activity and emotion data to the server and stores it in a database. For example, after cleaning is completed, the device uploads the cleaning activity and emotion data of the user for that day.

[1309] The server analyzes the data and distributes new cleaning algorithms to the robot, such as calculating and directing efficient cleaning routes for specific times of the day or days of the week.

[1310] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[1311] Examples of prompts for generative AI models

[1312] Example prompt:

[1313] "Please explain how the cleaning robot captures its environment with a night vision camera. Please also explain the basic image processing flow and autonomous navigation."

[1314] Please explain, with specific algorithms and examples, how a cleaning robot can recognize a user's emotions and adjust its behavior accordingly.

[1315] In this way, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[1316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1317] Step 1: Initialization and Data Collection

[1318] When the system is powered on, the terminal (robot) performs an initialization process. It starts up the night vision camera, sensors, navigation module, and emotion engine, and collects initial data on the working area. Specifically, it uses the night vision camera to capture the surrounding environment, converts the image to grayscale, and processes it. At the same time, it obtains location information and obstacle distance data from various sensors (such as LIDAR and infrared sensors).

[1319] input:

[1320] Powering on the system

[1321] Environmental data from night vision cameras and sensors

[1322] Data processing / data calculation:

[1323] Converting an image to grayscale

[1324] Sensor information acquisition and integration

[1325] output:

[1326] Initialization complete message

[1327] Environmental Datasets

[1328] Specific behavior:

[1329] The device captures an initial image with its night vision camera and records the location of each obstacle.

[1330] Step 2: Data processing and mapping

[1331] The device processes the collected data and creates a map of the room. Specifically, it uses optical recognition technology to extract feature points from grayscale images and generate a map based on each piece of data. An autonomous navigation system then updates the device's position and surrounding terrain in real time.

[1332] input:

[1333] Environmental Datasets

[1334] Grayscale images

[1335] Data processing / data calculation:

[1336] Extracting feature points from images

[1337] Running the map generation algorithm

[1338] output:

[1339] Generated map data

[1340] Current location information

[1341] Specific behavior:

[1342] The device applies edge detection algorithms to identify the location of furniture and walls.

[1343] The terminal generates a map and passes it to the navigation algorithm.

[1344] Step 3: Navigation and Cleaning Plan

[1345] The device uses map data and navigation algorithms to plan the optimal cleaning route, with the navigation system calculating the next location and guiding the robot.

[1346] input:

[1347] Map data

[1348] Current location information

[1349] Data processing / data calculation:

[1350] Execution of the navigation algorithm

[1351] Cleaning route calculation

[1352] output:

[1353] Cleaning route planning

[1354] Next Movement Point

[1355] Specific behavior:

[1356] The device uses algorithms such as the A algorithm to determine an efficient cleaning route while avoiding obstacles.

[1357] The terminal sets the cleaning pattern and indicates the cleaning sequence.

[1358] Step 4: Cleaning execution and real-time monitoring

[1359] The device moves along a planned cleaning route, performs cleaning actions when dirt is detected, and transmits the cleaning status to the server in real time.

[1360] input:

[1361] Cleaning route planning

[1362] Dirt detection data

[1363] Data processing / data calculation:

[1364] Running the dirt detection algorithm

[1365] Cleaning trigger

[1366] output:

[1367] Cleaning progress data

[1368] Real-time upload to server

[1369] Specific behavior:

[1370] The device uses suction or a brush to clean the area where dirt is detected.

[1371] The terminal reports cleaning progress to the server in real time.

[1372] Step 5: Emotion recognition and behavior regulation

[1373] The device uses an emotion engine to recognize the user's emotions and adjusts its behavior based on that information.

[1374] input:

[1375] User voice and facial expression data

[1376] Analysis results from the emotion engine

[1377] Data processing / data calculation:

[1378] Applying speech and facial expression recognition algorithms

[1379] Emotional state estimation

[1380] output:

[1381] Emotion analysis results

[1382] Operation adjustment command

[1383] Specific behavior:

[1384] If the user is in a stressful state, the terminal switches to a silent mode that reduces the operating noise.

[1385] Step 6: Data analysis and feedback

[1386] The server analyzes the cleaning data and emotion data sent from the device, updates the efficient cleaning algorithm, and provides feedback to the device on a new cleaning plan based on the analysis results.

[1387] input:

[1388] Cleaning Data

[1389] Emotional Data

[1390] Data processing / data calculation:

[1391] Executing data analysis algorithms

[1392] Optimized cleaning algorithm

[1393] output:

[1394] Updated cleaning algorithm

[1395] Feedback data to the device

[1396] Specific behavior:

[1397] The server analyzes cleaning frequency and effective cleaning order and sends the results to the terminal.

[1398] Step 7: User monitoring and guidance

[1399] Users can use a smartphone or tablet to monitor the robot's operating status and issue instructions to the robot as needed.

[1400] input:

[1401] Instructions from a smartphone app

[1402] Real-time Monitoring Data

[1403] Data processing / data calculation:

[1404] Analysis of instruction data

[1405] output:

[1406] Commands and instructions for robots

[1407] Situation Report Data

[1408] Specific behavior:

[1409] Users send specific instructions to the robot through the app, such as "prioritize cleaning the living room."

[1410] (Application example 2)

[1411] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1412] Efficient cleaning activities are important in modern factories, but the effectiveness of automated cleaning systems is limited in environments with low visibility at night. Furthermore, while there is a need to adjust operations based on worker emotions, conventional systems lack emotion recognition capabilities and are therefore unable to meet this requirement. Furthermore, it is difficult to integrate cleaning activity data with employee emotion data to provide optimized cleaning patterns.

[1413] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for detecting dirt using optical recognition technology, means for autonomously navigating and creating a map, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for adjusting operations based on the user's emotions with an emotion recognition function, means for optimizing cleaning activities using a learning algorithm and improving subsequent cleaning routes, and means for transmitting cleaning logs and emotion data to the server and storing them in a database. This enables effective cleaning operations even in environments with low visibility at night, realizes operation adjustments based on the user's emotions, and provides optimized cleaning patterns.

[1414] A "night vision camera" is a camera device that can capture clear images of objects even in low-light environments.

[1415] "Capturing" refers to the process of taking in video or images.

[1416] "Feature extraction" means finding important information or patterns in captured video or images.

[1417] "Updating location information" means recording the current location and coordinate data and replacing it with the latest information.

[1418] "Autonomous navigation" means determining the direction of travel and moving independently without external instructions.

[1419] "Creating a map" means recording the surrounding environment and the route you are taking and generating visualized data.

[1420] "Optical recognition technology" is a technology that uses cameras and sensors to determine the shape and characteristics of objects.

[1421] "Detecting dirt" means finding dirt or abnormalities on the surface.

[1422] "Performing a cleaning action" means taking action to remove the detected dirt.

[1423] "Real-time alerting" means being notified immediately of an event or situation that occurs.

[1424] The "emotion recognition function" is a function that determines a person's emotions from video and audio data.

[1425] "Adjusting behavior" means changing the way equipment or systems behave depending on the situation or conditions.

[1426] A "learning algorithm" is a computational method for analyzing accumulated data and discovering new knowledge and patterns.

[1427] "Optimizing cleaning activities" means finding ways to make cleaning tasks more efficient and effective.

[1428] "Improving the cleaning route from next time onwards" means improving the route for the next cleaning work based on past data.

[1429] "Cleaning log and emotional data" refers to a record of cleaning activities and data indicating the user's emotional state.

[1430] "Send to server" means sending data over the network to a central management system.

[1431] "Storing in a database" means organizing and storing the acquired data.

[1432] Server Roles and Operations

[1433] The server captures the environment using a night vision camera and receives data from the device that processes the image and extracts features. This process includes converting the captured image to grayscale and detecting dirt using optical recognition technology. The server also stores navigation data, map data, cleaning history, and emotion data from emotion recognition in an integrated database to support efficient cleaning activities. Furthermore, the server analyzes data sent from each robot and identifies frequently soiled areas to provide an optimal cleaning plan for future cleanings.

[1434] Examples:

[1435] The server stores the map data and cleaning history sent from the factory's nighttime cleaning robots in an integrated database.

[1436] The server analyzes the cleaning logs and emotional data to customize and optimize the robot's cleaning patterns.

[1437] Roles and actions of the terminal (robot)

[1438] The terminal (robot) autonomously moves around the factory to perform cleaning activities. The robot is equipped with a night vision camera, captures the environment, and converts the images to grayscale to detect dirt. It uses an autonomous navigation system to move efficiently while updating a map of its current location. It performs cleaning operations at points where dirt is detected and sends the results to a server in real time after cleaning is completed. It also uses emotion recognition functionality to recognize the emotions of nearby workers and adjust its behavior accordingly. For example, if a worker is feeling stressed, the robot will operate in silent mode.

[1439] Examples:

[1440] The robot uses night vision cameras to capture and map the environment in the factory at night.

[1441] It performs cleaning wherever dirt is detected and reports the results to the server in real time.

[1442] Emotion recognition allows the robot to switch to silent mode if the worker is feeling stressed.

[1443] User Roles and Actions

[1444] Users can remotely monitor the robot's status and issue instructions as needed. Using a smartphone or other device, users can monitor the robot's location, cleaning status, and emotional data in real time. Furthermore, the operating mode can be switched based on the emotional feedback received from the robot. For example, the robot can be set to operate in quiet mode when family members or redundant staff are relaxing. In this way, users can perform optimal cleaning operations that are tailored to the factory environment and the emotions of employees.

[1445] Examples:

[1446] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[1447] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[1448] The learning and optimization process

[1449] The device (robot) sends cleaning activity and emotion recognition data to a server, which analyzes the data. Based on the analysis results, a learning algorithm is used to optimize cleaning activities and improve cleaning routes and patterns for future cleanings. This improves the accuracy and efficiency of cleaning activities, even in environments with low visibility at night. In addition, the pattern of each cleaning activity can be customized based on the user's emotion data, allowing for even more detailed optimization.

[1450] Examples:

[1451] The terminal (robot) transmits cleaning activity and emotion data to the server and stores it in a database.

[1452] The server analyzes the data sent and distributes the cleaning algorithm to the robot for the next cleaning cycle.

[1453] Based on the user's emotional data, it creates a customized cleaning pattern and optimizes cleaning activities.

[1454] Prompt Sentence Examples

[1455] In factories, cleaning robots use night vision cameras to capture the environment and detect dirt. They use emotion recognition to analyze the facial expressions of workers while they work. If the robot detects "emotion: stress," for example, it switches to silent mode. Cleaning status and emotion data are sent to a server in real time, allowing it to learn optimal cleaning patterns.

[1456] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1457] Step 1:

[1458] The device uses a night vision camera to capture the environment. The input is the image captured by the camera, and the output is the video data, which is the basis for capturing detailed information, especially in low-light environments.

[1459] Step 2:

[1460] The device converts the captured video to grayscale and extracts features. This conversion process removes unnecessary color information, making it easier to detect dirt and obstacles. The input is the video data obtained in step 1, and the output is the grayscale image and extracted feature data.

[1461] Step 3:

[1462] The device detects dirt using optical recognition technology. The input is a grayscale image and feature data, and it determines whether dirt is present based on specific patterns and features. The output is the presence or absence of dirt and its location. For example, it performs noise reduction and edge detection to identify areas that appear to be dirty.

[1463] Step 4:

[1464] The device uses an autonomous navigation system to create a map and update its current location. The input is the device's current location information and feature data acquired from the environment, and the output is updated map information and location information. Based on this, the device can calculate where to go next and move efficiently.

[1465] Step 5:

[1466] The device performs cleaning operations at the location where the stain is detected. The input is the location of the stain, and the output is a cleaning completion report. The device's cleaning unit moves to the stained location and performs the specific stain removal task.

[1467] Step 6:

[1468] The device sends a real-time alert after cleaning. The input is the cleaning completion report and its location information, and the output is notification data sent to the server, allowing the server to keep track of the progress of each cleaning activity.

[1469] Step 7:

[1470] The device uses emotion recognition to recognize the user's emotions and adjusts its behavior based on those emotions. The input is camera video data, and the output is the emotion recognition results and the adjusted behavior mode. For example, if the user is feeling stressed, the device can switch to silent mode.

[1471] Step 8:

[1472] The server stores all data (cleaning logs, emotion data, and map information) in an integrated database. The input is the data sent from the device, and the output is the saved database state. The server analyzes this and uses it in an algorithm to optimize future cleaning patterns.

[1473] Step 9:

[1474] The user can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. The input is real-time data sent from the server, and the output is information displayed on the monitoring screen. The user can issue commands to switch the operating mode as needed.

[1475] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1476] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1477] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1478] [Fourth embodiment]

[1479] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1480] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1481] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1482] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1483] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1484] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1485] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1486] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1487] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1488] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1489] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1490] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1491] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1492] The present invention relates to a cleaning robot system using a night vision camera, and specific embodiments thereof will be described below. Details of program processing will be explained with specific examples, divided into the subjects of the server, terminal, and user.

[1493] overview

[1494] The cleaning robot system first captures the environment using a night vision camera, analyzes the footage, and extracts features. It then creates a map using an autonomous navigation system and detects dirt using optical recognition technology. When dirt is detected, the robot performs cleaning operations and sends the results as an alert in real time. It also uses a learning algorithm to optimize cleaning activities, and sends cleaning logs to a server for storage in a database. This ensures efficient and accurate late-night cleaning.

[1495] Server Roles and Operations

[1496] The server receives data sent from the robot and updates the integrated database. Specifically, it stores information such as navigation data, map data, and cleaning history, and uses it as a reference for the robot to perform cleaning activities efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[1497] example:

[1498] The server receives the map data sent from each robot and stores it in an integrated database.

[1499] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[1500] Roles and actions of the terminal (robot)

[1501] The terminal (robot) operates autonomously and performs cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins processing. Next, optical recognition technology is used to extract features from the image and update the map of the current location. After that, it uses an autonomous navigation system to calculate the next position to go to, and if dirt is detected, it performs cleaning operations.

[1502] example:

[1503] The device uses a night vision camera to capture and map the environment at night.

[1504] The device will start cleaning when it detects dirt in a specific location.

[1505] User Roles and Actions

[1506] Users can remotely monitor the robot's status and issue instructions as needed. Users receive real-time alerts to see the cleaning robot's activity and can also give instructions to re-clean specific designated areas.

[1507] example:

[1508] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[1509] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[1510] The learning and optimization process

[1511] The robot learns using cleaning activity data and map data, optimizing future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[1512] example:

[1513] The device sends data on cleaning activities to a server, which stores it in a database.

[1514] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[1515] The processing flow will be explained below.

[1516] Step 1:

[1517] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[1518] Step 2:

[1519] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale to facilitate further processing.

[1520] Step 3:

[1521] The device extracts features from the captured video and updates a map of the device's current location based on the extracted features, accurately determining the device's location.

[1522] Step 4:

[1523] The device uses an autonomous navigation system to calculate and navigate its next location, allowing the device to efficiently cover the entire area.

[1524] Step 5:

[1525] The device detects dirt using optical recognition technology, capturing the environment with a night vision camera and analyzing the footage to identify dirt and foreign objects.

[1526] Step 6:

[1527] The device will perform cleaning operations at the location where the dirt is detected, and once the cleaning operation is complete, the device will send the results to the server in real time and also send an alert that the cleaning is complete.

[1528] Step 7:

[1529] The server stores the cleaning log received from the device in a database, analyzes the cleaning history and map data, derives efficient cleaning patterns, updates the learning algorithm, and provides feedback to the device on an optimized navigation system.

[1530] Step 8:

[1531] Users remotely monitor the device's location and cleaning status in real time, and can send instructions to the device to re-clean specific areas as needed.

[1532] Step 9:

[1533] The device receives updated information from the server and instructions from the user to continue cleaning, and then initiates the next cleaning activity accordingly. Through this cyclical process, the device continuously improves its cleaning efficiency and contributes to environmental conservation.

[1534] Example 1

[1535] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1536] Current cleaning robot systems have difficulty performing efficient and accurate cleaning at night, and there are many challenges with dirt detection and map creation, especially in environments that use night vision. This reduces cleaning accuracy and efficiency, and increases time and costs. Furthermore, few systems utilize cleaning activity logs to further improve efficiency and optimization. To solve these problems, a more accurate and efficient cleaning robot system is needed.

[1537] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1538] In this invention, the server includes means for capturing the environment using a night vision camera, means for converting the captured image to grayscale to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, and means for sending cleaning data to the server and storing it in an integrated database. This makes it possible to detect dirt with high accuracy even in nighttime environments, perform efficient cleaning, and optimize future cleanings by utilizing cleaning activity log data.

[1539] A "night vision camera" is a special imaging device that can capture images even in darkness or low-light environments.

[1540] The "means for capturing the environment" is a method for capturing an image of the surroundings using a night vision camera.

[1541] "Means for converting to grayscale" refers to a method for converting the acquired image into a format that is expressed in shades of black and white.

[1542] The "means for extracting features and updating position information" is a method for detecting specific patterns or objects from the image converted to grayscale and using that information to update the robot's position.

[1543] "Means for autonomous navigation and map creation" refers to a method in which a robot determines its own route and generates a map based on information about its surroundings.

[1544] "Optical recognition technology" is a technology that analyzes video data acquired by a camera and recognizes objects and features.

[1545] The "means for detecting dirt" is a method for identifying dirt from captured images using optical recognition technology.

[1546] "Means for performing cleaning actions" refers to a series of actions that the robot takes to remove the detected soiling.

[1547] "Means for sending real-time alerts" refers to a method for sending immediate notification to users or systems after cleaning is completed.

[1548] "Means for transmitting cleaning data to a server and storing it in an integrated database" refers to a method for transmitting records of cleaning activities to a server via the Internet or the like and storing them there.

[1549] The present invention relates to an autonomous cleaning robot system using a night vision camera, and specific embodiments thereof will be described below.

[1550] Server Roles and Operations

[1551] The server receives data sent from the robot (navigation data, map data, cleaning history, etc.) and stores it in an integrated database. The server analyzes the information stored in the database and identifies areas that are frequently soiled. This allows for the creation of an efficient and effective cleaning plan. The server also analyzes the cleaning history, generates new cleaning algorithms, and distributes them to the terminal (robot).

[1552] Examples:

[1553] The server receives the map data sent from each robot and stores it in an integrated database.

[1554] The server analyzes the cleaning history and derives an efficient cleaning pattern.

[1555] Roles and actions of the terminal (robot)

[1556] The terminal (robot) operates autonomously to carry out cleaning activities. It uses a night vision camera to capture the environment, converts the image to grayscale, and begins image processing. Optical recognition technology is used to extract features from the grayscale image and update the robot's current location on a map. Next, an autonomous navigation system is used to calculate an appropriate route, and if dirt is detected, it immediately begins cleaning.

[1557] Examples:

[1558] The device uses a night vision camera to capture and map the environment at night.

[1559] The device will start cleaning when it detects dirt in a specific location.

[1560] User Roles and Actions

[1561] Users can remotely monitor the robot's status. They can check the robot's location and cleaning status in real time using a smartphone or other device. If necessary, users can also issue instructions to re-clean a specific area. If an alert occurs, they can quickly check the details and take appropriate action.

[1562] Examples:

[1563] Users can check the robot's location on their smartphone and monitor the cleaning progress in real time.

[1564] Users can receive alerts from the robot and instruct it to re-clean specific areas.

[1565] The learning and optimization process

[1566] The device (robot) learns from cleaning activity data and map data and optimizes future cleaning activities. The server analyzes the overall cleaning pattern based on the learning data sent from the device and updates the efficient cleaning algorithm, thereby improving cleaning accuracy and efficiency.

[1567] Examples:

[1568] The device sends data on cleaning activities to a server, which stores it in a database.

[1569] The server analyzes all the data and delivers new cleaning algorithms to the robots.

[1570] Example prompt for a generative AI model:

[1571] "Please explain how to save map data of a cleaning robot acquired at night by a night vision camera to a server."

[1572] "Explain how you can use a night vision camera to capture video, convert it to grayscale, detect dirt, and update the map."

[1573] "Explain how you can monitor the status of your cleaning robot on your smartphone and receive alerts to re-clean specific areas."

[1574] "Explain how data collected from a cleaning robot can be used to optimize the next cleaning session."

[1575] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1576] Step 1:

[1577] The terminal (robot) activates the night vision camera and starts capturing the environment. The night vision camera acquires images of the environment and sends the data to the computer vision system. The input is the raw image captured by the night vision camera. The output is the image data itself.

[1578] Specific behavior:

[1579] The device switches to night setting mode.

[1580] Activate the night vision camera and capture video continuously.

[1581] Step 2:

[1582] The device converts the captured video to grayscale. The input is color video data acquired from a night vision camera, and the output is grayscale video data. This conversion makes the video data easier to analyze.

[1583] Specific behavior:

[1584] The terminal activates an image processing module to convert the color image into grayscale format.

[1585] The converted grayscale image is stored in a temporary memory.

[1586] Step 3:

[1587] The device uses optical recognition technology to extract features from the grayscale image and update the location information. The input is the grayscale image data obtained in step 2, and the output is feature data and updated location information. The feature data indicates important parts of the image, such as dirt or obstacles.

[1588] Specific behavior:

[1589] The device runs a feature extraction algorithm to recognize dirt and obstacles from the video data.

[1590] The current location information is updated based on the extracted features.

[1591] Step 4:

[1592] The device updates the map using the autonomous navigation system. The input is the feature data and location information obtained in step 3, and the output is updated map data. This map is modified by adding new data based on the known range.

[1593] Specific behavior:

[1594] The device runs a navigation algorithm and updates the map to reflect the current location and feature data.

[1595] The updated map is stored in memory and prepared for the next action.

[1596] Step 5:

[1597] The device performs cleaning operation when dirt is detected. The input is the map data updated in step 4 and the location information of the dirt, and the output is the cleaning completion data.

[1598] Specific behavior:

[1599] An autonomous navigation system is used to navigate to the location where the contamination is detected.

[1600] Activates the cleaning module and cleans the designated area.

[1601] Step 6:

[1602] After cleaning, the terminal sends a real-time alert to the server. The input is the cleaning completion data, and the output is the alert notification to the server.

[1603] Specific behavior:

[1604] The terminal activates the communication module and transmits cleaning completion data to the server.

[1605] An alert is sent to the server and the user is notified.

[1606] Step 7:

[1607] The server receives the cleaning data sent from the terminal and stores it in an integrated database. The input is the cleaning data sent from the terminal, and the output is the history data stored in the database.

[1608] Specific behavior:

[1609] The server starts the communication module and receives data from the terminal.

[1610] The received data is analyzed and stored in an integrated database.

[1611] Step 8:

[1612] The server analyzes the received cleaning data and generates a new cleaning algorithm. The input is the stored cleaning history data, and the output is the updated cleaning algorithm.

[1613] Specific behavior:

[1614] The server launches a data analysis module and calculates an efficient cleaning pattern based on past cleaning history.

[1615] Generate new algorithms and prepare them for distribution to the robots.

[1616] Step 9:

[1617] The user remotely monitors the robot's status and issues instructions for re-cleaning as necessary. The input is alert notifications from the server and the robot's location information, and the output is instructions for re-cleaning.

[1618] Specific behavior:

[1619] The user launches the smartphone app and checks the robot's status.

[1620] If necessary, instructions to re-clean specific areas will be sent via the app.

[1621] (Application example 1)

[1622] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] In large facilities such as logistics centers, efficient cleaning is difficult in environments with poor visibility at night. Furthermore, to efficiently clean large areas, it is necessary to manage the robot's real-time location information, detect dirt, and respond immediately. However, current cleaning systems lack the means to effectively solve these issues. There is also a need for a method to optimize the efficiency of cleaning activities and improve future cleaning routes based on past cleaning data.

[1624] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1625] In this invention, the server includes: means for capturing the environment using a night vision camera; means for processing the captured image to extract features and update location information; means for autonomously navigating and creating maps; means for detecting dirt using optical recognition technology; means for performing cleaning operations to clean the detected dirt; means for sending real-time alerts after cleaning; means including a smartphone application for displaying the robot's real-time location information; means for viewing the environmental image captured by the night vision camera; means for issuing alerts when dirt is detected; means for remotely operating the robot; and means for displaying cleaning history and checking learning data. This enables efficient and accurate cleaning even in environments with poor visibility, even at night, and enables real-time status monitoring and immediate response. Furthermore, it enables optimization of cleaning activities and improvement of cleaning routes based on past data.

[1626] A "night vision camera" is a camera used to ensure visibility in the dark, and is a device that uses infrared light to capture the environment.

[1627] The "means for capturing the environment" is a mechanism that has the function of acquiring the surrounding situation as video data using a night vision camera.

[1628] The "means for extracting features and updating location information" is a technology that detects specific patterns or objects from captured video data and updates the current location information based on that.

[1629] "Means for autonomous navigation and map creation" refers to a system that automatically moves based on acquired location information, aggregates information from each location, and generates a map.

[1630] "Optical recognition technology" is a technology that uses optical devices such as cameras to detect specific objects or features from image data.

[1631] The "means for detecting dirt" is a system that uses optical recognition technology to identify dirt from video data and pinpoint its location.

[1632] "Means for performing cleaning actions" refers to a series of operations that the robot performs to remove the detected dirt.

[1633] "Means for sending alerts in real time" refers to a system that has a communication function to immediately notify users of detected dirt and the cleaning status after cleaning.

[1634] The "smartphone application for displaying real-time location information of a robot" is an application that displays the current location of a cleaning robot on a smartphone in real time.

[1635] "Means for viewing environmental images captured by a night vision camera" refers to a mechanism for displaying images captured by a night vision camera on a smartphone or other device.

[1636] "Means for issuing an alert notification when dirt is detected" refers to a mechanism that has the function of sending an immediate notification when dirt is confirmed.

[1637] "Means for remotely operating a robot" refers to remote control technology for remotely controlling the operation of a cleaning robot.

[1638] "Means for displaying cleaning history and checking learning data" refers to a mechanism that displays the history of past cleaning activities and learns from the acquired data to improve the cleaning algorithm.

[1639] This section describes a specific embodiment of a cleaning robot system using a night vision camera. This invention utilizes an autonomously operating cleaning robot to achieve efficient cleaning, particularly in large-scale facilities such as logistics centers.

[1640] System Configuration

[1641] The cleaning robot system of the present invention consists of the following main components:

[1642] 1. Autonomous mobile robot equipped with a night vision camera

[1643] 2. Smartphone Applications

[1644] 3. Server

[1645] 4. Communications Infrastructure

[1646] Server Roles

[1647] The server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for autonomously navigating and creating a map, means for detecting dirt using optical recognition technology, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning routes, and means for sending a cleaning log to the server and storing it in a database.

[1648] Specifically, the server processes the map data and cleaning history sent from the robot, saving them in an integrated database, and analyzing this data to calculate efficient cleaning patterns.The main software used is PostgreSQL for database management and Python libraries for analysis.

[1649] The role of the terminal (robot)

[1650] The terminal (robot) captures the environment with a night vision camera, converts the image to grayscale, and detects dirt using optical recognition technology. It then performs cleaning operations on the detected dirt and updates the map. The hardware used is a night vision camera and an autonomous mobile robot, and the software uses ROS (Robot Operating System). This enables autonomous navigation and cleaning.

[1651] User Roles

[1652] Users can monitor the robot's status through a smartphone application and remotely control it as needed. The application displays the robot's real-time location information, allows viewing of environmental images from a night vision camera, sends alerts when dirt is detected, displays cleaning history, and checks learning data. The application was developed using Android Studio.

[1653] A specific example is the use of a smartphone app to check the cleaning status of warehouse area A in real time. For example, when the robot detects dirt and begins cleaning, a notification is sent to the user, who can then view footage of the site on their smartphone. If necessary, the user can remotely instruct the robot to re-clean a specific area.

[1654] Prompt Sentence Examples

[1655] Enter the following prompt into the generative AI model to generate an alert message:

[1656] "A cleaning robot using a night vision camera in a distribution center has detected dirt in a specific area. Cleaning is required as an emergency response. Please generate a message for an alert notification."

[1657] Example: "Urgent Alert: Soil detected in Warehouse Area A of the Distribution Center. Immediate management attention required."

[1658] Thus, the present invention is a cleaning robot system using a night vision camera, and provides a specific method for effectively cleaning environments such as logistics centers.

[1659] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1660] Program processing flow and detailed explanations for each step

[1661] Step 1: Capture your environment

[1662] Subject: Terminal

[1663] The terminal (robot) captures the environment using a night vision camera. The night vision camera acquires video data of the surroundings and converts the video data into grayscale. This allows for highly accurate video data to be obtained even in dark places. The acquired video data is also temporarily stored in local storage.

[1664] Input: Surrounding video data

[1665] Output: Grayscale converted video data

[1666] Step 2: Feature extraction and location update

[1667] Subject: Terminal

[1668] The video data converted to grayscale is analyzed and features are extracted using optical recognition technology. Image analysis algorithms (such as the OpenCV library) are used for feature extraction. The current location information is updated based on the feature information obtained from this analysis. This allows the robot's position to be accurately displayed on a map.

[1669] Input: Grayscale converted video data

[1670] Output: extracted feature information, updated location information

[1671] Step 3: Create a map

[1672] Subject: Terminal

[1673] Based on the extracted feature information and location information, the system performs autonomous navigation and creates a map of the surrounding area using the SLAM (Simultaneous Localization and Mapping) algorithm. This map data is temporarily stored in the device's local memory.

[1674] Input: extracted feature information, updated location information

[1675] Output: Created map data

[1676] Step 4: Detecting Contamination

[1677] Subject: Terminal

[1678] The device detects dirt from map data created using optical recognition technology, a process that uses algorithms to identify specific colors and shapes, and when a dirt is detected, its location is recorded in a log.

[1679] Input: Created map data

[1680] Output: Specific location information of stains

[1681] Step 5: Perform cleaning actions

[1682] Subject: Terminal

[1683] When dirt is detected, the device initiates a cleaning operation, the robot moves to a specific location, performs the cleaning function, and when the cleaning is complete, the information is recorded in a log.

[1684] Input: Specific location of stain

[1685] Output: Cleaning completion log

[1686] Step 6: Sending an alert

[1687] Subject: Terminal

[1688] Once the cleaning is complete, the device sends a real-time alert that includes information about the cleaning completion and the specific location of the cleaning. The alert is sent to the smartphone application and server.

[1689] Input: Cleanup completion log

[1690] Output: Alert notification

[1691] Step 7: Save your data

[1692] Subject: Server

[1693] The server receives the map data and cleaning logs sent from the device and stores them in an integrated database, which centrally manages past cleaning history and map information.

[1694] Input: Map data, cleaning log

[1695] Output: Data stored in the integrated database

[1696] Step 8: Learn and optimize

[1697] Subject: Server

[1698] The server analyzes the stored data and uses a learning algorithm to optimize cleaning activities. This process includes identifying frequently soiled areas and calculating efficient cleaning patterns. The analysis results are reflected in the cleaning algorithm and used for future cleaning activities.

[1699] Input: Data from the integrated database

[1700] Output: Optimized cleaning algorithm

[1701] In this way, the present invention realizes efficient cleaning in large-scale facilities such as logistics centers through specific processing steps of a cleaning robot system using a night vision camera.

[1702] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1703] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and achieves efficient cleaning through optical recognition technology and autonomous navigation. This system has the ability to recognize the user's emotions and adjust its behavior based on those emotions. Below, we will explain the details of the program processing, with concrete examples, divided into the subjects of the server, terminal, and user.

[1704] Server Roles and Operations

[1705] The server receives data sent from the robot and updates the integrated database. It stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[1706] example:

[1707] The server receives the map data sent from each robot and stores it in an integrated database.

[1708] The server analyzes cleaning history and emotional data to derive efficient cleaning patterns and areas for improvement.

[1709] Roles and actions of the terminal (robot)

[1710] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera and converts the image to grayscale for processing. It extracts features from the image using optical recognition technology and updates a map of the current location. It uses an autonomous navigation system to calculate the next location to go to and navigate. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[1711] example:

[1712] The device uses a night vision camera to capture and map the environment.

[1713] Cleaning is performed at the location where dirt is detected, and the cleaning status is reported to the server.

[1714] The terminal recognizes the user's emotions using an emotion engine, and operates quietly, for example, when the user is feeling stressed.

[1715] User Roles and Actions

[1716] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[1717] example:

[1718] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[1719] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[1720] The learning and optimization process

[1721] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[1722] example:

[1723] The device transmits cleaning activity and emotion data to a server and stores it in a database.

[1724] The server analyzes the data and distributes new cleaning algorithms to the robots.

[1725] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[1726] As described above, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[1727] The processing flow will be explained below.

[1728] Step 1:

[1729] The device starts the system, initializes the various sensors and night vision camera, and connects to the network to establish communication with the server.

[1730] Step 2:

[1731] The device uses a night vision camera to capture the surrounding environment, and the captured image is converted to grayscale for easier further processing.

[1732] Step 3:

[1733] The device extracts features from the captured video, including the patterns on walls, furniture, and floors, and uses these features to create a map and update its current location.

[1734] Step 4:

[1735] The device uses an autonomous navigation system to calculate the next location to go to and navigate, moving while efficiently covering the designated area.

[1736] Step 5:

[1737] The device uses optical recognition technology to detect dirt and debris on the floor, analyzing images from a night vision camera and recognizing distinctive dirt patterns.

[1738] Step 6:

[1739] The device performs cleaning operations at the location where dirt is detected, including sweeping, vacuuming, wiping with water, etc. Once the cleaning is complete, the results are recorded.

[1740] Step 7:

[1741] The device sends information about the location where cleaning was completed and the cleaning results to the server in real time. The server receives this information and stores it in a database. The device also sends an alert to the user when cleaning is complete.

[1742] Step 8:

[1743] Users can check the robot's location and cleaning status in real time on their smartphone or PC, and can also instruct it to clean a specific area again.

[1744] Step 9:

[1745] The device uses an emotion engine to recognize the user's emotions. For this purpose, the device is equipped with a camera and microphone, and analyzes emotions from the user's facial expressions and voice.

[1746] Step 10:

[1747] The device adjusts its cleaning behavior based on the user's emotions, for example, if the user is feeling stressed, the device will operate in quiet mode to maintain a relaxing environment for the user.

[1748] Step 11:

[1749] The device sends cleaning activity and emotion data to the server, which stores it in a database. A learning algorithm optimizes future cleaning activities.

[1750] Step 12:

[1751] The server analyzes the data, updates the cleaning algorithm and emotion-based cleaning patterns, and feeds them back to the device, continuously improving cleaning accuracy and user satisfaction.

[1752] Example 2

[1753] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1754] Conventional cleaning robot systems were able to accurately recognize their environment and autonomously clean efficiently, but they were unable to adjust their behavior based on the user's emotions, making it difficult to clean in a way that took the user's comfort into consideration. Furthermore, they lacked real-time cleaning status reports and adequate optimization of cleaning activities, creating challenges in improving cleaning efficiency.

[1755] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for transmitting the cleaning status to the server in real time, a means for recognizing the user's emotion using an emotion recognition engine, and a means for adjusting the cleaning mode based on the user's emotion. This enables the terminal to operate in accordance with the user's emotion, providing comfortable cleaning and monitoring the cleaning status in real time, thereby realizing efficient cleaning activities.

[1756] A "night vision camera" is a camera that can capture clear images even in dark environments.

[1757] The "means for capturing the environment" is a method for capturing an image of the surrounding environment using a night vision camera.

[1758] The "means for processing the image to extract features and update the position information" is a method for extracting important data from the captured image and calculating and updating the current position of the robot.

[1759] "Means for autonomous navigation and mapping" refers to a method that uses optical recognition technology and sensors to determine one's own position, plan a route, and generate a map of the surrounding area.

[1760] "Optical recognition technology" is a technology that uses image processing technology to identify objects and features.

[1761] A "means for detecting soiling" is a method for identifying soiling on a surface using optical recognition technology.

[1762] The "means for performing a cleaning action" refers to an action that the robot takes to remove the detected dirt, such as suction or brushing.

[1763] "Means for transmitting cleaning status to a server in real time" refers to a method for transmitting data on the progress and status of cleaning to a server in real time.

[1764] An "emotion recognition engine" is software or hardware that identifies the emotional state of a user from their voice and facial expressions.

[1765] The "means for recognizing the user's emotions" is a method for determining the user's emotional state using an emotion recognition engine.

[1766] The "means for adjusting the cleaning mode" is a method for changing the operation mode of the cleaning robot based on the emotional state of the user.

[1767] A "learning algorithm" is an artificial intelligence technology that uses past data and patterns to optimize future actions.

[1768] "Means to optimize cleaning activities and improve future cleaning routes" refers to a method that uses a learning algorithm to calculate efficient cleaning patterns and routes.

[1769] "Means for customizing cleaning patterns using user emotional data" refers to a method for designing an optimal cleaning plan for a user based on collected emotional data.

[1770] A "cleaning log" is a record of cleaning activities performed by a robot.

[1771] A "database" is a system for systematically organizing and storing information.

[1772] This invention combines an emotion engine with a cleaning robot system that uses a night vision camera to capture the environment and uses optical recognition technology and autonomous navigation to achieve efficient cleaning. The system has the ability to recognize the user's emotions and adjust its behavior based on those emotions.

[1773] Server Roles and Operations

[1774] The server receives data sent from the robot and updates the integrated database. For example, it stores information such as navigation data, map data, cleaning history, and emotion data from the emotion engine, and uses this information to help the robot clean efficiently in the future. The server also analyzes cleaning logs to identify frequently soiled areas and use this information to plan future cleanings.

[1775] Specific behavior:

[1776] The server receives the map data sent by each robot and stores it in an integrated database, accumulating information such as the overall image of the room and the cleaning frequency of each area.

[1777] The server analyzes cleaning history and emotion data to derive efficient cleaning patterns and areas for improvement. For example, if it finds that a particular area gets dirty frequently during a certain time period, it will create a cleaning plan to focus on that area.

[1778] Roles and actions of the terminal (robot)

[1779] The terminal (robot) operates autonomously and performs cleaning activities. It captures the environment with a night vision camera, converts the image to grayscale, and processes it. It uses optical recognition technology to extract features from the image and updates a map of its current location. It uses an autonomous navigation system to calculate and navigate to the next location. When dirt is detected, it performs cleaning operations, and after cleaning is complete, it sends the results to a server and sends an alert in real time. It also uses an emotion engine to recognize the user's emotions and adjusts its operations based on those emotions.

[1780] Specific behavior:

[1781] The device uses a night vision camera to capture and map the environment, for example, accurately detecting furniture placement and obstacles even in the dark and generating a movement map.

[1782] The device performs cleaning at the location where the dirt is detected and reports the cleaning status to the server. For example, it detects a drink spilled on a carpet and soaks it up on the spot.

[1783] The device uses an emotion engine to recognize the user's emotions and, for example, operates quietly if the user is feeling stressed. For example, if the user wants to relax at night, the device will clean in quiet mode with reduced noise.

[1784] User Roles and Actions

[1785] Users can remotely monitor the robot's status and issue commands as needed. The emotion engine allows users to receive emotional feedback from the robot and switch the robot's mode based on that feedback. For example, users can set the robot to operate in quiet mode when the family is relaxing. Users can also receive real-time alerts to check the cleaning robot's activity status.

[1786] Specific behavior:

[1787] Users can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. For example, they can check using the smartphone app to see if a particular room has been cleaned.

[1788] The user receives emotional feedback from the robot and can instruct it to change its operating mode as needed, for example, cleaning in normal mode during the day and switching to quiet mode at night.

[1789] The learning and optimization process

[1790] The device sends data from the cleaning activity and emotion engine to the server, which uses a learning algorithm to optimize future cleaning activities. The server analyzes the overall cleaning pattern and emotion data, updates the efficient cleaning algorithm, and feeds it back to the device. This continuously improves cleaning accuracy and efficiency. It can also provide customized cleaning patterns that respond to the user's emotions.

[1791] Specific behavior:

[1792] The device sends cleaning activity and emotion data to the server and stores it in a database. For example, after cleaning is completed, the device uploads the cleaning activity and emotion data of the user for that day.

[1793] The server analyzes the data and distributes new cleaning algorithms to the robot, such as calculating and directing efficient cleaning routes for specific times of the day or days of the week.

[1794] Based on the user's emotional data, a customized cleaning pattern is derived for further optimization.

[1795] Examples of prompts for generative AI models

[1796] Example prompt:

[1797] "Please explain how the cleaning robot captures its environment with a night vision camera. Please also explain the basic image processing flow and autonomous navigation."

[1798] Please explain, with specific algorithms and examples, how a cleaning robot can recognize a user's emotions and adjust its behavior accordingly.

[1799] In this way, the present invention is a system that combines an emotion engine with a cleaning robot system that uses a night vision camera to achieve advanced cleaning operations that respond to the user's emotions, contributing to efficient and accurate late-night cleaning and environmental conservation.

[1800] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1801] Step 1: Initialization and Data Collection

[1802] When the system is powered on, the terminal (robot) performs an initialization process. It starts up the night vision camera, sensors, navigation module, and emotion engine, and collects initial data on the working area. Specifically, it uses the night vision camera to capture the surrounding environment, converts the image to grayscale, and processes it. At the same time, it obtains location information and obstacle distance data from various sensors (such as LIDAR and infrared sensors).

[1803] input:

[1804] Powering on the system

[1805] Environmental data from night vision cameras and sensors

[1806] Data processing / data calculation:

[1807] Converting an image to grayscale

[1808] Sensor information acquisition and integration

[1809] output:

[1810] Initialization complete message

[1811] Environmental Datasets

[1812] Specific behavior:

[1813] The device captures an initial image with its night vision camera and records the location of each obstacle.

[1814] Step 2: Data processing and mapping

[1815] The device processes the collected data and creates a map of the room. Specifically, it uses optical recognition technology to extract feature points from grayscale images and generate a map based on each piece of data. An autonomous navigation system then updates the device's position and surrounding terrain in real time.

[1816] input:

[1817] Environmental Datasets

[1818] Grayscale images

[1819] Data processing / data calculation:

[1820] Extracting feature points from images

[1821] Running the map generation algorithm

[1822] output:

[1823] Generated map data

[1824] Current location information

[1825] Specific behavior:

[1826] The device applies edge detection algorithms to identify the location of furniture and walls.

[1827] The terminal generates a map and passes it to the navigation algorithm.

[1828] Step 3: Navigation and Cleaning Plan

[1829] The device uses map data and navigation algorithms to plan the optimal cleaning route, with the navigation system calculating the next location and guiding the robot.

[1830] input:

[1831] Map data

[1832] Current location information

[1833] Data processing / data calculation:

[1834] Execution of the navigation algorithm

[1835] Cleaning route calculation

[1836] output:

[1837] Cleaning route planning

[1838] Next Movement Point

[1839] Specific behavior:

[1840] The device uses algorithms such as the A algorithm to determine an efficient cleaning route while avoiding obstacles.

[1841] The terminal sets the cleaning pattern and indicates the cleaning sequence.

[1842] Step 4: Cleaning execution and real-time monitoring

[1843] The device moves along a planned cleaning route, performs cleaning actions when dirt is detected, and transmits the cleaning status to the server in real time.

[1844] input:

[1845] Cleaning route planning

[1846] Dirt detection data

[1847] Data processing / data calculation:

[1848] Running the dirt detection algorithm

[1849] Cleaning trigger

[1850] output:

[1851] Cleaning progress data

[1852] Real-time upload to server

[1853] Specific behavior:

[1854] The device uses suction or a brush to clean the area where dirt is detected.

[1855] The terminal reports cleaning progress to the server in real time.

[1856] Step 5: Emotion recognition and behavior regulation

[1857] The device uses an emotion engine to recognize the user's emotions and adjusts its behavior based on that information.

[1858] input:

[1859] User voice and facial expression data

[1860] Analysis results from the emotion engine

[1861] Data processing / data calculation:

[1862] Applying speech and facial expression recognition algorithms

[1863] Emotional state estimation

[1864] output:

[1865] Emotion analysis results

[1866] Operation adjustment command

[1867] Specific behavior:

[1868] If the user is in a stressful state, the terminal switches to a silent mode that reduces the operating noise.

[1869] Step 6: Data analysis and feedback

[1870] The server analyzes the cleaning data and emotion data sent from the device, updates the efficient cleaning algorithm, and provides feedback to the device on a new cleaning plan based on the analysis results.

[1871] input:

[1872] Cleaning Data

[1873] Emotional Data

[1874] Data processing / data calculation:

[1875] Executing data analysis algorithms

[1876] Optimized cleaning algorithm

[1877] output:

[1878] Updated cleaning algorithm

[1879] Feedback data to the device

[1880] Specific behavior:

[1881] The server analyzes cleaning frequency and effective cleaning order and sends the results to the terminal.

[1882] Step 7: User monitoring and guidance

[1883] Users can use a smartphone or tablet to monitor the robot's operating status and issue instructions to the robot as needed.

[1884] input:

[1885] Instructions from a smartphone app

[1886] Real-time Monitoring Data

[1887] Data processing / data calculation:

[1888] Analysis of instruction data

[1889] output:

[1890] Commands and instructions for robots

[1891] Situation Report Data

[1892] Specific behavior:

[1893] Users send specific instructions to the robot through the app, such as "prioritize cleaning the living room."

[1894] (Application example 2)

[1895] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1896] Efficient cleaning activities are important in modern factories, but the effectiveness of automated cleaning systems is limited in environments with low visibility at night. Furthermore, while there is a need to adjust operations based on worker emotions, conventional systems lack emotion recognition capabilities and are therefore unable to meet this requirement. Furthermore, it is difficult to integrate cleaning activity data with employee emotion data to provide optimized cleaning patterns.

[1897] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the environment using a night vision camera, means for processing the captured image to extract features and update location information, means for detecting dirt using optical recognition technology, means for autonomously navigating and creating a map, means for performing cleaning operations to clean the detected dirt, means for sending an alert in real time after cleaning, means for adjusting operations based on the user's emotions with an emotion recognition function, means for optimizing cleaning activities using a learning algorithm and improving subsequent cleaning routes, and means for transmitting cleaning logs and emotion data to the server and storing them in a database. This enables effective cleaning operations even in environments with low visibility at night, realizes operation adjustments based on the user's emotions, and provides optimized cleaning patterns.

[1898] A "night vision camera" is a camera device that can capture clear images of objects even in low-light environments.

[1899] "Capturing" refers to the process of taking in video or images.

[1900] "Feature extraction" means finding important information or patterns in captured video or images.

[1901] "Updating location information" means recording the current location and coordinate data and replacing it with the latest information.

[1902] "Autonomous navigation" means determining the direction of travel and moving independently without external instructions.

[1903] "Creating a map" means recording the surrounding environment and the route you are taking and generating visualized data.

[1904] "Optical recognition technology" is a technology that uses cameras and sensors to determine the shape and characteristics of objects.

[1905] "Detecting dirt" means finding dirt or abnormalities on the surface.

[1906] "Performing a cleaning action" means taking action to remove the detected dirt.

[1907] "Real-time alerting" means being notified immediately of an event or situation that occurs.

[1908] The "emotion recognition function" is a function that determines a person's emotions from video and audio data.

[1909] "Adjusting behavior" means changing the way equipment or systems behave depending on the situation or conditions.

[1910] A "learning algorithm" is a computational method for analyzing accumulated data and discovering new knowledge and patterns.

[1911] "Optimizing cleaning activities" means finding ways to make cleaning tasks more efficient and effective.

[1912] "Improving the cleaning route from next time onwards" means improving the route for the next cleaning work based on past data.

[1913] "Cleaning log and emotional data" refers to a record of cleaning activities and data indicating the user's emotional state.

[1914] "Send to server" means sending data over the network to a central management system.

[1915] "Storing in a database" means organizing and storing the acquired data.

[1916] Server Roles and Operations

[1917] The server captures the environment using a night vision camera and receives data from the device that processes the image and extracts features. This process includes converting the captured image to grayscale and detecting dirt using optical recognition technology. The server also stores navigation data, map data, cleaning history, and emotion data from emotion recognition in an integrated database to support efficient cleaning activities. Furthermore, the server analyzes data sent from each robot and identifies frequently soiled areas to provide an optimal cleaning plan for future cleanings.

[1918] Examples:

[1919] The server stores the map data and cleaning history sent from the factory's nighttime cleaning robots in an integrated database.

[1920] The server analyzes the cleaning logs and emotional data to customize and optimize the robot's cleaning patterns.

[1921] Roles and actions of the terminal (robot)

[1922] The terminal (robot) autonomously moves around the factory to perform cleaning activities. The robot is equipped with a night vision camera, captures the environment, and converts the images to grayscale to detect dirt. It uses an autonomous navigation system to move efficiently while updating a map of its current location. It performs cleaning operations at points where dirt is detected and sends the results to a server in real time after cleaning is completed. It also uses emotion recognition functionality to recognize the emotions of nearby workers and adjust its behavior accordingly. For example, if a worker is feeling stressed, the robot will operate in silent mode.

[1923] Examples:

[1924] The robot uses night vision cameras to capture and map the environment in the factory at night.

[1925] It performs cleaning wherever dirt is detected and reports the results to the server in real time.

[1926] Emotion recognition allows the robot to switch to silent mode if the worker is feeling stressed.

[1927] User Roles and Actions

[1928] Users can remotely monitor the robot's status and issue instructions as needed. Using a smartphone or other device, users can monitor the robot's location, cleaning status, and emotional data in real time. Furthermore, the operating mode can be switched based on the emotional feedback received from the robot. For example, the robot can be set to operate in quiet mode when family members or redundant staff are relaxing. In this way, users can perform optimal cleaning operations that are tailored to the factory environment and the emotions of employees.

[1929] Examples:

[1930] Users can check the robot's location on their smartphone and monitor its cleaning progress and emotional feedback in real time.

[1931] The user receives emotional feedback from the robot and issues instructions to change the operating mode as needed.

[1932] The learning and optimization process

[1933] The device (robot) sends cleaning activity and emotion recognition data to a server, which analyzes the data. Based on the analysis results, a learning algorithm is used to optimize cleaning activities and improve cleaning routes and patterns for future cleanings. This improves the accuracy and efficiency of cleaning activities, even in environments with low visibility at night. In addition, the pattern of each cleaning activity can be customized based on the user's emotion data, allowing for even more detailed optimization.

[1934] Examples:

[1935] The terminal (robot) transmits cleaning activity and emotion data to the server and stores it in a database.

[1936] The server analyzes the data sent and distributes the cleaning algorithm to the robot for the next cleaning cycle.

[1937] Based on the user's emotional data, it creates a customized cleaning pattern and optimizes cleaning activities.

[1938] Prompt Sentence Examples

[1939] In factories, cleaning robots use night vision cameras to capture the environment and detect dirt. They use emotion recognition to analyze the facial expressions of workers while they work. If the robot detects "emotion: stress," for example, it switches to silent mode. Cleaning status and emotion data are sent to a server in real time, allowing it to learn optimal cleaning patterns.

[1940] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1941] Step 1:

[1942] The device uses a night vision camera to capture the environment. The input is the image captured by the camera, and the output is the video data, which is the basis for capturing detailed information, especially in low-light environments.

[1943] Step 2:

[1944] The device converts the captured video to grayscale and extracts features. This conversion process removes unnecessary color information, making it easier to detect dirt and obstacles. The input is the video data obtained in step 1, and the output is the grayscale image and extracted feature data.

[1945] Step 3:

[1946] The device detects dirt using optical recognition technology. The input is a grayscale image and feature data, and it determines whether dirt is present based on specific patterns and features. The output is the presence or absence of dirt and its location. For example, it performs noise reduction and edge detection to identify areas that appear to be dirty.

[1947] Step 4:

[1948] The device uses an autonomous navigation system to create a map and update its current location. The input is the device's current location information and feature data acquired from the environment, and the output is updated map information and location information. Based on this, the device can calculate where to go next and move efficiently.

[1949] Step 5:

[1950] The device performs cleaning operations at the location where the stain is detected. The input is the location of the stain, and the output is a cleaning completion report. The device's cleaning unit moves to the stained location and performs the specific stain removal task.

[1951] Step 6:

[1952] The device sends a real-time alert after cleaning. The input is the cleaning completion report and its location information, and the output is notification data sent to the server, allowing the server to keep track of the progress of each cleaning activity.

[1953] Step 7:

[1954] The device uses emotion recognition to recognize the user's emotions and adjusts its behavior based on those emotions. The input is camera video data, and the output is the emotion recognition results and the adjusted behavior mode. For example, if the user is feeling stressed, the device can switch to silent mode.

[1955] Step 8:

[1956] The server stores all data (cleaning logs, emotion data, and map information) in an integrated database. The input is the data sent from the device, and the output is the saved database state. The server analyzes this and uses it in an algorithm to optimize future cleaning patterns.

[1957] Step 9:

[1958] The user can check the robot's location on their smartphone and monitor the cleaning progress and emotional feedback in real time. The input is real-time data sent from the server, and the output is information displayed on the monitoring screen. The user can issue commands to switch the operating mode as needed.

[1959] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1960] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1961] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1962] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1963] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1964] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1965] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1966] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1967] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1968] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1969] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1970] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1971] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1972] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1973] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1974] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1975] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1976] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1977] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1978] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1979] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1980] The following is further disclosed regarding the above embodiment.

[1981] (Claim 1)

[1982] means for capturing an environment using a night vision camera;

[1983] means for processing the captured video to extract features and update location information;

[1984] A means of autonomously navigating and creating maps;

[1985] means for detecting contamination using optical recognition technology;

[1986] means for performing a cleaning operation to clean the detected soiling;

[1987] A means to send real-time alerts after cleaning, and

[1988] A system including:

[1989] (Claim 2)

[1990] 10. The system of claim 1, further comprising means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning paths.

[1991] (Claim 3)

[1992] 10. The system of claim 1, further comprising means for transmitting the cleaning log to a server and storing it in a database.

[1993] "Example 1"

[1994] (Claim 1)

[1995] means for capturing an environment using a night vision camera;

[1996] a means for converting the captured video to grayscale, extracting features, and updating location information;

[1997] A means of autonomously navigating and creating maps;

[1998] means for detecting contamination using optical recognition technology;

[1999] means for performing a cleaning operation to clean the detected soiling;

[2000] A means to send real-time alerts after cleaning, and

[2001] means for transmitting the cleaning data to a server and storing the data in an integrated database;

[2002] A system including:

[2003] (Claim 2)

[2004] 10. The system of claim 1, further comprising means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning paths.

[2005] (Claim 3)

[2006] 10. The system of claim 1, wherein the server further comprises means for analyzing the cleaning history and generating an efficient cleaning pattern.

[2007] "Application Example 1"

[2008] (Claim 1)

[2009] means for capturing an environment using a night vision camera;

[2010] means for processing the captured video to extract features and update location information;

[2011] A means of autonomously navigating and creating maps;

[2012] means for detecting contamination using optical recognition technology;

[2013] means for performing a cleaning operation to clean the detected soiling;

[2014] A means to send real-time alerts after cleaning, and

[2015] a means including a smartphone application for displaying real-time location information of the robot;

[2016] a means for viewing environmental images captured by a night vision camera;

[2017] a means for providing an alert notification when contamination is detected;

[2018] a means for remotely operating the robot;

[2019] A means to view cleaning history and check learning data,

[2020] A system including:

[2021] (Claim 2)

[2022] 10. The system of claim 1, further comprising means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning paths.

[2023] (Claim 3)

[2024] 10. Th...

Claims

1. means for capturing an environment using a night vision camera; means for processing the captured video to extract features and update location information; A means of autonomously navigating and creating maps; means for detecting contamination using optical recognition technology; means for performing a cleaning operation to clean the detected soiling; A means to send real-time alerts after cleaning, and A system including:

2. The system of claim 1 , further comprising means for optimizing cleaning activities using a learning algorithm to improve subsequent cleaning paths.

3. The system of claim 1 , further comprising means for transmitting the cleaning log to a server and storing it in a database.

Images