System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring an image or a moving image of a room; means for analyzing the image or the moving image and detecting dirt and an arrangement of objects in the room; means for proposing a cleaning place and a cleaning method based on the detected dirt and arrangement; means for generating a state of the room after cleaning using a virtual reality technology; and means for providing the generated virtual reality environment to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many people find cleaning their rooms a burden and worry about how to clean efficiently. As a result, they put off cleaning, which can lead to a vicious cycle of making their rooms even dirtier. Another issue is that it's difficult to maintain motivation because cleaning doesn't seem to have a positive effect or sense of accomplishment. The present invention aims to solve these problems and provide a support system for cleaning efficiently and effectively. [Means for solving the problem]

[0005] The present invention provides a system including means for acquiring images or videos of a room, means for analyzing the images or videos to detect dirt and the arrangement of objects in the room, means for proposing areas to be cleaned and cleaning methods based on the detected dirt and arrangement, means for generating the state of the room after cleaning using virtual reality technology, and means for providing the generated virtual reality environment to a user.The system also includes means for identifying tools required for cleaning and providing a link to purchase them to the user, and means for recording the state of the room after cleaning and providing a comparison of the before and after cleaning using virtual reality technology, thereby improving cleaning efficiency and motivation.

[0006] "Means for acquiring images or videos" refers to a camera or smartphone application function that allows a user to photograph or record the current state of a room.

[0007] "Means for analyzing images or video to detect dirt and object placement in a room" refers to image analysis algorithms that process the captured image or video data to identify dirt, clutter, and the location of objects in the room.

[0008] "Means for suggesting areas to clean and cleaning methods" refers to a system that provides users with efficient areas to clean and specific cleaning methods based on the results of image analysis.

[0009] "Means for generating the state of a room after cleaning using virtual reality technology" refers to VR (Virtual Reality) technology that simulates the state of a room after cleaning is completed and allows the user to experience it as a virtual environment.

[0010] "Means for providing the user with the generated virtual reality environment" refers to a system that allows the user to virtually experience the cleaned room visually using VR goggles and a compatible application.

[0011] "Means for identifying cleaning tools and providing users with links to purchase them" refers to a system that identifies items needed for the proposed cleaning method and provides users with online links to easily purchase them.

[0012] "Means for recording the state after cleaning and providing a comparison between before and after cleaning using virtual reality technology" refers to technology and systems for storing the state of a room before and after cleaning, comparing these states, and presenting them to a user in a virtual reality environment. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention provides a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment. Specific embodiments of the present invention are described below.

[0035] System configuration

[0036] The system consists of four main parts:

[0037] 1. Data Acquisition Module

[0038] 2. Image Analysis Module

[0039] 3. Cleaning suggestion module

[0040] 4. Virtual Reality (VR) Module

[0041] Data Acquisition Module

[0042] Users can take photos and videos of the room using an application on their smartphone or a dedicated device.

[0043] The device prepares the captured data for transmission to the server, specifically by compressing the data, converting the format, and adding metadata.

[0044] Image Analysis Module

[0045] The server inputs the received data into an image analysis module and processes it to identify dirty or cluttered areas in the room.

[0046] Image analysis uses deep learning-based algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[0047] The identified information is organized and stored in a database.

[0048] Cleaning suggestion module

[0049] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[0050] The suggestions include specific cleaning procedures and essential items (e.g., vacuum cleaner, detergent, brush).

[0051] The device notifies the user of this information and displays it in a concise and easy-to-understand format.

[0052] Virtual Reality (VR) Module

[0053] The server runs a VR module to simulate the state of the room after the user has completed cleaning.

[0054] By comparing data before and after cleaning, users can visually experience the difference in a VR environment, giving them a sense of accomplishment from cleaning.

[0055] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[0056] Specific examples

[0057] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[0058] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[0059] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the cleaned up state. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[0060] In this way, the present invention aims to achieve both improved cleaning efficiency and motivation.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user launches a dedicated application and takes photos and videos of the room.

[0064] Step 2:

[0065] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[0066] Step 3:

[0067] The terminal transmits the prepared data to the server.

[0068] Step 4:

[0069] The server passes the received data to the image analysis module.

[0070] Step 5:

[0071] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[0072] Step 6:

[0073] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[0074] Step 7:

[0075] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[0076] Step 8:

[0077] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[0078] Step 9:

[0079] The user cleans the room using the cleaning method presented as a reference.

[0080] Step 10:

[0081] The user takes a picture of the state of the room after cleaning again and uploads the new photo or video data to the terminal.

[0082] Step 11:

[0083] The device sends the post-cleaning data to the server.

[0084] Step 12:

[0085] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[0086] Step 13:

[0087] The virtual reality data generated by the server is sent to the terminal.

[0088] Step 14:

[0089] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[0090] Step 15:

[0091] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Cleaning a room is labor-intensive and troublesome for many people. It is also difficult to determine which areas need cleaning and the best cleaning methods, making it extremely difficult to clean effectively. Furthermore, the effectiveness of cleaning cannot be reliably confirmed, which can lead to a decline in motivation. There is a need for a system that can solve these issues, improve cleaning efficiency, and increase motivation.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for identifying dirty and messy areas in a room using an image analysis model, means for storing the analysis results in a database, means for proposing cleaning methods based on the analysis results, and means for comparing data before and after cleaning and generating the state of the room after cleaning using virtual reality technology, thereby enabling the user to accurately grasp the current state of the room, practice effective cleaning methods, and virtually experience the sense of accomplishment after cleaning.

[0097] "User" refers to an individual or group that uses this system to clean a room.

[0098] "Terminal" refers to the device that the user uses to take pictures and videos of the room and send them to the server. Specifically, this applies to smartphones and dedicated terminals.

[0099] "Server" refers to a central control unit for analyzing, storing, and making suggestions about data sent by users.

[0100] "Image analysis models" refer to programs that use deep learning algorithms to identify dirty or cluttered areas in a room, such as YOLO and Mask R-CNN.

[0101] A "database" refers to a system for systematically storing and searching analysis results and related information.

[0102] "Cleaning method" refers to information on the optimal cleaning procedures and cleaning tools to be used, identified based on the analysis results.

[0103] "Virtual reality technology" refers to technology that allows users to visually experience the state of a room after cleaning, specifically 3D modeling software and VR devices.

[0104] "Data before and after cleaning" refers to images and videos showing the state of the room before and after the user cleans.

[0105] "Virtual reality environment" refers to a virtual space generated by the server that visually reproduces the state of a room after cleaning.

[0106] The present invention is a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment.

[0107] 1. Specific data acquisition procedures

[0108] Users use their smartphones or dedicated devices to take photos and videos of their rooms. Specifically, they launch a smartphone app and take photos of various parts of the room. The device receives this data and organizes it by compressing and converting the format. For example, it converts it to JPEG format and adds metadata (date and time of shooting, location information).

[0109] 2. Specific operations for data transmission

[0110] The device sends the data the user has acquired to the server using Wi-Fi or mobile data, encrypting the data using security protocols such as SSL.

[0111] 3. Specific procedures for image analysis

[0112] The server inputs the received data into an image analysis module and uses deep learning-based image analysis models (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. The analysis results are organized in JSON format or similar and stored in a database.

[0113] 4. Specific cleaning operations

[0114] The server uses a generative AI model (e.g., OpenAI® GPT-4®) to generate effective cleaning methods based on the results of image analysis. The recommendations include specific cleaning steps and information on the tools needed. For example, it suggests that a specific detergent and brush are needed to remove stains from a carpet.

[0115] 5. Specific operation of VR simulation

[0116] After cleaning, the user takes photos and videos of the room again and uploads them to the server via their device. The server compares the data before and after cleaning and uses 3D modeling software (e.g., Blender) to generate a VR environment of the room after cleaning.

[0117] 6. Specific operations for VR experiences

[0118] The device connects to a VR device (e.g., Oculus Rift or HTC Vive) and downloads the generated VR environment data. Users can then wear VR goggles and visually experience the state of the room before and after cleaning, giving them a sense of accomplishment from cleaning.

[0119] Specific examples

[0120] For example, a user takes a photo of their living room and uploads it to their device through an application. This data is compressed, converted, and sent to a server. After analyzing the image, the server detects that there is a coffee stain on the carpet. The server then suggests using a specific detergent and brush to remove the stain from the carpet and provides a link to purchase the necessary tools. The device notifies the user, and the user completes the cleaning. By taking another photo and uploading it using the same steps, the server compares the state before and after cleaning and generates a VR environment. The device then plays this on a VR device, allowing the user to experience the virtual environment.

[0121] Prompt Sentence Examples

[0122] Below are some example prompts to input the behavior of this system into the generative AI model.

[0123] "Please explain in detail how this system uses image analysis and virtual reality technology to efficiently clean rooms. A user takes a photo of the room with their smartphone and sends it to a server via their device. The server then analyzes the image, detects dirt and clutter, and makes cleaning suggestions. After cleaning is complete, a VR module is used to compare the state before and after cleaning, allowing the user to experience the state after cleaning in a virtual environment."

[0124] This system improves the efficiency of the cleaning process while also increasing user motivation.

[0125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0126] Step 1:

[0127] The user takes photos and videos of the room using a smartphone or dedicated device. Specifically, the user starts the app and takes multiple photos of various parts of the room from various angles. The input is the images and videos of the room, and the output is this data.

[0128] Step 2:

[0129] The device compresses the captured data, converts the format, and adds metadata. The device converts the data to JPEG or MPEG format, and adds the date and time of the capture and location information as metadata. The input is the captured image or video, and the output is the compressed and converted data.

[0130] Step 3:

[0131] The device sends data to the server. The compressed and converted data is then sent to the server using Wi-Fi or mobile data. An encryption protocol such as SSL is used during transmission. The input is the compressed and converted data, and the output is the data sent to the server.

[0132] Step 4:

[0133] The server inputs the received data into an image analysis module, which uses a deep learning-based image analysis model (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. In the process, the server identifies dirt, stains, improper placement, etc. The input is the data sent to the server, and the output is the analysis results.

[0134] Step 5:

[0135] The server saves the analysis results in a database. The analysis results are organized in JSON format or similar and stored in the database in a format that allows for efficient search and reference. The input is the analysis results, and the output is the data saved in the database.

[0136] Step 6:

[0137] The server proposes cleaning methods based on the analysis results. Using a generative AI model (e.g., OpenAI GPT-4), it generates recommendations regarding optimal cleaning procedures and cleaning tools to use. The input is the analysis results, and the output is the recommendations.

[0138] Step 7:

[0139] The terminal notifies the user of the suggestions from the server. The terminal displays the suggested cleaning procedures and necessary tools in an easy-to-understand format. The input is the suggestions, and the output is the information notified to the user.

[0140] Step 8:

[0141] The user actually cleans based on the suggestions. The user uses the necessary tools and follows the suggested procedure to clean. The input is the suggestions and cleaning tools, and the output is the clean state after cleaning.

[0142] Step 9:

[0143] The user then takes photos or videos of the state of the room after cleaning and uploads them to the server via the device in the same way. The input is the post-cleaning images or videos, and the output is the post-cleaning data sent to the server.

[0144] Step 10:

[0145] The server compares the data before and after cleaning and generates the state of the room after cleaning using virtual reality technology. The server uses 3D modeling software (e.g., Blender) to visualize the difference between the state before and after cleaning. The input is the data before and after cleaning, and the output is the generated virtual reality environment.

[0146] Step 11:

[0147] The device provides the generated virtual reality environment to the user. The device works in conjunction with a VR device (e.g., Oculus Rift, HTC Vive) to allow the user to experience the virtual environment. The input is the generated virtual reality environment, and the output is the virtual experience provided to the user.

[0148] (Application example 1)

[0149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] Cleaning work in factories and other physical spaces is often manual and inefficient. Particularly in large factories, it is difficult to identify dirty or messy areas and clean them efficiently, increasing the burden on workers. Furthermore, there is no way to immediately check the results of cleaning, making it difficult to visualize the effectiveness of the work. To solve these problems, a system is needed that automates cleaning work and makes it easy to check the results.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0152] In this invention, the server includes means for analyzing images or videos to detect dirt and object placement in the physical space, means for proposing areas to be cleaned and cleaning methods based on the detected dirt and placement, and means for generating the state of the physical space after cleaning using virtual reality technology. This allows for efficient automation of cleaning work in physical spaces such as factories, and enables users to immediately visually confirm the cleaning results.

[0153] An "image or video capturing means" is a device or method for capturing images or video within a physical space.

[0154] "Means for analyzing images or videos" refers to an algorithm or device for detecting the placement of dirt or objects using captured images or videos.

[0155] A "means for detecting dirt or placement" is a system or algorithm that identifies the presence of dirt or the placement of an object based on analyzed images or video.

[0156] The "means for suggesting cleaning locations and cleaning methods" is a system or method that suggests specific locations to be cleaned and appropriate cleaning methods based on detected dirt and location information.

[0157] "Means for generating the state of a physical space after cleaning using virtual reality technology" refers to a system or technology that uses virtual reality technology to simulate and visually present what a physical space will look like after cleaning.

[0158] The "means for providing a generated virtual reality environment to a user" refers to a device or platform that allows a user to experience a virtual reality environment.

[0159] "Means for a robot to capture images or videos in a physical space and transmit them to a server" refers to a system or method for a robot to acquire images or videos using a photographing device and transmit the data to a server.

[0160] The "robot means for carrying out cleaning in accordance with the provided suggestions" is a robot capable of actually carrying out the cleaning method suggested based on the image analysis results.

[0161] "Means for comparing images before and after cleaning to visually confirm the cleaning effect" refers to a system or method that compares images or videos before and after cleaning and visually shows the difference to the user.

[0162] The following system is designed as an embodiment of the present invention. The system is configured to use a robot to efficiently clean a physical space such as a factory and to visually confirm the cleaning results. The system mainly consists of the following components:

[0163] Data Acquisition Module

[0164] The robot is equipped with a camera that takes photos and videos of the factory. The robot takes images and videos as it moves around the physical space and sends them to a server. The captured data is compressed and formatted within the robot before being sent to the server.

[0165] Image Analysis Module

[0166] The server receives the transmitted image and video data and analyzes it using the image analysis module. This analysis mainly uses deep learning-based algorithms such as TENSORFLOW (registered trademark) and OpenCV. As a result of the analysis, dirt and improperly placed objects within the factory are identified. The analysis results are stored in a database and used for subsequent processing.

[0167] Cleaning suggestion module

[0168] Based on the results of the image analysis module, the server proposes efficient cleaning methods and necessary tools. These proposals include specific steps, such as "If oil stains are detected, clean using a specific detergent and cloth." Links to purchase the necessary tools are also provided. This information is then sent to the robot and the user's device.

[0169] Virtual Reality (VR) Module

[0170] The server uses virtual reality technology to simulate the state of the physical space before and after cleaning. In particular, it predicts the state after cleaning and allows users to visually confirm the cleaning results. This virtual reality environment can be experienced by users through VR goggles or a corresponding application.

[0171] Specific examples

[0172] For example, if oil stains are found on the floor of a factory, the robot takes a photo of the area and sends the image to the server. The server then uses an image analysis module to detect the oil stains and a cleaning suggestion module to suggest cleaning using a specified detergent and cloth. This suggestion is sent to the robot, which then carries out the cleaning. After cleaning, the robot takes another image, and a virtual reality module compares the before and after cleaning, allowing the user to check the results using VR goggles.

[0173] Prompt Sentence Examples

[0174] "Analyze these images to identify dirt and unwanted materials in the factory and suggest effective cleaning methods."

[0175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0176] Step 1:

[0177] As the robot moves around the factory, it uses a camera to take photos and videos. In this step, the robot acquires data from the actual environment. The input is visual data in the physical space, and the output is compressed and formatted image and video data.

[0178] Step 2:

[0179] The terminal receives the data captured by the robot and sends it to the server. The input is compressed and format-converted data, and the output is image and video data uploaded to the server.

[0180] Step 3:

[0181] The data received by the server is input into the image analysis module for analysis. The input is image and video data uploaded to the server, and the output is information on dirt and object placement in the physical space. Specifically, data calculations using a deep learning model detect dirt and identify improper object placement.

[0182] Step 4:

[0183] The server then proposes cleaning areas and methods based on the results of image analysis. The input is dirt and placement information obtained from the image analysis module, and the output is a list of specific cleaning procedures and necessary tools. Specific actions such as "if oil stains are detected, clean using a specific detergent and cloth" are generated.

[0184] Step 5:

[0185] The terminal sends the proposed cleaning content along with a prompt to the robot, which then actually performs the cleaning. The input is the cleaning proposal information from the server, and the output is the performed cleaning task. The robot cleans the physical space using the specified cleaning tools.

[0186] Step 6:

[0187] After cleaning, the robot takes images and videos again and sends them to the server. The input is visual data of the physical space after cleaning, and the output is image and video data uploaded to the server.

[0188] Step 7:

[0189] The server compares image data before and after cleaning and generates the results in a virtual reality environment. The input is image data before and after cleaning, and the output is comparative information before and after cleaning. Specifically, the before-cleaning and after-cleaning states are displayed side by side, visually highlighting the differences.

[0190] Step 8:

[0191] The device provides a virtual reality environment to the user, who then uses VR goggles to check the cleaning results. The input is comparison information from the server before and after cleaning, and the output is the user's visual confirmation experience. The user can wear the VR goggles and check the before and after cleaning status in 3D.

[0192] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0193] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention helps a user understand the current state of the room, implement effective cleaning procedures, recognize emotions during the cleaning process, and virtually experience a sense of accomplishment. Specific embodiments of the present invention are described below.

[0194] System configuration

[0195] The system is broadly divided into five main parts:

[0196] 1. Data Acquisition Module

[0197] 2. Image Analysis Module

[0198] 3. Cleaning suggestion module

[0199] 4. Virtual Reality (VR) Module

[0200] 5. Emotion Recognition Module

[0201] Data Acquisition Module

[0202] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[0203] The device compresses the captured data and prepares it for transmission to the server.

[0204] Image Analysis Module

[0205] The server passes the received data to an image analysis module, which processes it to identify dirty or cluttered areas in the room.

[0206] Image analysis uses deep learning algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[0207] Cleaning suggestion module

[0208] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[0209] The suggestions include specific cleaning procedures and essential items.

[0210] The device notifies the user of this information and displays it in a concise, easy-to-understand format.

[0211] Virtual Reality (VR) Module

[0212] Run the VR module to simulate the state of the room after the server has finished cleaning.

[0213] Data before and after cleaning is compared, allowing users to visually experience the difference in a VR environment.

[0214] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[0215] Emotion Recognition Module

[0216] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[0217] The server analyzes the acquired data and recognizes the user's emotions in real time.

[0218] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[0219] The terminal notifies the user of these advices.

[0220] Specific examples

[0221] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[0222] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[0223] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[0224] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[0225] In this way, the present invention achieves improved cleaning efficiency and motivation through comprehensive support, including emotion recognition technology.

[0226] The processing flow will be explained below.

[0227] Step 1:

[0228] The user launches a dedicated application and takes photos and videos of the room.

[0229] Step 2:

[0230] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[0231] Step 3:

[0232] The terminal transmits the prepared data to the server.

[0233] Step 4:

[0234] The server passes the received data to the image analysis module.

[0235] Step 5:

[0236] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[0237] Step 6:

[0238] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[0239] Step 7:

[0240] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[0241] Step 8:

[0242] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[0243] Step 9:

[0244] The user starts cleaning the room according to the presented cleaning method.

[0245] Step 10:

[0246] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[0247] Step 11:

[0248] The server uses an emotion recognition module to analyze the acquired data and recognize the user's emotions in real time.

[0249] Step 12:

[0250] The server generates appropriate advice or encouraging messages according to the user's emotional state and sends them to the terminal.

[0251] Step 13:

[0252] The device notifies the user of advice and messages received and displays suggestions to increase motivation.

[0253] Step 14:

[0254] After the user has completed cleaning, they can take photos or videos again and upload the new data to the device.

[0255] Step 15:

[0256] The device sends the post-cleaning data to the server.

[0257] Step 16:

[0258] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[0259] Step 17:

[0260] The virtual reality data generated by the server is sent to the terminal.

[0261] Step 18:

[0262] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[0263] Step 19:

[0264] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[0265] Example 2

[0266] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0267] In recent years, cleaning a room has become a burden for individuals with busy lifestyles. Maintaining motivation and choosing an efficient cleaning method are particularly difficult. Furthermore, conventional cleaning methods offer limited means for visually confirming the results of cleaning, making it difficult to achieve a sense of accomplishment. Given this background, there is a demand for a system that allows users to efficiently clean a room, maintain their motivation, and visually experience the results of cleaning.

[0268] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring images or videos of the room taken by the user; means for compressing the acquired images or videos and transmitting them to the server; means for analyzing the images or videos received by the server and identifying stains and the location of objects in the room using a deep learning algorithm; means for suggesting areas to be cleaned and cleaning methods based on the identified stains and locations; means for notifying and displaying the suggestions to the user in an easy-to-understand format; means for generating a post-cleaning state of the room using virtual reality technology and visually presenting it to the user; means for acquiring the user's facial expression, voice, and behavior data, analyzing the data, recognizing emotions, and providing a cleaning method according to the user's emotional state and advice for improving motivation based on the recognition results; and means for notifying the user of the advice based on the acquired emotional data. This not only improves cleaning efficiency but also maintains the user's motivation and enhances the sense of accomplishment after cleaning.

[0269] "User" refers to an individual who uses the system to improve cleaning efficiency and check the effectiveness.

[0270] A "terminal" refers to a smartphone or dedicated device used by a user, which captures images and videos of the room and sends them to a server.

[0271] "Server" refers to a central processing unit that receives data, analyzes it, makes recommendations, and generates the virtual reality environment.

[0272] "Image analysis" refers to the process of using deep learning algorithms to identify dirt and object placement in a room.

[0273] A "deep learning algorithm" is a type of machine learning used to perform complex image analysis and automatically detect specific objects or conditions.

[0274] "Cleaning suggestions" refers to the process of suggesting effective cleaning locations and cleaning methods to the user based on the analyzed data.

[0275] "Virtual reality technology" refers to a simulation technology that allows users to visually experience the state of a room before and after cleaning.

[0276] "Emotion recognition" refers to the process of analyzing a user's facial, vocal, and behavioral data to identify their emotional state.

[0277] "Advice" refers to guidance and recommendations on cleaning methods and motivation provided to users based on emotion recognition results.

[0278] "Compression" refers to a data processing technique for reducing the volume of captured image or video data.

[0279] "Notification" refers to the process by which a device communicates suggestions or advice to a user.

[0280] MODE FOR CARRYING OUT THE INVENTION

[0281] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention supports a user in understanding the current state of the room, implementing effective cleaning procedures, and virtually experiencing a sense of accomplishment while recognizing emotions during the cleaning process. Specific embodiments of the present invention are described below.

[0282] The system consists of five main parts:

[0283] 1. Data Acquisition Module

[0284] 2. Image Analysis Module

[0285] 3. Cleaning suggestion module

[0286] 4. Virtual Reality (VR) Module

[0287] 5. Emotion Recognition Module

[0288] Data Acquisition Module

[0289] User: Take photos and videos of the room using a smartphone or dedicated device application.

[0290] Terminal: Compresses the captured data and prepares it for transmission to the server.

[0291] Specifically, the user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the device records the detailed state of the room. The device then uses compression algorithms such as JPEG or H.264 to reduce the data size and sends it to the server.

[0292] Image Analysis Module

[0293] Server: Passes the received data to the image analysis module, which uses deep learning algorithms to identify dirty and cluttered areas in the room.

[0294] The server receives the data via HTTP or WebSocket, saves it to disk, and then forwards it to the analysis module, which uses machine learning frameworks such as TensorFlow and PyTorch to analyze it. The model detects dust particles and blemishes in the image and outputs their location information.

[0295] Cleaning suggestion module

[0296] Server: Based on the results from the image analysis module, it proposes effective cleaning methods.

[0297] Terminal: Informs the user of the suggestions and displays them in a concise and easy-to-understand format.

[0298] Specifically, the server evaluates the analysis results and pulls out appropriate cleaning methods from the database. For example, it generates suggestions such as "A specific detergent and brush are needed to remove stains from the carpet." These suggestions and links are then notified and displayed to the user via their device.

[0299] Virtual Reality (VR) Module

[0300] Server: Runs the VR module to simulate the state of the room after cleaning is completed.

[0301] Device: Helps users experience virtual environments using VR goggles and compatible apps.

[0302] By comparing image data before and after cleaning, a 3D model of the clean room is generated. This model is sent to the device as a data file for the VR app, and the user can visually experience the state of the room after cleaning by wearing VR goggles.

[0303] Emotion Recognition Module

[0304] Terminal: Captures the user's facial expression, voice, and behavioral data and sends it to the emotion recognition module.

[0305] Server: Analyzes the acquired data and recognizes the user's emotions in real time. Based on the emotion recognition results, it provides advice on cleaning methods and motivation that correspond to the user's emotional state.

[0306] Terminal: Informs the user of these suggestions.

[0307] Specifically, the device collects the user's facial expressions and voice through a camera and microphone and transmits them to a server in real time. An emotion recognition algorithm uses Python's OpenCV and deep learning to classify emotions, and if a user shows a tired expression, it generates advice such as "Take a break" or "The next cleaning step is light work."

[0308] Specific examples

[0309] For example, a user can take a photo of their living room and upload it to their device through the application. This data is then sent to the server and fed into the image analysis module, which can then detect a coffee stain on the carpet.

[0310] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[0311] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[0312] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[0313] Recommended prompt sentence examples

[0314] "Take a photo of your living room and upload it to the app. If it finds a coffee stain on your carpet, it will suggest a specific cleaner and brush to use."

[0315] In this way, the present invention achieves increased cleaning efficiency and motivation through comprehensive support including emotion recognition technology.

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] Step 1: Data Acquisition

[0318] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[0319] Input: Current state of the room (photo or video)

[0320] Data processing: Capture images or videos using the camera function of your smartphone or device.

[0321] Output: Captured photo or video data

[0322] Specific operation: The user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the detailed condition of the room is recorded.

[0323] Step 2: Compress data and prepare for transmission

[0324] The device compresses the captured data and prepares it for transmission to the server.

[0325] Input: Photo or video data taken

[0326] Data processing: Compressing the data using compression algorithms such as JPEG or H.264.

[0327] Output: Compressed photo or video data

[0328] Specific operation: The device compresses the captured data in the specified format and prepares to send it to the server.

[0329] Step 3: Receiving data

[0330] The server receives the compressed data sent from the terminal.

[0331] Input: Compressed photo or video data

[0332] Data processing: Receive data using HTTP or WebSocket.

[0333] Output: Received photo or video data

[0334] Specific operation: The server receives and stores data at the specified port or address.

[0335] Step 4: Image analysis

[0336] The server passes the received data to an image analysis module, which processes it to identify areas of dirt or clutter.

[0337] Input: Received photo or video data

[0338] Data Computing: Uses deep learning algorithms to detect dirt and clutter.

[0339] Output: Identification of dirty and cluttered areas (including location information)

[0340] Specific operation: The server uses machine learning frameworks such as TensorFlow and PyTorch, and the model detects dust, blemishes, etc. in the image and outputs their location information.

[0341] Step 5: Generate cleaning suggestions

[0342] The server suggests effective cleaning methods based on the results from the image analysis module.

[0343] Input: Identification of dirty and cluttered areas

[0344] Data calculation: Evaluate the analysis results and extract the appropriate cleaning method from the database.

[0345] Output: Cleaning suggestions

[0346] Specific behavior: The server retrieves cleaning methods from a database and generates suggestions, such as "a specific detergent is needed to remove stains from the carpet."

[0347] Step 6: Proposal Notification

[0348] The device will notify the user of the suggestions and display them in an easy-to-understand format.

[0349] Input: Cleaning suggestions

[0350] Data processing: Convert the proposal content into a notification format.

[0351] Output: User notification

[0352] What it does: The device will communicate suggestions to the user via push notifications or in-app displays, such as "Use the following items to remove stains from your carpet."

[0353] Step 7: Acquire data after cleaning

[0354] After the user has finished cleaning, they take photos and videos again and send them to the server.

[0355] Input: Photo or video data after cleaning

[0356] Data processing: Images or videos are taken using the camera function of a smartphone or device and compressed.

[0357] Output: Compressed cleaned photo or video data

[0358] Specific operation: The user takes a photo of the room after cleaning and sends it back to the server.

[0359] Step 8: Virtual Reality (VR) Generation

[0360] The server compares the data before and after cleaning and generates a virtual reality representation of the state of the room after cleaning.

[0361] Input: Photos or video data before and after cleaning

[0362] Data calculation: Compare image data before and after cleaning to generate a virtual reality environment.

[0363] Output: Virtual reality environment data

[0364] Specific operation: Based on the comparison results, the server generates a 3D model of the clean room and sends it to the terminal as VR data.

[0365] Step 9: Experience the virtual environment

[0366] The device supports users in experiencing virtual environments using VR goggles and compatible apps.

[0367] Input: Virtual reality environment data

[0368] Data processing: Converting VR data into a format that can be displayed on VR goggles.

[0369] Output: Virtual reality experience to the user

[0370] Specific operation: The device reads the transmitted VR data and displays it to the user through the VR goggles, allowing the user to visually experience the state before and after cleaning in a virtual reality environment.

[0371] Step 10: Acquire emotion data

[0372] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[0373] Input: User's facial expression, voice, and behavior data

[0374] Data processing: Collect data using cameras and microphones.

[0375] Output: Obtained emotion data

[0376] Specific operation: The device uses a camera and microphone to record the user's facial expressions, voice, and actions in real time and transmits them to the server.

[0377] Step 11: Sentiment Analysis

[0378] The server analyzes the acquired emotional data and recognizes the user's emotions in real time.

[0379] Input: Acquired emotion data

[0380] Data Computing: Emotion recognition using deep learning algorithms.

[0381] Output: User's emotional state

[0382] Specific behavior: Emotion recognition algorithms analyze facial, voice, and behavioral data to classify the user's emotional state.

[0383] Step 12: Emotionally adaptive advice

[0384] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[0385] Input: User's emotional state

[0386] Data calculation: Generate appropriate advice according to emotional state.

[0387] Output: Personalized advice

[0388] Specific actions: For example, if the user is tired, advice such as "take a break" or "the next task is light work" is generated.

[0389] Step 13: Advice Notification

[0390] The terminal notifies the user of the advice.

[0391] Input: Personalized advice

[0392] Data processing: Convert the advice content into a notification format.

[0393] Output: Advice notice to user

[0394] Specific behavior: The device will communicate advice to the user via push notifications or in-app displays.

[0395] (Application example 2)

[0396] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] Conventional cleaning systems not only detect dirt and obstacles, but also lack the ability to propose efficient cleaning methods based on the detection results, or support for maintaining the motivation of cleaning staff. Effective cleaning management is also difficult because there are limited ways to visually check the status after cleaning. Especially in industrial environments, rapid detection and processing of dirt and obstacles is important, and methods for reducing fatigue and stress for cleaning staff are also required.

[0398] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images or videos of a dirty environment, means for analyzing the acquired images or videos to detect dirt and obstacles in the environment, means for proposing cleaning areas and cleaning methods based on the detected dirt and obstacles, means for generating a post-cleaning state using virtual reality technology, means for providing the generated virtual reality environment to the user, means for analyzing the facial expressions and voice of the cleaning staff and recognizing their emotions, and means for providing the user with advice on cleaning methods and motivation based on the recognized emotions. This enables effective cleaning management, reducing the burden on the staff and improving the efficiency and accuracy of cleaning.

[0399] "Images or videos of a dirty environment" means visual information, including dirt or obstacles present in a particular location or object, obtained using a camera or sensor.

[0400] "Means for analyzing captured images or videos and detecting environmental dirt or obstacles" refers to technology that uses deep learning algorithms and image processing technology to identify specific dirt or obstacles from captured visual information.

[0401] "Means for suggesting cleaning areas and cleaning methods" refers to technology that suggests optimal cleaning procedures and necessary tools to the user based on detected dirt and obstacles.

[0402] "Means for generating the post-cleaning state using virtual reality technology" refers to technology that compares the state before and after cleaning and reproduces the ideal post-cleaning environment in a virtual reality environment.

[0403] "Means for providing a generated virtual reality environment to a user" refers to technology that allows a user to visually experience the generated virtual environment using virtual reality goggles or other display devices.

[0404] "Means for analyzing the facial expressions and voice of cleaning staff and recognizing their emotions" refers to technology that acquires the facial expressions and speech of cleaning staff and analyzes that data to identify the emotional state of the staff.

[0405] "Means for providing users with advice on cleaning methods and motivation based on recognized emotions" refers to technology that provides users with appropriate changes to their cleaning methods and encouraging messages based on analyzed emotional data.

[0406] The system of the present invention aims to improve the efficiency of cleaning work in a factory environment and to increase the motivation of the person in charge. Specific embodiments of the present invention will be described below.

[0407] System Configuration

[0408] This system is composed of various modules, each of which is explained below.

[0409] Data Acquisition Module

[0410] The server uses a robot equipped with a camera to capture images or video of the environment to be cleaned. The robot collects visual data as it patrols the factory, which is then compressed and prepared for transmission to the server.

[0411] Image Analysis Module

[0412] The server analyzes the captured image or video data using deep learning algorithms, such as TensorFlow and Keras software, to identify dirt and obstacles within the factory.

[0413] Cleaning Suggestion Module

[0414] The server then proposes optimal cleaning methods based on the image analysis results, including specific instructions on the cleaning agents, brushes, and other cleaning equipment to be used. The proposal results are then sent to the user via their device.

[0415] Virtual Reality (VR) Module

[0416] The server uses virtual reality technology to generate the post-cleaning environment. It compares data before and after cleaning and simulates the effect in the VR environment. Users can experience the generated virtual environment using VR goggles.

[0417] Emotion Recognition Module

[0418] The device captures the facial expressions and voice of the cleaner and sends them to an emotion recognition module. The server then analyzes the facial and voice data using deep learning algorithms to identify the cleaner's emotional state. The software used includes a Keras model that supports emotion recognition.

[0419] Based on the emotion recognition results, the server can suggest changes to cleaning methods or provide encouraging messages to the user. For example, if fatigue is detected, the server can provide advice such as "Take a break" or "Try a lighter task next time."

[0420] Specific examples

[0421] In a specific scenario, a robot detects dirt in a factory and the server suggests appropriate cleaning methods. After the cleaning is complete, the user can visually check the cleaning results using VR goggles. Furthermore, the server analyzes the operator's facial expressions and suggests timely breaks if fatigue is detected.

[0422] Prompt Sentence Examples

[0423] I am developing a highly automated factory cleaning robot. This robot uses cameras to detect dirt in the factory and suggests the best cleaning method. It also simulates the cleaning state using VR goggles and analyzes the operator's emotions to provide real-time feedback. This system incorporates image analysis, emotion recognition, and virtual reality technology using deep learning algorithms. I would like your detailed feedback and suggestions for improvement based on this project.

[0424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0425] Step 1:

[0426] The user starts the robot to acquire environmental data. Specifically, the camera mounted on the robot is used to acquire images and videos of the factory interior. Libraries such as OpenCV are used to acquire this data. The input is real-time video of the factory interior, and the output is a file that stores this in image or video data format.

[0427] Step 2:

[0428] The device compresses the acquired data and prepares it for transmission to the server. Specifically, it converts image and video files into a format suitable for communication and compresses them. The input is the image and video data generated in step 1, and the output is a compressed data file.

[0429] Step 3:

[0430] The server receives the compressed data and passes it to the image analysis module. The server decodes the received data and analyzes it using deep learning algorithms. This processing uses TensorFlow and Keras. The input is the compressed data file, and the output is information about detected dirt and obstacles.

[0431] Step 4:

[0432] The server proposes the optimal cleaning method based on the results of image analysis. Based on the analyzed data, it recommends the cleaning tools, detergents, and cleaning procedures to be used. This information is generated by the cleaning suggestion module. The input is the image analysis results, and the output is a detailed cleaning procedure proposal.

[0433] Step 5:

[0434] The device notifies the user of the suggested cleaning method by displaying or reading out the required tools and cleaning steps to the user using the device's display or audio output. The input is the cleaning suggestion obtained from the server, and the output is a visual or audio notification to the user.

[0435] Step 6:

[0436] The user performs the cleaning as suggested and takes photos and videos during the process. The device sends these data to the server. The input is the image and video data after cleaning, and the output is compressed data sent to the server.

[0437] Step 7:

[0438] The server compares the data before and after cleaning and uses virtual reality technology to generate an ideal post-cleaning environment, allowing users to visually experience the cleaning results. The input is the data before and after cleaning, and the output is the generated virtual environment data.

[0439] Step 8:

[0440] The device captures the facial expressions and voice of the cleaner and sends them to the emotion recognition module. The facial expression and voice data are analyzed using emotion recognition algorithms (such as Keras). The input is the cleaner's facial expression and voice data, and the output is the analyzed emotional state.

[0441] Step 9:

[0442] The server then provides the user with advice on how to change their cleaning method or improve their motivation based on the analyzed emotional data. Specifically, it suggests taking a break as needed or showing the next cleaning step. The input is the analyzed emotional state, and the output is advice for the user.

[0443] Step 10:

[0444] The user receives advice via the terminal and continues cleaning. In this way, the system achieves effective cleaning management and reduces the burden on the person in charge.

[0445] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0446] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0447] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0448] [Second embodiment]

[0449] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0450] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0451] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0452] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0453] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0454] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0455] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0456] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0457] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0458] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0459] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0460] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0461] The present invention provides a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment. Specific embodiments of the present invention are described below.

[0462] System configuration

[0463] The system consists of four main parts:

[0464] 1. Data Acquisition Module

[0465] 2. Image Analysis Module

[0466] 3. Cleaning suggestion module

[0467] 4. Virtual Reality (VR) Module

[0468] Data Acquisition Module

[0469] Users can take photos and videos of the room using an application on their smartphone or a dedicated device.

[0470] The device prepares the captured data for transmission to the server, specifically by compressing the data, converting the format, and adding metadata.

[0471] Image Analysis Module

[0472] The server inputs the received data into an image analysis module and processes it to identify dirty or cluttered areas in the room.

[0473] Image analysis uses deep learning-based algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[0474] The identified information is organized and stored in a database.

[0475] Cleaning suggestion module

[0476] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[0477] The suggestions include specific cleaning procedures and essential items (e.g., vacuum cleaner, detergent, brush).

[0478] The device notifies the user of this information and displays it in a concise and easy-to-understand format.

[0479] Virtual Reality (VR) Module

[0480] The server runs a VR module to simulate the state of the room after the user has completed cleaning.

[0481] By comparing data before and after cleaning, users can visually experience the difference in a VR environment, giving them a sense of accomplishment from cleaning.

[0482] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[0483] Specific examples

[0484] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[0485] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[0486] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the cleaned up state. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[0487] In this way, the present invention aims to achieve both improved cleaning efficiency and motivation.

[0488] The processing flow will be explained below.

[0489] Step 1:

[0490] The user launches a dedicated application and takes photos and videos of the room.

[0491] Step 2:

[0492] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[0493] Step 3:

[0494] The terminal transmits the prepared data to the server.

[0495] Step 4:

[0496] The server passes the received data to the image analysis module.

[0497] Step 5:

[0498] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[0499] Step 6:

[0500] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[0501] Step 7:

[0502] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[0503] Step 8:

[0504] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[0505] Step 9:

[0506] The user cleans the room using the cleaning method presented as a reference.

[0507] Step 10:

[0508] The user takes a picture of the state of the room after cleaning again and uploads the new photo or video data to the terminal.

[0509] Step 11:

[0510] The device sends the post-cleaning data to the server.

[0511] Step 12:

[0512] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[0513] Step 13:

[0514] The virtual reality data generated by the server is sent to the terminal.

[0515] Step 14:

[0516] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[0517] Step 15:

[0518] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[0519] Example 1

[0520] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0521] Cleaning a room is labor-intensive and troublesome for many people. It is also difficult to determine which areas need cleaning and the best cleaning methods, making it extremely difficult to clean effectively. Furthermore, the effectiveness of cleaning cannot be reliably confirmed, which can lead to a decline in motivation. There is a need for a system that can solve these issues, improve cleaning efficiency, and increase motivation.

[0522] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0523] In this invention, the server includes means for identifying dirty and messy areas in a room using an image analysis model, means for storing the analysis results in a database, means for proposing cleaning methods based on the analysis results, and means for comparing data before and after cleaning and generating the state of the room after cleaning using virtual reality technology, thereby enabling the user to accurately grasp the current state of the room, practice effective cleaning methods, and virtually experience the sense of accomplishment after cleaning.

[0524] "User" refers to an individual or group that uses this system to clean a room.

[0525] "Terminal" refers to the device that the user uses to take pictures and videos of the room and send them to the server. Specifically, this applies to smartphones and dedicated terminals.

[0526] "Server" refers to a central control unit for analyzing, storing, and making suggestions about data sent by users.

[0527] "Image analysis models" refer to programs that use deep learning algorithms to identify dirty or cluttered areas in a room, such as YOLO and Mask R-CNN.

[0528] A "database" refers to a system for systematically storing and searching analysis results and related information.

[0529] "Cleaning method" refers to information on the optimal cleaning procedures and cleaning tools to be used, identified based on the analysis results.

[0530] "Virtual reality technology" refers to technology that allows users to visually experience the state of a room after cleaning, specifically 3D modeling software and VR devices.

[0531] "Data before and after cleaning" refers to images and videos showing the state of the room before and after the user cleans.

[0532] "Virtual reality environment" refers to a virtual space generated by the server that visually reproduces the state of a room after cleaning.

[0533] The present invention is a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment.

[0534] 1. Specific data acquisition procedures

[0535] Users use their smartphones or dedicated devices to take photos and videos of their rooms. Specifically, they launch a smartphone app and take photos of various parts of the room. The device receives this data and organizes it by compressing and converting the format. For example, it converts it to JPEG format and adds metadata (date and time of shooting, location information).

[0536] 2. Specific operations for data transmission

[0537] The device sends the data the user has acquired to the server using Wi-Fi or mobile data, encrypting the data using security protocols such as SSL.

[0538] 3. Specific procedures for image analysis

[0539] The server inputs the received data into an image analysis module and uses deep learning-based image analysis models (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. The analysis results are organized in JSON format or similar and stored in a database.

[0540] 4. Specific cleaning operations

[0541] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate effective cleaning methods based on the results of image analysis. The recommendations include specific cleaning steps and information on the tools needed. For example, it suggests that a specific detergent and brush are needed to remove stains from a carpet.

[0542] 5. Specific operation of VR simulation

[0543] After cleaning, the user takes photos and videos of the room again and uploads them to the server via their device. The server compares the data before and after cleaning and uses 3D modeling software (e.g., Blender) to generate a VR environment of the room after cleaning.

[0544] 6. Specific operations for VR experiences

[0545] The device connects to a VR device (e.g., Oculus Rift or HTC Vive) and downloads the generated VR environment data. Users can then wear VR goggles and visually experience the state of the room before and after cleaning, giving them a sense of accomplishment from cleaning.

[0546] Specific examples

[0547] For example, a user takes a photo of their living room and uploads it to their device through an application. This data is compressed, converted, and sent to a server. After analyzing the image, the server detects that there is a coffee stain on the carpet. The server then suggests using a specific detergent and brush to remove the stain from the carpet and provides a link to purchase the necessary tools. The device notifies the user, and the user completes the cleaning. By taking another photo and uploading it using the same steps, the server compares the state before and after cleaning and generates a VR environment. The device then plays this on a VR device, allowing the user to experience the virtual environment.

[0548] Prompt Sentence Examples

[0549] Below are some example prompts to input the behavior of this system into the generative AI model.

[0550] "Please explain in detail how this system uses image analysis and virtual reality technology to efficiently clean rooms. A user takes a photo of the room with their smartphone and sends it to a server via their device. The server then analyzes the image, detects dirt and clutter, and makes cleaning suggestions. After cleaning is complete, a VR module is used to compare the state before and after cleaning, allowing the user to experience the state after cleaning in a virtual environment."

[0551] This system improves the efficiency of the cleaning process while also increasing user motivation.

[0552] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0553] Step 1:

[0554] The user takes photos and videos of the room using a smartphone or dedicated device. Specifically, the user starts the app and takes multiple photos of various parts of the room from various angles. The input is the images and videos of the room, and the output is this data.

[0555] Step 2:

[0556] The device compresses the captured data, converts the format, and adds metadata. The device converts the data to JPEG or MPEG format, and adds the date and time of the capture and location information as metadata. The input is the captured image or video, and the output is the compressed and converted data.

[0557] Step 3:

[0558] The device sends data to the server. The compressed and converted data is then sent to the server using Wi-Fi or mobile data. An encryption protocol such as SSL is used during transmission. The input is the compressed and converted data, and the output is the data sent to the server.

[0559] Step 4:

[0560] The server inputs the received data into an image analysis module, which uses a deep learning-based image analysis model (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. In the process, the server identifies dirt, stains, improper placement, etc. The input is the data sent to the server, and the output is the analysis results.

[0561] Step 5:

[0562] The server saves the analysis results in a database. The analysis results are organized in JSON format or similar and stored in the database in a format that allows for efficient search and reference. The input is the analysis results, and the output is the data saved in the database.

[0563] Step 6:

[0564] The server proposes cleaning methods based on the analysis results. Using a generative AI model (e.g., OpenAI GPT-4), it generates recommendations regarding optimal cleaning procedures and cleaning tools to use. The input is the analysis results, and the output is the recommendations.

[0565] Step 7:

[0566] The terminal notifies the user of the suggestions from the server. The terminal displays the suggested cleaning procedures and necessary tools in an easy-to-understand format. The input is the suggestions, and the output is the information notified to the user.

[0567] Step 8:

[0568] The user actually cleans based on the suggestions. The user uses the necessary tools and follows the suggested procedure to clean. The input is the suggestions and cleaning tools, and the output is the clean state after cleaning.

[0569] Step 9:

[0570] The user then takes photos or videos of the state of the room after cleaning and uploads them to the server via the device in the same way. The input is the post-cleaning images or videos, and the output is the post-cleaning data sent to the server.

[0571] Step 10:

[0572] The server compares the data before and after cleaning and generates the state of the room after cleaning using virtual reality technology. The server uses 3D modeling software (e.g., Blender) to visualize the difference between the state before and after cleaning. The input is the data before and after cleaning, and the output is the generated virtual reality environment.

[0573] Step 11:

[0574] The device provides the generated virtual reality environment to the user. The device works in conjunction with a VR device (e.g., Oculus Rift, HTC Vive) to allow the user to experience the virtual environment. The input is the generated virtual reality environment, and the output is the virtual experience provided to the user.

[0575] (Application example 1)

[0576] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0577] Cleaning work in factories and other physical spaces is often manual and inefficient. Particularly in large factories, it is difficult to identify dirty or messy areas and clean them efficiently, increasing the burden on workers. Furthermore, there is no way to immediately check the results of cleaning, making it difficult to visualize the effectiveness of the work. To solve these problems, a system is needed that automates cleaning work and makes it easy to check the results.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0579] In this invention, the server includes means for analyzing images or videos to detect dirt and object placement in the physical space, means for proposing areas to be cleaned and cleaning methods based on the detected dirt and placement, and means for generating the state of the physical space after cleaning using virtual reality technology. This allows for efficient automation of cleaning work in physical spaces such as factories, and enables users to immediately visually confirm the cleaning results.

[0580] An "image or video capturing means" is a device or method for capturing images or video within a physical space.

[0581] "Means for analyzing images or videos" refers to an algorithm or device for detecting the placement of dirt or objects using captured images or videos.

[0582] A "means for detecting dirt or placement" is a system or algorithm that identifies the presence of dirt or the placement of an object based on analyzed images or video.

[0583] The "means for suggesting cleaning locations and cleaning methods" is a system or method that suggests specific locations to be cleaned and appropriate cleaning methods based on detected dirt and location information.

[0584] "Means for generating the state of a physical space after cleaning using virtual reality technology" refers to a system or technology that uses virtual reality technology to simulate and visually present what a physical space will look like after cleaning.

[0585] The "means for providing a generated virtual reality environment to a user" refers to a device or platform that allows a user to experience a virtual reality environment.

[0586] "Means for a robot to capture images or videos in a physical space and transmit them to a server" refers to a system or method for a robot to acquire images or videos using a photographing device and transmit the data to a server.

[0587] The "robot means for carrying out cleaning in accordance with the provided suggestions" is a robot capable of actually carrying out the cleaning method suggested based on the image analysis results.

[0588] "Means for comparing images before and after cleaning to visually confirm the cleaning effect" refers to a system or method that compares images or videos before and after cleaning and visually shows the difference to the user.

[0589] The following system is designed as an embodiment of the present invention. The system is configured to use a robot to efficiently clean a physical space such as a factory and to visually confirm the cleaning results. The system mainly consists of the following components:

[0590] Data Acquisition Module

[0591] The robot is equipped with a camera that takes photos and videos of the factory. The robot takes images and videos as it moves around the physical space and sends them to a server. The captured data is compressed and formatted within the robot before being sent to the server.

[0592] Image Analysis Module

[0593] The server receives the transmitted image and video data and analyzes it using the image analysis module. This analysis mainly uses deep learning-based algorithms such as TensorFlow and OpenCV. As a result of the analysis, dirt and improperly placed objects within the factory are identified. The analysis results are stored in a database and used for subsequent processing.

[0594] Cleaning suggestion module

[0595] Based on the results of the image analysis module, the server proposes efficient cleaning methods and necessary tools. These proposals include specific steps, such as "If oil stains are detected, clean using a specific detergent and cloth." Links to purchase the necessary tools are also provided. This information is then sent to the robot and the user's device.

[0596] Virtual Reality (VR) Module

[0597] The server uses virtual reality technology to simulate the state of the physical space before and after cleaning. In particular, it predicts the state after cleaning and allows users to visually confirm the cleaning results. This virtual reality environment can be experienced by users through VR goggles or a corresponding application.

[0598] Specific examples

[0599] For example, if oil stains are found on the floor of a factory, the robot takes a photo of the area and sends the image to the server. The server then uses an image analysis module to detect the oil stains and a cleaning suggestion module to suggest cleaning using a specified detergent and cloth. This suggestion is sent to the robot, which then carries out the cleaning. After cleaning, the robot takes another image, and a virtual reality module compares the before and after cleaning, allowing the user to check the results using VR goggles.

[0600] Prompt Sentence Examples

[0601] "Analyze these images to identify dirt and unwanted materials in the factory and suggest effective cleaning methods."

[0602] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0603] Step 1:

[0604] As the robot moves around the factory, it uses a camera to take photos and videos. In this step, the robot acquires data from the actual environment. The input is visual data in the physical space, and the output is compressed and formatted image and video data.

[0605] Step 2:

[0606] The terminal receives the data captured by the robot and sends it to the server. The input is compressed and format-converted data, and the output is image and video data uploaded to the server.

[0607] Step 3:

[0608] The data received by the server is input into the image analysis module for analysis. The input is image and video data uploaded to the server, and the output is information on dirt and object placement in the physical space. Specifically, data calculations using a deep learning model detect dirt and identify improper object placement.

[0609] Step 4:

[0610] The server then proposes cleaning areas and methods based on the results of image analysis. The input is dirt and placement information obtained from the image analysis module, and the output is a list of specific cleaning procedures and necessary tools. Specific actions such as "if oil stains are detected, clean using a specific detergent and cloth" are generated.

[0611] Step 5:

[0612] The terminal sends the proposed cleaning content along with a prompt to the robot, which then actually performs the cleaning. The input is the cleaning proposal information from the server, and the output is the performed cleaning task. The robot cleans the physical space using the specified cleaning tools.

[0613] Step 6:

[0614] After cleaning, the robot takes images and videos again and sends them to the server. The input is visual data of the physical space after cleaning, and the output is image and video data uploaded to the server.

[0615] Step 7:

[0616] The server compares image data before and after cleaning and generates the results in a virtual reality environment. The input is image data before and after cleaning, and the output is comparative information before and after cleaning. Specifically, the before-cleaning and after-cleaning states are displayed side by side, visually highlighting the differences.

[0617] Step 8:

[0618] The device provides a virtual reality environment to the user, who then uses VR goggles to check the cleaning results. The input is comparison information from the server before and after cleaning, and the output is the user's visual confirmation experience. The user can wear the VR goggles and check the before and after cleaning status in 3D.

[0619] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0620] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention helps a user understand the current state of the room, implement effective cleaning procedures, recognize emotions during the cleaning process, and virtually experience a sense of accomplishment. Specific embodiments of the present invention are described below.

[0621] System configuration

[0622] The system is broadly divided into five main parts:

[0623] 1. Data Acquisition Module

[0624] 2. Image Analysis Module

[0625] 3. Cleaning suggestion module

[0626] 4. Virtual Reality (VR) Module

[0627] 5. Emotion Recognition Module

[0628] Data Acquisition Module

[0629] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[0630] The device compresses the captured data and prepares it for transmission to the server.

[0631] Image Analysis Module

[0632] The server passes the received data to an image analysis module, which processes it to identify dirty or cluttered areas in the room.

[0633] Image analysis uses deep learning algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[0634] Cleaning suggestion module

[0635] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[0636] The suggestions include specific cleaning procedures and essential items.

[0637] The device notifies the user of this information and displays it in a concise, easy-to-understand format.

[0638] Virtual Reality (VR) Module

[0639] Run the VR module to simulate the state of the room after the server has finished cleaning.

[0640] Data before and after cleaning is compared, allowing users to visually experience the difference in a VR environment.

[0641] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[0642] Emotion Recognition Module

[0643] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[0644] The server analyzes the acquired data and recognizes the user's emotions in real time.

[0645] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[0646] The terminal notifies the user of these advices.

[0647] Specific examples

[0648] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[0649] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[0650] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[0651] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[0652] In this way, the present invention achieves improved cleaning efficiency and motivation through comprehensive support, including emotion recognition technology.

[0653] The processing flow will be explained below.

[0654] Step 1:

[0655] The user launches a dedicated application and takes photos and videos of the room.

[0656] Step 2:

[0657] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[0658] Step 3:

[0659] The terminal transmits the prepared data to the server.

[0660] Step 4:

[0661] The server passes the received data to the image analysis module.

[0662] Step 5:

[0663] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[0664] Step 6:

[0665] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[0666] Step 7:

[0667] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[0668] Step 8:

[0669] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[0670] Step 9:

[0671] The user starts cleaning the room according to the presented cleaning method.

[0672] Step 10:

[0673] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[0674] Step 11:

[0675] The server uses an emotion recognition module to analyze the acquired data and recognize the user's emotions in real time.

[0676] Step 12:

[0677] The server generates appropriate advice or encouraging messages according to the user's emotional state and sends them to the terminal.

[0678] Step 13:

[0679] The device notifies the user of advice and messages received and displays suggestions to increase motivation.

[0680] Step 14:

[0681] After the user has completed cleaning, they can take photos or videos again and upload the new data to the device.

[0682] Step 15:

[0683] The device sends the post-cleaning data to the server.

[0684] Step 16:

[0685] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[0686] Step 17:

[0687] The virtual reality data generated by the server is sent to the terminal.

[0688] Step 18:

[0689] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[0690] Step 19:

[0691] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[0692] Example 2

[0693] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0694] In recent years, cleaning a room has become a burden for individuals with busy lifestyles. Maintaining motivation and choosing an efficient cleaning method are particularly difficult. Furthermore, conventional cleaning methods offer limited means for visually confirming the results of cleaning, making it difficult to achieve a sense of accomplishment. Given this background, there is a demand for a system that allows users to efficiently clean a room, maintain their motivation, and visually experience the results of cleaning.

[0695] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring images or videos of the room taken by the user; means for compressing the acquired images or videos and transmitting them to the server; means for analyzing the images or videos received by the server and identifying stains and the location of objects in the room using a deep learning algorithm; means for suggesting areas to be cleaned and cleaning methods based on the identified stains and locations; means for notifying and displaying the suggestions to the user in an easy-to-understand format; means for generating a post-cleaning state of the room using virtual reality technology and visually presenting it to the user; means for acquiring the user's facial expression, voice, and behavior data, analyzing the data, recognizing emotions, and providing a cleaning method according to the user's emotional state and advice for improving motivation based on the recognition results; and means for notifying the user of the advice based on the acquired emotional data. This not only improves cleaning efficiency but also maintains the user's motivation and enhances the sense of accomplishment after cleaning.

[0696] "User" refers to an individual who uses the system to improve cleaning efficiency and check the effectiveness.

[0697] A "terminal" refers to a smartphone or dedicated device used by a user, which captures images and videos of the room and sends them to a server.

[0698] "Server" refers to a central processing unit that receives data, analyzes it, makes recommendations, and generates the virtual reality environment.

[0699] "Image analysis" refers to the process of using deep learning algorithms to identify dirt and object placement in a room.

[0700] A "deep learning algorithm" is a type of machine learning used to perform complex image analysis and automatically detect specific objects or conditions.

[0701] "Cleaning suggestions" refers to the process of suggesting effective cleaning locations and cleaning methods to the user based on the analyzed data.

[0702] "Virtual reality technology" refers to a simulation technology that allows users to visually experience the state of a room before and after cleaning.

[0703] "Emotion recognition" refers to the process of analyzing a user's facial, vocal, and behavioral data to identify their emotional state.

[0704] "Advice" refers to guidance and recommendations on cleaning methods and motivation provided to users based on emotion recognition results.

[0705] "Compression" refers to a data processing technique for reducing the volume of captured image or video data.

[0706] "Notification" refers to the process by which a device communicates suggestions or advice to a user.

[0707] MODE FOR CARRYING OUT THE INVENTION

[0708] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention supports a user in understanding the current state of the room, implementing effective cleaning procedures, and virtually experiencing a sense of accomplishment while recognizing emotions during the cleaning process. Specific embodiments of the present invention are described below.

[0709] The system consists of five main parts:

[0710] 1. Data Acquisition Module

[0711] 2. Image Analysis Module

[0712] 3. Cleaning suggestion module

[0713] 4. Virtual Reality (VR) Module

[0714] 5. Emotion Recognition Module

[0715] Data Acquisition Module

[0716] User: Take photos and videos of the room using a smartphone or dedicated device application.

[0717] Terminal: Compresses the captured data and prepares it for transmission to the server.

[0718] Specifically, the user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the device records the detailed state of the room. The device then uses compression algorithms such as JPEG or H.264 to reduce the data size and sends it to the server.

[0719] Image Analysis Module

[0720] Server: Passes the received data to the image analysis module, which uses deep learning algorithms to identify dirty and cluttered areas in the room.

[0721] The server receives the data via HTTP or WebSocket, saves it to disk, and then forwards it to the analysis module, which uses machine learning frameworks such as TensorFlow and PyTorch to analyze it. The model detects dust particles and blemishes in the image and outputs their location information.

[0722] Cleaning suggestion module

[0723] Server: Based on the results from the image analysis module, it proposes effective cleaning methods.

[0724] Terminal: Informs the user of the suggestions and displays them in a concise and easy-to-understand format.

[0725] Specifically, the server evaluates the analysis results and pulls out appropriate cleaning methods from the database. For example, it generates suggestions such as "A specific detergent and brush are needed to remove stains from the carpet." These suggestions and links are then notified and displayed to the user via their device.

[0726] Virtual Reality (VR) Module

[0727] Server: Runs the VR module to simulate the state of the room after cleaning is completed.

[0728] Device: Helps users experience virtual environments using VR goggles and compatible apps.

[0729] By comparing image data before and after cleaning, a 3D model of the clean room is generated. This model is sent to the device as a data file for the VR app, and the user can visually experience the state of the room after cleaning by wearing VR goggles.

[0730] Emotion Recognition Module

[0731] Terminal: Captures the user's facial expression, voice, and behavioral data and sends it to the emotion recognition module.

[0732] Server: Analyzes the acquired data and recognizes the user's emotions in real time. Based on the emotion recognition results, it provides advice on cleaning methods and motivation that correspond to the user's emotional state.

[0733] Terminal: Informs the user of these suggestions.

[0734] Specifically, the device collects the user's facial expressions and voice through a camera and microphone and transmits them to a server in real time. An emotion recognition algorithm uses Python's OpenCV and deep learning to classify emotions, and if a user shows a tired expression, it generates advice such as "Take a break" or "The next cleaning step is light work."

[0735] Specific examples

[0736] For example, a user can take a photo of their living room and upload it to their device through the application. This data is then sent to the server and fed into the image analysis module, which can then detect a coffee stain on the carpet.

[0737] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[0738] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[0739] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[0740] Recommended prompt sentence examples

[0741] "Take a photo of your living room and upload it to the app. If it finds a coffee stain on your carpet, it will suggest a specific cleaner and brush to use."

[0742] In this way, the present invention achieves increased cleaning efficiency and motivation through comprehensive support including emotion recognition technology.

[0743] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0744] Step 1: Data Acquisition

[0745] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[0746] Input: Current state of the room (photo or video)

[0747] Data processing: Capture images or videos using the camera function of your smartphone or device.

[0748] Output: Captured photo or video data

[0749] Specific operation: The user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the detailed condition of the room is recorded.

[0750] Step 2: Compress data and prepare for transmission

[0751] The device compresses the captured data and prepares it for transmission to the server.

[0752] Input: Photo or video data taken

[0753] Data processing: Compressing the data using compression algorithms such as JPEG or H.264.

[0754] Output: Compressed photo or video data

[0755] Specific operation: The device compresses the captured data in the specified format and prepares to send it to the server.

[0756] Step 3: Receiving data

[0757] The server receives the compressed data sent from the terminal.

[0758] Input: Compressed photo or video data

[0759] Data processing: Receive data using HTTP or WebSocket.

[0760] Output: Received photo or video data

[0761] Specific operation: The server receives and stores data at the specified port or address.

[0762] Step 4: Image analysis

[0763] The server passes the received data to an image analysis module, which processes it to identify areas of dirt or clutter.

[0764] Input: Received photo or video data

[0765] Data Computing: Uses deep learning algorithms to detect dirt and clutter.

[0766] Output: Identification of dirty and cluttered areas (including location information)

[0767] Specific operation: The server uses machine learning frameworks such as TensorFlow and PyTorch, and the model detects dust, blemishes, etc. in the image and outputs their location information.

[0768] Step 5: Generate cleaning suggestions

[0769] The server suggests effective cleaning methods based on the results from the image analysis module.

[0770] Input: Identification of dirty and cluttered areas

[0771] Data calculation: Evaluate the analysis results and extract the appropriate cleaning method from the database.

[0772] Output: Cleaning suggestions

[0773] Specific behavior: The server retrieves cleaning methods from a database and generates suggestions, such as "a specific detergent is needed to remove stains from the carpet."

[0774] Step 6: Proposal Notification

[0775] The device will notify the user of the suggestions and display them in an easy-to-understand format.

[0776] Input: Cleaning suggestions

[0777] Data processing: Convert the proposal content into a notification format.

[0778] Output: User notification

[0779] What it does: The device will communicate suggestions to the user via push notifications or in-app displays, such as "Use the following items to remove stains from your carpet."

[0780] Step 7: Acquire data after cleaning

[0781] After the user has finished cleaning, they take photos and videos again and send them to the server.

[0782] Input: Photo or video data after cleaning

[0783] Data processing: Images or videos are taken using the camera function of a smartphone or device and compressed.

[0784] Output: Compressed cleaned photo or video data

[0785] Specific operation: The user takes a photo of the room after cleaning and sends it back to the server.

[0786] Step 8: Virtual Reality (VR) Generation

[0787] The server compares the data before and after cleaning and generates a virtual reality representation of the state of the room after cleaning.

[0788] Input: Photos or video data before and after cleaning

[0789] Data calculation: Compare image data before and after cleaning to generate a virtual reality environment.

[0790] Output: Virtual reality environment data

[0791] Specific operation: Based on the comparison results, the server generates a 3D model of the clean room and sends it to the terminal as VR data.

[0792] Step 9: Experience the virtual environment

[0793] The device supports users in experiencing virtual environments using VR goggles and compatible apps.

[0794] Input: Virtual reality environment data

[0795] Data processing: Converting VR data into a format that can be displayed on VR goggles.

[0796] Output: Virtual reality experience to the user

[0797] Specific operation: The device reads the transmitted VR data and displays it to the user through the VR goggles, allowing the user to visually experience the state before and after cleaning in a virtual reality environment.

[0798] Step 10: Acquire emotion data

[0799] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[0800] Input: User's facial expression, voice, and behavior data

[0801] Data processing: Collect data using cameras and microphones.

[0802] Output: Obtained emotion data

[0803] Specific operation: The device uses a camera and microphone to record the user's facial expressions, voice, and actions in real time and transmits them to the server.

[0804] Step 11: Sentiment Analysis

[0805] The server analyzes the acquired emotional data and recognizes the user's emotions in real time.

[0806] Input: Acquired emotion data

[0807] Data Computing: Emotion recognition using deep learning algorithms.

[0808] Output: User's emotional state

[0809] Specific behavior: Emotion recognition algorithms analyze facial, voice, and behavioral data to classify the user's emotional state.

[0810] Step 12: Emotionally adaptive advice

[0811] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[0812] Input: User's emotional state

[0813] Data calculation: Generate appropriate advice according to emotional state.

[0814] Output: Personalized advice

[0815] Specific actions: For example, if the user is tired, advice such as "take a break" or "the next task is light work" is generated.

[0816] Step 13: Advice Notification

[0817] The terminal notifies the user of the advice.

[0818] Input: Personalized advice

[0819] Data processing: Convert the advice content into a notification format.

[0820] Output: Advice notice to user

[0821] Specific behavior: The device will communicate advice to the user via push notifications or in-app displays.

[0822] (Application example 2)

[0823] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0824] Conventional cleaning systems not only detect dirt and obstacles, but also lack the ability to propose efficient cleaning methods based on the detection results, or support for maintaining the motivation of cleaning staff. Effective cleaning management is also difficult because there are limited ways to visually check the status after cleaning. Especially in industrial environments, rapid detection and processing of dirt and obstacles is important, and methods for reducing fatigue and stress for cleaning staff are also required.

[0825] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images or videos of a dirty environment, means for analyzing the acquired images or videos to detect dirt and obstacles in the environment, means for proposing cleaning areas and cleaning methods based on the detected dirt and obstacles, means for generating a post-cleaning state using virtual reality technology, means for providing the generated virtual reality environment to the user, means for analyzing the facial expressions and voice of the cleaning staff and recognizing their emotions, and means for providing the user with advice on cleaning methods and motivation based on the recognized emotions. This enables effective cleaning management, reducing the burden on the staff and improving the efficiency and accuracy of cleaning.

[0826] "Images or videos of a dirty environment" means visual information, including dirt or obstacles present in a particular location or object, obtained using a camera or sensor.

[0827] "Means for analyzing captured images or videos and detecting environmental dirt or obstacles" refers to technology that uses deep learning algorithms and image processing technology to identify specific dirt or obstacles from captured visual information.

[0828] "Means for suggesting cleaning areas and cleaning methods" refers to technology that suggests optimal cleaning procedures and necessary tools to the user based on detected dirt and obstacles.

[0829] "Means for generating the post-cleaning state using virtual reality technology" refers to technology that compares the state before and after cleaning and reproduces the ideal post-cleaning environment in a virtual reality environment.

[0830] "Means for providing a generated virtual reality environment to a user" refers to technology that allows a user to visually experience the generated virtual environment using virtual reality goggles or other display devices.

[0831] "Means for analyzing the facial expressions and voice of cleaning staff and recognizing their emotions" refers to technology that acquires the facial expressions and speech of cleaning staff and analyzes that data to identify the emotional state of the staff.

[0832] "Means for providing users with advice on cleaning methods and motivation based on recognized emotions" refers to technology that provides users with appropriate changes to their cleaning methods and encouraging messages based on analyzed emotional data.

[0833] The system of the present invention aims to improve the efficiency of cleaning work in a factory environment and to increase the motivation of the person in charge. Specific embodiments of the present invention will be described below.

[0834] System Configuration

[0835] This system is composed of various modules, each of which is explained below.

[0836] Data Acquisition Module

[0837] The server uses a robot equipped with a camera to capture images or video of the environment to be cleaned. The robot collects visual data as it patrols the factory, which is then compressed and prepared for transmission to the server.

[0838] Image Analysis Module

[0839] The server analyzes the captured image or video data using deep learning algorithms, such as TensorFlow and Keras software, to identify dirt and obstacles within the factory.

[0840] Cleaning Suggestion Module

[0841] The server then proposes optimal cleaning methods based on the image analysis results, including specific instructions on the cleaning agents, brushes, and other cleaning equipment to be used. The proposal results are then sent to the user via their device.

[0842] Virtual Reality (VR) Module

[0843] The server uses virtual reality technology to generate the post-cleaning environment. It compares data before and after cleaning and simulates the effect in the VR environment. Users can experience the generated virtual environment using VR goggles.

[0844] Emotion Recognition Module

[0845] The device captures the facial expressions and voice of the cleaner and sends them to an emotion recognition module. The server then analyzes the facial and voice data using deep learning algorithms to identify the cleaner's emotional state. The software used includes a Keras model that supports emotion recognition.

[0846] Based on the emotion recognition results, the server can suggest changes to cleaning methods or provide encouraging messages to the user. For example, if fatigue is detected, the server can provide advice such as "Take a break" or "Try a lighter task next time."

[0847] Specific examples

[0848] In a specific scenario, a robot detects dirt in a factory and the server suggests appropriate cleaning methods. After the cleaning is complete, the user can visually check the cleaning results using VR goggles. Furthermore, the server analyzes the operator's facial expressions and suggests timely breaks if fatigue is detected.

[0849] Prompt Sentence Examples

[0850] I am developing a highly automated factory cleaning robot. This robot uses cameras to detect dirt in the factory and suggests the best cleaning method. It also simulates the cleaning state using VR goggles and analyzes the operator's emotions to provide real-time feedback. This system incorporates image analysis, emotion recognition, and virtual reality technology using deep learning algorithms. I would like your detailed feedback and suggestions for improvement based on this project.

[0851] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0852] Step 1:

[0853] The user starts the robot to acquire environmental data. Specifically, the camera mounted on the robot is used to acquire images and videos of the factory interior. Libraries such as OpenCV are used to acquire this data. The input is real-time video of the factory interior, and the output is a file that stores this in image or video data format.

[0854] Step 2:

[0855] The device compresses the acquired data and prepares it for transmission to the server. Specifically, it converts image and video files into a format suitable for communication and compresses them. The input is the image and video data generated in step 1, and the output is a compressed data file.

[0856] Step 3:

[0857] The server receives the compressed data and passes it to the image analysis module. The server decodes the received data and analyzes it using deep learning algorithms. This processing uses TensorFlow and Keras. The input is the compressed data file, and the output is information about detected dirt and obstacles.

[0858] Step 4:

[0859] The server proposes the optimal cleaning method based on the results of image analysis. Based on the analyzed data, it recommends the cleaning tools, detergents, and cleaning procedures to be used. This information is generated by the cleaning suggestion module. The input is the image analysis results, and the output is a detailed cleaning procedure proposal.

[0860] Step 5:

[0861] The device notifies the user of the suggested cleaning method by displaying or reading out the required tools and cleaning steps to the user using the device's display or audio output. The input is the cleaning suggestion obtained from the server, and the output is a visual or audio notification to the user.

[0862] Step 6:

[0863] The user performs the cleaning as suggested and takes photos and videos during the process. The device sends these data to the server. The input is the image and video data after cleaning, and the output is compressed data sent to the server.

[0864] Step 7:

[0865] The server compares the data before and after cleaning and uses virtual reality technology to generate an ideal post-cleaning environment, allowing users to visually experience the cleaning results. The input is the data before and after cleaning, and the output is the generated virtual environment data.

[0866] Step 8:

[0867] The device captures the facial expressions and voice of the cleaner and sends them to the emotion recognition module. The facial expression and voice data are analyzed using emotion recognition algorithms (such as Keras). The input is the cleaner's facial expression and voice data, and the output is the analyzed emotional state.

[0868] Step 9:

[0869] The server then provides the user with advice on how to change their cleaning method or improve their motivation based on the analyzed emotional data. Specifically, it suggests taking a break as needed or showing the next cleaning step. The input is the analyzed emotional state, and the output is advice for the user.

[0870] Step 10:

[0871] The user receives advice via the terminal and continues cleaning. In this way, the system achieves effective cleaning management and reduces the burden on the person in charge.

[0872] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0873] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0874] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0875] [Third embodiment]

[0876] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0877] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0878] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0879] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0880] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0881] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0882] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0883] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0884] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0885] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0886] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0887] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0888] The present invention provides a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment. Specific embodiments of the present invention are described below.

[0889] System configuration

[0890] The system consists of four main parts:

[0891] 1. Data Acquisition Module

[0892] 2. Image Analysis Module

[0893] 3. Cleaning suggestion module

[0894] 4. Virtual Reality (VR) Module

[0895] Data Acquisition Module

[0896] Users can take photos and videos of the room using an application on their smartphone or a dedicated device.

[0897] The device prepares the captured data for transmission to the server, specifically by compressing the data, converting the format, and adding metadata.

[0898] Image Analysis Module

[0899] The server inputs the received data into an image analysis module and processes it to identify dirty or cluttered areas in the room.

[0900] Image analysis uses deep learning-based algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[0901] The identified information is organized and stored in a database.

[0902] Cleaning suggestion module

[0903] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[0904] The suggestions include specific cleaning procedures and essential items (e.g., vacuum cleaner, detergent, brush).

[0905] The device notifies the user of this information and displays it in a concise and easy-to-understand format.

[0906] Virtual Reality (VR) Module

[0907] The server runs a VR module to simulate the state of the room after the user has completed cleaning.

[0908] By comparing data before and after cleaning, users can visually experience the difference in a VR environment, giving them a sense of accomplishment from cleaning.

[0909] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[0910] Specific examples

[0911] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[0912] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[0913] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the cleaned up state. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[0914] In this way, the present invention aims to achieve both improved cleaning efficiency and motivation.

[0915] The processing flow will be explained below.

[0916] Step 1:

[0917] The user launches a dedicated application and takes photos and videos of the room.

[0918] Step 2:

[0919] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[0920] Step 3:

[0921] The terminal transmits the prepared data to the server.

[0922] Step 4:

[0923] The server passes the received data to the image analysis module.

[0924] Step 5:

[0925] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[0926] Step 6:

[0927] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[0928] Step 7:

[0929] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[0930] Step 8:

[0931] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[0932] Step 9:

[0933] The user cleans the room using the cleaning method presented as a reference.

[0934] Step 10:

[0935] The user takes a picture of the state of the room after cleaning again and uploads the new photo or video data to the terminal.

[0936] Step 11:

[0937] The device sends the post-cleaning data to the server.

[0938] Step 12:

[0939] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[0940] Step 13:

[0941] The virtual reality data generated by the server is sent to the terminal.

[0942] Step 14:

[0943] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[0944] Step 15:

[0945] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[0946] Example 1

[0947] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0948] Cleaning a room is labor-intensive and troublesome for many people. It is also difficult to determine which areas need cleaning and the best cleaning methods, making it extremely difficult to clean effectively. Furthermore, the effectiveness of cleaning cannot be reliably confirmed, which can lead to a decline in motivation. There is a need for a system that can solve these issues, improve cleaning efficiency, and increase motivation.

[0949] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0950] In this invention, the server includes means for identifying dirty and messy areas in a room using an image analysis model, means for storing the analysis results in a database, means for proposing cleaning methods based on the analysis results, and means for comparing data before and after cleaning and generating the state of the room after cleaning using virtual reality technology, thereby enabling the user to accurately grasp the current state of the room, practice effective cleaning methods, and virtually experience the sense of accomplishment after cleaning.

[0951] "User" refers to an individual or group that uses this system to clean a room.

[0952] "Terminal" refers to the device that the user uses to take pictures and videos of the room and send them to the server. Specifically, this applies to smartphones and dedicated terminals.

[0953] "Server" refers to a central control unit for analyzing, storing, and making suggestions about data sent by users.

[0954] "Image analysis models" refer to programs that use deep learning algorithms to identify dirty or cluttered areas in a room, such as YOLO and Mask R-CNN.

[0955] A "database" refers to a system for systematically storing and searching analysis results and related information.

[0956] "Cleaning method" refers to information on the optimal cleaning procedures and cleaning tools to be used, identified based on the analysis results.

[0957] "Virtual reality technology" refers to technology that allows users to visually experience the state of a room after cleaning, specifically 3D modeling software and VR devices.

[0958] "Data before and after cleaning" refers to images and videos showing the state of the room before and after the user cleans.

[0959] "Virtual reality environment" refers to a virtual space generated by the server that visually reproduces the state of a room after cleaning.

[0960] The present invention is a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment.

[0961] 1. Specific data acquisition procedures

[0962] Users use their smartphones or dedicated devices to take photos and videos of their rooms. Specifically, they launch a smartphone app and take photos of various parts of the room. The device receives this data and organizes it by compressing and converting the format. For example, it converts it to JPEG format and adds metadata (date and time of shooting, location information).

[0963] 2. Specific operations for data transmission

[0964] The device sends the data the user has acquired to the server using Wi-Fi or mobile data, encrypting the data using security protocols such as SSL.

[0965] 3. Specific procedures for image analysis

[0966] The server inputs the received data into an image analysis module and uses deep learning-based image analysis models (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. The analysis results are organized in JSON format or similar and stored in a database.

[0967] 4. Specific cleaning operations

[0968] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate effective cleaning methods based on the results of image analysis. The recommendations include specific cleaning steps and information on the tools needed. For example, it suggests that a specific detergent and brush are needed to remove stains from a carpet.

[0969] 5. Specific operation of VR simulation

[0970] After cleaning, the user takes photos and videos of the room again and uploads them to the server via their device. The server compares the data before and after cleaning and uses 3D modeling software (e.g., Blender) to generate a VR environment of the room after cleaning.

[0971] 6. Specific operations for VR experiences

[0972] The device connects to a VR device (e.g., Oculus Rift or HTC Vive) and downloads the generated VR environment data. Users can then wear VR goggles and visually experience the state of the room before and after cleaning, giving them a sense of accomplishment from cleaning.

[0973] Specific examples

[0974] For example, a user takes a photo of their living room and uploads it to their device through an application. This data is compressed, converted, and sent to a server. After analyzing the image, the server detects that there is a coffee stain on the carpet. The server then suggests using a specific detergent and brush to remove the stain from the carpet and provides a link to purchase the necessary tools. The device notifies the user, and the user completes the cleaning. By taking another photo and uploading it using the same steps, the server compares the state before and after cleaning and generates a VR environment. The device then plays this on a VR device, allowing the user to experience the virtual environment.

[0975] Prompt Sentence Examples

[0976] Below are some example prompts to input the behavior of this system into the generative AI model.

[0977] "Please explain in detail how this system uses image analysis and virtual reality technology to efficiently clean rooms. A user takes a photo of the room with their smartphone and sends it to a server via their device. The server then analyzes the image, detects dirt and clutter, and makes cleaning suggestions. After cleaning is complete, a VR module is used to compare the state before and after cleaning, allowing the user to experience the state after cleaning in a virtual environment."

[0978] This system improves the efficiency of the cleaning process while also increasing user motivation.

[0979] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0980] Step 1:

[0981] The user takes photos and videos of the room using a smartphone or dedicated device. Specifically, the user starts the app and takes multiple photos of various parts of the room from various angles. The input is the images and videos of the room, and the output is this data.

[0982] Step 2:

[0983] The device compresses the captured data, converts the format, and adds metadata. The device converts the data to JPEG or MPEG format, and adds the date and time of the capture and location information as metadata. The input is the captured image or video, and the output is the compressed and converted data.

[0984] Step 3:

[0985] The device sends data to the server. The compressed and converted data is then sent to the server using Wi-Fi or mobile data. An encryption protocol such as SSL is used during transmission. The input is the compressed and converted data, and the output is the data sent to the server.

[0986] Step 4:

[0987] The server inputs the received data into an image analysis module, which uses a deep learning-based image analysis model (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. In the process, the server identifies dirt, stains, improper placement, etc. The input is the data sent to the server, and the output is the analysis results.

[0988] Step 5:

[0989] The server saves the analysis results in a database. The analysis results are organized in JSON format or similar and stored in the database in a format that allows for efficient search and reference. The input is the analysis results, and the output is the data saved in the database.

[0990] Step 6:

[0991] The server proposes cleaning methods based on the analysis results. Using a generative AI model (e.g., OpenAI GPT-4), it generates recommendations regarding optimal cleaning procedures and cleaning tools to use. The input is the analysis results, and the output is the recommendations.

[0992] Step 7:

[0993] The terminal notifies the user of the suggestions from the server. The terminal displays the suggested cleaning procedures and necessary tools in an easy-to-understand format. The input is the suggestions, and the output is the information notified to the user.

[0994] Step 8:

[0995] The user actually cleans based on the suggestions. The user uses the necessary tools and follows the suggested procedure to clean. The input is the suggestions and cleaning tools, and the output is the clean state after cleaning.

[0996] Step 9:

[0997] The user then takes photos or videos of the state of the room after cleaning and uploads them to the server via the device in the same way. The input is the post-cleaning images or videos, and the output is the post-cleaning data sent to the server.

[0998] Step 10:

[0999] The server compares the data before and after cleaning and generates the state of the room after cleaning using virtual reality technology. The server uses 3D modeling software (e.g., Blender) to visualize the difference between the state before and after cleaning. The input is the data before and after cleaning, and the output is the generated virtual reality environment.

[1000] Step 11:

[1001] The device provides the generated virtual reality environment to the user. The device works in conjunction with a VR device (e.g., Oculus Rift, HTC Vive) to allow the user to experience the virtual environment. The input is the generated virtual reality environment, and the output is the virtual experience provided to the user.

[1002] (Application example 1)

[1003] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1004] Cleaning work in factories and other physical spaces is often manual and inefficient. Particularly in large factories, it is difficult to identify dirty or messy areas and clean them efficiently, increasing the burden on workers. Furthermore, there is no way to immediately check the results of cleaning, making it difficult to visualize the effectiveness of the work. To solve these problems, a system is needed that automates cleaning work and makes it easy to check the results.

[1005] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1006] In this invention, the server includes means for analyzing images or videos to detect dirt and object placement in the physical space, means for proposing areas to be cleaned and cleaning methods based on the detected dirt and placement, and means for generating the state of the physical space after cleaning using virtual reality technology. This allows for efficient automation of cleaning work in physical spaces such as factories, and enables users to immediately visually confirm the cleaning results.

[1007] An "image or video capturing means" is a device or method for capturing images or video within a physical space.

[1008] "Means for analyzing images or videos" refers to an algorithm or device for detecting the placement of dirt or objects using captured images or videos.

[1009] A "means for detecting dirt or placement" is a system or algorithm that identifies the presence of dirt or the placement of an object based on analyzed images or video.

[1010] The "means for suggesting cleaning locations and cleaning methods" is a system or method that suggests specific locations to be cleaned and appropriate cleaning methods based on detected dirt and location information.

[1011] "Means for generating the state of a physical space after cleaning using virtual reality technology" refers to a system or technology that uses virtual reality technology to simulate and visually present what a physical space will look like after cleaning.

[1012] The "means for providing a generated virtual reality environment to a user" refers to a device or platform that allows a user to experience a virtual reality environment.

[1013] "Means for a robot to capture images or videos in a physical space and transmit them to a server" refers to a system or method for a robot to acquire images or videos using a photographing device and transmit the data to a server.

[1014] The "robot means for carrying out cleaning in accordance with the provided suggestions" is a robot capable of actually carrying out the cleaning method suggested based on the image analysis results.

[1015] "Means for comparing images before and after cleaning to visually confirm the cleaning effect" refers to a system or method that compares images or videos before and after cleaning and visually shows the difference to the user.

[1016] The following system is designed as an embodiment of the present invention. The system is configured to use a robot to efficiently clean a physical space such as a factory and to visually confirm the cleaning results. The system mainly consists of the following components:

[1017] Data Acquisition Module

[1018] The robot is equipped with a camera that takes photos and videos of the factory. The robot takes images and videos as it moves around the physical space and sends them to a server. The captured data is compressed and formatted within the robot before being sent to the server.

[1019] Image Analysis Module

[1020] The server receives the transmitted image and video data and analyzes it using the image analysis module. This analysis mainly uses deep learning-based algorithms such as TensorFlow and OpenCV. As a result of the analysis, dirt and improperly placed objects within the factory are identified. The analysis results are stored in a database and used for subsequent processing.

[1021] Cleaning suggestion module

[1022] Based on the results of the image analysis module, the server proposes efficient cleaning methods and necessary tools. These proposals include specific steps, such as "If oil stains are detected, clean using a specific detergent and cloth." Links to purchase the necessary tools are also provided. This information is then sent to the robot and the user's device.

[1023] Virtual Reality (VR) Module

[1024] The server uses virtual reality technology to simulate the state of the physical space before and after cleaning. In particular, it predicts the state after cleaning and allows users to visually confirm the cleaning results. This virtual reality environment can be experienced by users through VR goggles or a corresponding application.

[1025] Specific examples

[1026] For example, if oil stains are found on the floor of a factory, the robot takes a photo of the area and sends the image to the server. The server then uses an image analysis module to detect the oil stains and a cleaning suggestion module to suggest cleaning using a specified detergent and cloth. This suggestion is sent to the robot, which then carries out the cleaning. After cleaning, the robot takes another image, and a virtual reality module compares the before and after cleaning, allowing the user to check the results using VR goggles.

[1027] Prompt Sentence Examples

[1028] "Analyze these images to identify dirt and unwanted materials in the factory and suggest effective cleaning methods."

[1029] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1030] Step 1:

[1031] As the robot moves around the factory, it uses a camera to take photos and videos. In this step, the robot acquires data from the actual environment. The input is visual data in the physical space, and the output is compressed and formatted image and video data.

[1032] Step 2:

[1033] The terminal receives the data captured by the robot and sends it to the server. The input is compressed and format-converted data, and the output is image and video data uploaded to the server.

[1034] Step 3:

[1035] The data received by the server is input into the image analysis module for analysis. The input is image and video data uploaded to the server, and the output is information on dirt and object placement in the physical space. Specifically, data calculations using a deep learning model detect dirt and identify improper object placement.

[1036] Step 4:

[1037] The server then proposes cleaning areas and methods based on the results of image analysis. The input is dirt and placement information obtained from the image analysis module, and the output is a list of specific cleaning procedures and necessary tools. Specific actions such as "if oil stains are detected, clean using a specific detergent and cloth" are generated.

[1038] Step 5:

[1039] The terminal sends the proposed cleaning content along with a prompt to the robot, which then actually performs the cleaning. The input is the cleaning proposal information from the server, and the output is the performed cleaning task. The robot cleans the physical space using the specified cleaning tools.

[1040] Step 6:

[1041] After cleaning, the robot takes images and videos again and sends them to the server. The input is visual data of the physical space after cleaning, and the output is image and video data uploaded to the server.

[1042] Step 7:

[1043] The server compares image data before and after cleaning and generates the results in a virtual reality environment. The input is image data before and after cleaning, and the output is comparative information before and after cleaning. Specifically, the before-cleaning and after-cleaning states are displayed side by side, visually highlighting the differences.

[1044] Step 8:

[1045] The device provides a virtual reality environment to the user, who then uses VR goggles to check the cleaning results. The input is comparison information from the server before and after cleaning, and the output is the user's visual confirmation experience. The user can wear the VR goggles and check the before and after cleaning status in 3D.

[1046] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1047] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention helps a user understand the current state of the room, implement effective cleaning procedures, recognize emotions during the cleaning process, and virtually experience a sense of accomplishment. Specific embodiments of the present invention are described below.

[1048] System configuration

[1049] The system is broadly divided into five main parts:

[1050] 1. Data Acquisition Module

[1051] 2. Image Analysis Module

[1052] 3. Cleaning suggestion module

[1053] 4. Virtual Reality (VR) Module

[1054] 5. Emotion Recognition Module

[1055] Data Acquisition Module

[1056] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[1057] The device compresses the captured data and prepares it for transmission to the server.

[1058] Image Analysis Module

[1059] The server passes the received data to an image analysis module, which processes it to identify dirty or cluttered areas in the room.

[1060] Image analysis uses deep learning algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[1061] Cleaning suggestion module

[1062] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[1063] The suggestions include specific cleaning procedures and essential items.

[1064] The device notifies the user of this information and displays it in a concise, easy-to-understand format.

[1065] Virtual Reality (VR) Module

[1066] Run the VR module to simulate the state of the room after the server has finished cleaning.

[1067] Data before and after cleaning is compared, allowing users to visually experience the difference in a VR environment.

[1068] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[1069] Emotion Recognition Module

[1070] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[1071] The server analyzes the acquired data and recognizes the user's emotions in real time.

[1072] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[1073] The terminal notifies the user of these advices.

[1074] Specific examples

[1075] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[1076] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[1077] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[1078] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[1079] In this way, the present invention achieves improved cleaning efficiency and motivation through comprehensive support, including emotion recognition technology.

[1080] The processing flow will be explained below.

[1081] Step 1:

[1082] The user launches a dedicated application and takes photos and videos of the room.

[1083] Step 2:

[1084] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[1085] Step 3:

[1086] The terminal transmits the prepared data to the server.

[1087] Step 4:

[1088] The server passes the received data to the image analysis module.

[1089] Step 5:

[1090] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[1091] Step 6:

[1092] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[1093] Step 7:

[1094] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[1095] Step 8:

[1096] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[1097] Step 9:

[1098] The user starts cleaning the room according to the presented cleaning method.

[1099] Step 10:

[1100] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[1101] Step 11:

[1102] The server uses an emotion recognition module to analyze the acquired data and recognize the user's emotions in real time.

[1103] Step 12:

[1104] The server generates appropriate advice or encouraging messages according to the user's emotional state and sends them to the terminal.

[1105] Step 13:

[1106] The device notifies the user of advice and messages received and displays suggestions to increase motivation.

[1107] Step 14:

[1108] After the user has completed cleaning, they can take photos or videos again and upload the new data to the device.

[1109] Step 15:

[1110] The device sends the post-cleaning data to the server.

[1111] Step 16:

[1112] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[1113] Step 17:

[1114] The virtual reality data generated by the server is sent to the terminal.

[1115] Step 18:

[1116] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[1117] Step 19:

[1118] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[1119] Example 2

[1120] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1121] In recent years, cleaning a room has become a burden for individuals with busy lifestyles. Maintaining motivation and choosing an efficient cleaning method are particularly difficult. Furthermore, conventional cleaning methods offer limited means for visually confirming the results of cleaning, making it difficult to achieve a sense of accomplishment. Given this background, there is a demand for a system that allows users to efficiently clean a room, maintain their motivation, and visually experience the results of cleaning.

[1122] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring images or videos of the room taken by the user; means for compressing the acquired images or videos and transmitting them to the server; means for analyzing the images or videos received by the server and identifying stains and the location of objects in the room using a deep learning algorithm; means for suggesting areas to be cleaned and cleaning methods based on the identified stains and locations; means for notifying and displaying the suggestions to the user in an easy-to-understand format; means for generating a post-cleaning state of the room using virtual reality technology and visually presenting it to the user; means for acquiring the user's facial expression, voice, and behavior data, analyzing the data, recognizing emotions, and providing a cleaning method according to the user's emotional state and advice for improving motivation based on the recognition results; and means for notifying the user of the advice based on the acquired emotional data. This not only improves cleaning efficiency but also maintains the user's motivation and enhances the sense of accomplishment after cleaning.

[1123] "User" refers to an individual who uses the system to improve cleaning efficiency and check the effectiveness.

[1124] A "terminal" refers to a smartphone or dedicated device used by a user, which captures images and videos of the room and sends them to a server.

[1125] "Server" refers to a central processing unit that receives data, analyzes it, makes recommendations, and generates the virtual reality environment.

[1126] "Image analysis" refers to the process of using deep learning algorithms to identify dirt and object placement in a room.

[1127] A "deep learning algorithm" is a type of machine learning used to perform complex image analysis and automatically detect specific objects or conditions.

[1128] "Cleaning suggestions" refers to the process of suggesting effective cleaning locations and cleaning methods to the user based on the analyzed data.

[1129] "Virtual reality technology" refers to a simulation technology that allows users to visually experience the state of a room before and after cleaning.

[1130] "Emotion recognition" refers to the process of analyzing a user's facial, vocal, and behavioral data to identify their emotional state.

[1131] "Advice" refers to guidance and recommendations on cleaning methods and motivation provided to users based on emotion recognition results.

[1132] "Compression" refers to a data processing technique for reducing the volume of captured image or video data.

[1133] "Notification" refers to the process by which a device communicates suggestions or advice to a user.

[1134] MODE FOR CARRYING OUT THE INVENTION

[1135] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention supports a user in understanding the current state of the room, implementing effective cleaning procedures, and virtually experiencing a sense of accomplishment while recognizing emotions during the cleaning process. Specific embodiments of the present invention are described below.

[1136] The system consists of five main parts:

[1137] 1. Data Acquisition Module

[1138] 2. Image Analysis Module

[1139] 3. Cleaning suggestion module

[1140] 4. Virtual Reality (VR) Module

[1141] 5. Emotion Recognition Module

[1142] Data Acquisition Module

[1143] User: Take photos and videos of the room using a smartphone or dedicated device application.

[1144] Terminal: Compresses the captured data and prepares it for transmission to the server.

[1145] Specifically, the user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the device records the detailed state of the room. The device then uses compression algorithms such as JPEG or H.264 to reduce the data size and sends it to the server.

[1146] Image Analysis Module

[1147] Server: Passes the received data to the image analysis module, which uses deep learning algorithms to identify dirty and cluttered areas in the room.

[1148] The server receives the data via HTTP or WebSocket, saves it to disk, and then forwards it to the analysis module, which uses machine learning frameworks such as TensorFlow and PyTorch to analyze it. The model detects dust particles and blemishes in the image and outputs their location information.

[1149] Cleaning suggestion module

[1150] Server: Based on the results from the image analysis module, it proposes effective cleaning methods.

[1151] Terminal: Informs the user of the suggestions and displays them in a concise and easy-to-understand format.

[1152] Specifically, the server evaluates the analysis results and pulls out appropriate cleaning methods from the database. For example, it generates suggestions such as "A specific detergent and brush are needed to remove stains from the carpet." These suggestions and links are then notified and displayed to the user via their device.

[1153] Virtual Reality (VR) Module

[1154] Server: Runs the VR module to simulate the state of the room after cleaning is completed.

[1155] Device: Helps users experience virtual environments using VR goggles and compatible apps.

[1156] By comparing image data before and after cleaning, a 3D model of the clean room is generated. This model is sent to the device as a data file for the VR app, and the user can visually experience the state of the room after cleaning by wearing VR goggles.

[1157] Emotion Recognition Module

[1158] Terminal: Captures the user's facial expression, voice, and behavioral data and sends it to the emotion recognition module.

[1159] Server: Analyzes the acquired data and recognizes the user's emotions in real time. Based on the emotion recognition results, it provides advice on cleaning methods and motivation that correspond to the user's emotional state.

[1160] Terminal: Informs the user of these suggestions.

[1161] Specifically, the device collects the user's facial expressions and voice through a camera and microphone and transmits them to a server in real time. An emotion recognition algorithm uses Python's OpenCV and deep learning to classify emotions, and if a user shows a tired expression, it generates advice such as "Take a break" or "The next cleaning step is light work."

[1162] Specific examples

[1163] For example, a user can take a photo of their living room and upload it to their device through the application. This data is then sent to the server and fed into the image analysis module, which can then detect a coffee stain on the carpet.

[1164] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[1165] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[1166] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[1167] Recommended prompt sentence examples

[1168] "Take a photo of your living room and upload it to the app. If it finds a coffee stain on your carpet, it will suggest a specific cleaner and brush to use."

[1169] In this way, the present invention achieves increased cleaning efficiency and motivation through comprehensive support including emotion recognition technology.

[1170] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1171] Step 1: Data Acquisition

[1172] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[1173] Input: Current state of the room (photo or video)

[1174] Data processing: Capture images or videos using the camera function of your smartphone or device.

[1175] Output: Captured photo or video data

[1176] Specific operation: The user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the detailed condition of the room is recorded.

[1177] Step 2: Compress data and prepare for transmission

[1178] The device compresses the captured data and prepares it for transmission to the server.

[1179] Input: Photo or video data taken

[1180] Data processing: Compressing the data using compression algorithms such as JPEG or H.264.

[1181] Output: Compressed photo or video data

[1182] Specific operation: The device compresses the captured data in the specified format and prepares to send it to the server.

[1183] Step 3: Receiving data

[1184] The server receives the compressed data sent from the terminal.

[1185] Input: Compressed photo or video data

[1186] Data processing: Receive data using HTTP or WebSocket.

[1187] Output: Received photo or video data

[1188] Specific operation: The server receives and stores data at the specified port or address.

[1189] Step 4: Image analysis

[1190] The server passes the received data to an image analysis module, which processes it to identify areas of dirt or clutter.

[1191] Input: Received photo or video data

[1192] Data Computing: Uses deep learning algorithms to detect dirt and clutter.

[1193] Output: Identification of dirty and cluttered areas (including location information)

[1194] Specific operation: The server uses machine learning frameworks such as TensorFlow and PyTorch, and the model detects dust, blemishes, etc. in the image and outputs their location information.

[1195] Step 5: Generate cleaning suggestions

[1196] The server suggests effective cleaning methods based on the results from the image analysis module.

[1197] Input: Identification of dirty and cluttered areas

[1198] Data calculation: Evaluate the analysis results and extract the appropriate cleaning method from the database.

[1199] Output: Cleaning suggestions

[1200] Specific behavior: The server retrieves cleaning methods from a database and generates suggestions, such as "a specific detergent is needed to remove stains from the carpet."

[1201] Step 6: Proposal Notification

[1202] The device will notify the user of the suggestions and display them in an easy-to-understand format.

[1203] Input: Cleaning suggestions

[1204] Data processing: Convert the proposal content into a notification format.

[1205] Output: User notification

[1206] What it does: The device will communicate suggestions to the user via push notifications or in-app displays, such as "Use the following items to remove stains from your carpet."

[1207] Step 7: Acquire data after cleaning

[1208] After the user has finished cleaning, they take photos and videos again and send them to the server.

[1209] Input: Photo or video data after cleaning

[1210] Data processing: Images or videos are taken using the camera function of a smartphone or device and compressed.

[1211] Output: Compressed cleaned photo or video data

[1212] Specific operation: The user takes a photo of the room after cleaning and sends it back to the server.

[1213] Step 8: Virtual Reality (VR) Generation

[1214] The server compares the data before and after cleaning and generates a virtual reality representation of the state of the room after cleaning.

[1215] Input: Photos or video data before and after cleaning

[1216] Data calculation: Compare image data before and after cleaning to generate a virtual reality environment.

[1217] Output: Virtual reality environment data

[1218] Specific operation: Based on the comparison results, the server generates a 3D model of the clean room and sends it to the terminal as VR data.

[1219] Step 9: Experience the virtual environment

[1220] The device supports users in experiencing virtual environments using VR goggles and compatible apps.

[1221] Input: Virtual reality environment data

[1222] Data processing: Converting VR data into a format that can be displayed on VR goggles.

[1223] Output: Virtual reality experience to the user

[1224] Specific operation: The device reads the transmitted VR data and displays it to the user through the VR goggles, allowing the user to visually experience the state before and after cleaning in a virtual reality environment.

[1225] Step 10: Acquire emotion data

[1226] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[1227] Input: User's facial expression, voice, and behavior data

[1228] Data processing: Collect data using cameras and microphones.

[1229] Output: Obtained emotion data

[1230] Specific operation: The device uses a camera and microphone to record the user's facial expressions, voice, and actions in real time and transmits them to the server.

[1231] Step 11: Sentiment Analysis

[1232] The server analyzes the acquired emotional data and recognizes the user's emotions in real time.

[1233] Input: Acquired emotion data

[1234] Data Computing: Emotion recognition using deep learning algorithms.

[1235] Output: User's emotional state

[1236] Specific behavior: Emotion recognition algorithms analyze facial, voice, and behavioral data to classify the user's emotional state.

[1237] Step 12: Emotionally adaptive advice

[1238] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[1239] Input: User's emotional state

[1240] Data calculation: Generate appropriate advice according to emotional state.

[1241] Output: Personalized advice

[1242] Specific actions: For example, if the user is tired, advice such as "take a break" or "the next task is light work" is generated.

[1243] Step 13: Advice Notification

[1244] The terminal notifies the user of the advice.

[1245] Input: Personalized advice

[1246] Data processing: Convert the advice content into a notification format.

[1247] Output: Advice notice to user

[1248] Specific behavior: The device will communicate advice to the user via push notifications or in-app displays.

[1249] (Application example 2)

[1250] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1251] Conventional cleaning systems not only detect dirt and obstacles, but also lack the ability to propose efficient cleaning methods based on the detection results, or support for maintaining the motivation of cleaning staff. Effective cleaning management is also difficult because there are limited ways to visually check the status after cleaning. Especially in industrial environments, rapid detection and processing of dirt and obstacles is important, and methods for reducing fatigue and stress for cleaning staff are also required.

[1252] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images or videos of a dirty environment, means for analyzing the acquired images or videos to detect dirt and obstacles in the environment, means for proposing cleaning areas and cleaning methods based on the detected dirt and obstacles, means for generating a post-cleaning state using virtual reality technology, means for providing the generated virtual reality environment to the user, means for analyzing the facial expressions and voice of the cleaning staff and recognizing their emotions, and means for providing the user with advice on cleaning methods and motivation based on the recognized emotions. This enables effective cleaning management, reducing the burden on the staff and improving the efficiency and accuracy of cleaning.

[1253] "Images or videos of a dirty environment" means visual information, including dirt or obstacles present in a particular location or object, obtained using a camera or sensor.

[1254] "Means for analyzing captured images or videos and detecting environmental dirt or obstacles" refers to technology that uses deep learning algorithms and image processing technology to identify specific dirt or obstacles from captured visual information.

[1255] "Means for suggesting cleaning areas and cleaning methods" refers to technology that suggests optimal cleaning procedures and necessary tools to the user based on detected dirt and obstacles.

[1256] "Means for generating the post-cleaning state using virtual reality technology" refers to technology that compares the state before and after cleaning and reproduces the ideal post-cleaning environment in a virtual reality environment.

[1257] "Means for providing a generated virtual reality environment to a user" refers to technology that allows a user to visually experience the generated virtual environment using virtual reality goggles or other display devices.

[1258] "Means for analyzing the facial expressions and voice of cleaning staff and recognizing their emotions" refers to technology that acquires the facial expressions and speech of cleaning staff and analyzes that data to identify the emotional state of the staff.

[1259] "Means for providing users with advice on cleaning methods and motivation based on recognized emotions" refers to technology that provides users with appropriate changes to their cleaning methods and encouraging messages based on analyzed emotional data.

[1260] The system of the present invention aims to improve the efficiency of cleaning work in a factory environment and to increase the motivation of the person in charge. Specific embodiments of the present invention will be described below.

[1261] System Configuration

[1262] This system is composed of various modules, each of which is explained below.

[1263] Data Acquisition Module

[1264] The server uses a robot equipped with a camera to capture images or video of the environment to be cleaned. The robot collects visual data as it patrols the factory, which is then compressed and prepared for transmission to the server.

[1265] Image Analysis Module

[1266] The server analyzes the captured image or video data using deep learning algorithms, such as TensorFlow and Keras software, to identify dirt and obstacles within the factory.

[1267] Cleaning Suggestion Module

[1268] The server then proposes optimal cleaning methods based on the image analysis results, including specific instructions on the cleaning agents, brushes, and other cleaning equipment to be used. The proposal results are then sent to the user via their device.

[1269] Virtual Reality (VR) Module

[1270] The server uses virtual reality technology to generate the post-cleaning environment. It compares data before and after cleaning and simulates the effect in the VR environment. Users can experience the generated virtual environment using VR goggles.

[1271] Emotion Recognition Module

[1272] The device captures the facial expressions and voice of the cleaner and sends them to an emotion recognition module. The server then analyzes the facial and voice data using deep learning algorithms to identify the cleaner's emotional state. The software used includes a Keras model that supports emotion recognition.

[1273] Based on the emotion recognition results, the server can suggest changes to cleaning methods or provide encouraging messages to the user. For example, if fatigue is detected, the server can provide advice such as "Take a break" or "Try a lighter task next time."

[1274] Specific examples

[1275] In a specific scenario, a robot detects dirt in a factory and the server suggests appropriate cleaning methods. After the cleaning is complete, the user can visually check the cleaning results using VR goggles. Furthermore, the server analyzes the operator's facial expressions and suggests timely breaks if fatigue is detected.

[1276] Prompt Sentence Examples

[1277] I am developing a highly automated factory cleaning robot. This robot uses cameras to detect dirt in the factory and suggests the best cleaning method. It also simulates the cleaning state using VR goggles and analyzes the operator's emotions to provide real-time feedback. This system incorporates image analysis, emotion recognition, and virtual reality technology using deep learning algorithms. I would like your detailed feedback and suggestions for improvement based on this project.

[1278] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1279] Step 1:

[1280] The user starts the robot to acquire environmental data. Specifically, the camera mounted on the robot is used to acquire images and videos of the factory interior. Libraries such as OpenCV are used to acquire this data. The input is real-time video of the factory interior, and the output is a file that stores this in image or video data format.

[1281] Step 2:

[1282] The device compresses the acquired data and prepares it for transmission to the server. Specifically, it converts image and video files into a format suitable for communication and compresses them. The input is the image and video data generated in step 1, and the output is a compressed data file.

[1283] Step 3:

[1284] The server receives the compressed data and passes it to the image analysis module. The server decodes the received data and analyzes it using deep learning algorithms. This processing uses TensorFlow and Keras. The input is the compressed data file, and the output is information about detected dirt and obstacles.

[1285] Step 4:

[1286] The server proposes the optimal cleaning method based on the results of image analysis. Based on the analyzed data, it recommends the cleaning tools, detergents, and cleaning procedures to be used. This information is generated by the cleaning suggestion module. The input is the image analysis results, and the output is a detailed cleaning procedure proposal.

[1287] Step 5:

[1288] The device notifies the user of the suggested cleaning method by displaying or reading out the required tools and cleaning steps to the user using the device's display or audio output. The input is the cleaning suggestion obtained from the server, and the output is a visual or audio notification to the user.

[1289] Step 6:

[1290] The user performs the cleaning as suggested and takes photos and videos during the process. The device sends these data to the server. The input is the image and video data after cleaning, and the output is compressed data sent to the server.

[1291] Step 7:

[1292] The server compares the data before and after cleaning and uses virtual reality technology to generate an ideal post-cleaning environment, allowing users to visually experience the cleaning results. The input is the data before and after cleaning, and the output is the generated virtual environment data.

[1293] Step 8:

[1294] The device captures the facial expressions and voice of the cleaner and sends them to the emotion recognition module. The facial expression and voice data are analyzed using emotion recognition algorithms (such as Keras). The input is the cleaner's facial expression and voice data, and the output is the analyzed emotional state.

[1295] Step 9:

[1296] The server then provides the user with advice on how to change their cleaning method or improve their motivation based on the analyzed emotional data. Specifically, it suggests taking a break as needed or showing the next cleaning step. The input is the analyzed emotional state, and the output is advice for the user.

[1297] Step 10:

[1298] The user receives advice via the terminal and continues cleaning. In this way, the system achieves effective cleaning management and reduces the burden on the person in charge.

[1299] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1300] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1301] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1302] [Fourth embodiment]

[1303] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1304] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1305] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1306] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1307] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1308] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1309] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1310] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1311] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1312] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1313] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1314] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1315] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1316] The present invention provides a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment. Specific embodiments of the present invention are described below.

[1317] System configuration

[1318] The system consists of four main parts:

[1319] 1. Data Acquisition Module

[1320] 2. Image Analysis Module

[1321] 3. Cleaning suggestion module

[1322] 4. Virtual Reality (VR) Module

[1323] Data Acquisition Module

[1324] Users can take photos and videos of the room using an application on their smartphone or a dedicated device.

[1325] The device prepares the captured data for transmission to the server, specifically by compressing the data, converting the format, and adding metadata.

[1326] Image Analysis Module

[1327] The server inputs the received data into an image analysis module and processes it to identify dirty or cluttered areas in the room.

[1328] Image analysis uses deep learning-based algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[1329] The identified information is organized and stored in a database.

[1330] Cleaning suggestion module

[1331] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[1332] The suggestions include specific cleaning procedures and essential items (e.g., vacuum cleaner, detergent, brush).

[1333] The device notifies the user of this information and displays it in a concise and easy-to-understand format.

[1334] Virtual Reality (VR) Module

[1335] The server runs a VR module to simulate the state of the room after the user has completed cleaning.

[1336] By comparing data before and after cleaning, users can visually experience the difference in a VR environment, giving them a sense of accomplishment from cleaning.

[1337] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[1338] Specific examples

[1339] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[1340] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[1341] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the cleaned up state. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[1342] In this way, the present invention aims to achieve both improved cleaning efficiency and motivation.

[1343] The processing flow will be explained below.

[1344] Step 1:

[1345] The user launches a dedicated application and takes photos and videos of the room.

[1346] Step 2:

[1347] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[1348] Step 3:

[1349] The terminal transmits the prepared data to the server.

[1350] Step 4:

[1351] The server passes the received data to the image analysis module.

[1352] Step 5:

[1353] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[1354] Step 6:

[1355] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[1356] Step 7:

[1357] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[1358] Step 8:

[1359] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[1360] Step 9:

[1361] The user cleans the room using the cleaning method presented as a reference.

[1362] Step 10:

[1363] The user takes a picture of the state of the room after cleaning again and uploads the new photo or video data to the terminal.

[1364] Step 11:

[1365] The device sends the post-cleaning data to the server.

[1366] Step 12:

[1367] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[1368] Step 13:

[1369] The virtual reality data generated by the server is sent to the terminal.

[1370] Step 14:

[1371] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[1372] Step 15:

[1373] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[1374] Example 1

[1375] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1376] Cleaning a room is labor-intensive and troublesome for many people. It is also difficult to determine which areas need cleaning and the best cleaning methods, making it extremely difficult to clean effectively. Furthermore, the effectiveness of cleaning cannot be reliably confirmed, which can lead to a decline in motivation. There is a need for a system that can solve these issues, improve cleaning efficiency, and increase motivation.

[1377] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1378] In this invention, the server includes means for identifying dirty and messy areas in a room using an image analysis model, means for storing the analysis results in a database, means for proposing cleaning methods based on the analysis results, and means for comparing data before and after cleaning and generating the state of the room after cleaning using virtual reality technology, thereby enabling the user to accurately grasp the current state of the room, practice effective cleaning methods, and virtually experience the sense of accomplishment after cleaning.

[1379] "User" refers to an individual or group that uses this system to clean a room.

[1380] "Terminal" refers to the device that the user uses to take pictures and videos of the room and send them to the server. Specifically, this applies to smartphones and dedicated terminals.

[1381] "Server" refers to a central control unit for analyzing, storing, and making suggestions about data sent by users.

[1382] "Image analysis models" refer to programs that use deep learning algorithms to identify dirty or cluttered areas in a room, such as YOLO and Mask R-CNN.

[1383] A "database" refers to a system for systematically storing and searching analysis results and related information.

[1384] "Cleaning method" refers to information on the optimal cleaning procedures and cleaning tools to be used, identified based on the analysis results.

[1385] "Virtual reality technology" refers to technology that allows users to visually experience the state of a room after cleaning, specifically 3D modeling software and VR devices.

[1386] "Data before and after cleaning" refers to images and videos showing the state of the room before and after the user cleans.

[1387] "Virtual reality environment" refers to a virtual space generated by the server that visually reproduces the state of a room after cleaning.

[1388] The present invention is a system for efficiently cleaning a room, utilizing image analysis and virtual reality technology. The system of the present invention helps users understand the current state of the room, implement effective cleaning procedures, and virtually experience the sense of accomplishment.

[1389] 1. Specific data acquisition procedures

[1390] Users use their smartphones or dedicated devices to take photos and videos of their rooms. Specifically, they launch a smartphone app and take photos of various parts of the room. The device receives this data and organizes it by compressing and converting the format. For example, it converts it to JPEG format and adds metadata (date and time of shooting, location information).

[1391] 2. Specific operations for data transmission

[1392] The device sends the data the user has acquired to the server using Wi-Fi or mobile data, encrypting the data using security protocols such as SSL.

[1393] 3. Specific procedures for image analysis

[1394] The server inputs the received data into an image analysis module and uses deep learning-based image analysis models (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. The analysis results are organized in JSON format or similar and stored in a database.

[1395] 4. Specific cleaning operations

[1396] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate effective cleaning methods based on the results of image analysis. The recommendations include specific cleaning steps and information on the tools needed. For example, it suggests that a specific detergent and brush are needed to remove stains from a carpet.

[1397] 5. Specific operation of VR simulation

[1398] After cleaning, the user takes photos and videos of the room again and uploads them to the server via their device. The server compares the data before and after cleaning and uses 3D modeling software (e.g., Blender) to generate a VR environment of the room after cleaning.

[1399] 6. Specific operations for VR experiences

[1400] The device connects to a VR device (e.g., Oculus Rift or HTC Vive) and downloads the generated VR environment data. Users can then wear VR goggles and visually experience the state of the room before and after cleaning, giving them a sense of accomplishment from cleaning.

[1401] Specific examples

[1402] For example, a user takes a photo of their living room and uploads it to their device through an application. This data is compressed, converted, and sent to a server. After analyzing the image, the server detects that there is a coffee stain on the carpet. The server then suggests using a specific detergent and brush to remove the stain from the carpet and provides a link to purchase the necessary tools. The device notifies the user, and the user completes the cleaning. By taking another photo and uploading it using the same steps, the server compares the state before and after cleaning and generates a VR environment. The device then plays this on a VR device, allowing the user to experience the virtual environment.

[1403] Prompt Sentence Examples

[1404] Below are some example prompts to input the behavior of this system into the generative AI model.

[1405] "Please explain in detail how this system uses image analysis and virtual reality technology to efficiently clean rooms. A user takes a photo of the room with their smartphone and sends it to a server via their device. The server then analyzes the image, detects dirt and clutter, and makes cleaning suggestions. After cleaning is complete, a VR module is used to compare the state before and after cleaning, allowing the user to experience the state after cleaning in a virtual environment."

[1406] This system improves the efficiency of the cleaning process while also increasing user motivation.

[1407] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1408] Step 1:

[1409] The user takes photos and videos of the room using a smartphone or dedicated device. Specifically, the user starts the app and takes multiple photos of various parts of the room from various angles. The input is the images and videos of the room, and the output is this data.

[1410] Step 2:

[1411] The device compresses the captured data, converts the format, and adds metadata. The device converts the data to JPEG or MPEG format, and adds the date and time of the capture and location information as metadata. The input is the captured image or video, and the output is the compressed and converted data.

[1412] Step 3:

[1413] The device sends data to the server. The compressed and converted data is then sent to the server using Wi-Fi or mobile data. An encryption protocol such as SSL is used during transmission. The input is the compressed and converted data, and the output is the data sent to the server.

[1414] Step 4:

[1415] The server inputs the received data into an image analysis module, which uses a deep learning-based image analysis model (e.g., YOLO, Mask R-CNN) to identify dirty or cluttered areas in the room. In the process, the server identifies dirt, stains, improper placement, etc. The input is the data sent to the server, and the output is the analysis results.

[1416] Step 5:

[1417] The server saves the analysis results in a database. The analysis results are organized in JSON format or similar and stored in the database in a format that allows for efficient search and reference. The input is the analysis results, and the output is the data saved in the database.

[1418] Step 6:

[1419] The server proposes cleaning methods based on the analysis results. Using a generative AI model (e.g., OpenAI GPT-4), it generates recommendations regarding optimal cleaning procedures and cleaning tools to use. The input is the analysis results, and the output is the recommendations.

[1420] Step 7:

[1421] The terminal notifies the user of the suggestions from the server. The terminal displays the suggested cleaning procedures and necessary tools in an easy-to-understand format. The input is the suggestions, and the output is the information notified to the user.

[1422] Step 8:

[1423] The user actually cleans based on the suggestions. The user uses the necessary tools and follows the suggested procedure to clean. The input is the suggestions and cleaning tools, and the output is the clean state after cleaning.

[1424] Step 9:

[1425] The user then takes photos or videos of the state of the room after cleaning and uploads them to the server via the device in the same way. The input is the post-cleaning images or videos, and the output is the post-cleaning data sent to the server.

[1426] Step 10:

[1427] The server compares the data before and after cleaning and generates the state of the room after cleaning using virtual reality technology. The server uses 3D modeling software (e.g., Blender) to visualize the difference between the state before and after cleaning. The input is the data before and after cleaning, and the output is the generated virtual reality environment.

[1428] Step 11:

[1429] The device provides the generated virtual reality environment to the user. The device works in conjunction with a VR device (e.g., Oculus Rift, HTC Vive) to allow the user to experience the virtual environment. The input is the generated virtual reality environment, and the output is the virtual experience provided to the user.

[1430] (Application example 1)

[1431] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] Cleaning work in factories and other physical spaces is often manual and inefficient. Particularly in large factories, it is difficult to identify dirty or messy areas and clean them efficiently, increasing the burden on workers. Furthermore, there is no way to immediately check the results of cleaning, making it difficult to visualize the effectiveness of the work. To solve these problems, a system is needed that automates cleaning work and makes it easy to check the results.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1434] In this invention, the server includes means for analyzing images or videos to detect dirt and object placement in the physical space, means for proposing areas to be cleaned and cleaning methods based on the detected dirt and placement, and means for generating the state of the physical space after cleaning using virtual reality technology. This allows for efficient automation of cleaning work in physical spaces such as factories, and enables users to immediately visually confirm the cleaning results.

[1435] An "image or video capturing means" is a device or method for capturing images or video within a physical space.

[1436] "Means for analyzing images or videos" refers to an algorithm or device for detecting the placement of dirt or objects using captured images or videos.

[1437] A "means for detecting dirt or placement" is a system or algorithm that identifies the presence of dirt or the placement of an object based on analyzed images or video.

[1438] The "means for suggesting cleaning locations and cleaning methods" is a system or method that suggests specific locations to be cleaned and appropriate cleaning methods based on detected dirt and location information.

[1439] "Means for generating the state of a physical space after cleaning using virtual reality technology" refers to a system or technology that uses virtual reality technology to simulate and visually present what a physical space will look like after cleaning.

[1440] The "means for providing a generated virtual reality environment to a user" refers to a device or platform that allows a user to experience a virtual reality environment.

[1441] "Means for a robot to capture images or videos in a physical space and transmit them to a server" refers to a system or method for a robot to acquire images or videos using a photographing device and transmit the data to a server.

[1442] The "robot means for carrying out cleaning in accordance with the provided suggestions" is a robot capable of actually carrying out the cleaning method suggested based on the image analysis results.

[1443] "Means for comparing images before and after cleaning to visually confirm the cleaning effect" refers to a system or method that compares images or videos before and after cleaning and visually shows the difference to the user.

[1444] The following system is designed as an embodiment of the present invention. The system is configured to use a robot to efficiently clean a physical space such as a factory and to visually confirm the cleaning results. The system mainly consists of the following components:

[1445] Data Acquisition Module

[1446] The robot is equipped with a camera that takes photos and videos of the factory. The robot takes images and videos as it moves around the physical space and sends them to a server. The captured data is compressed and formatted within the robot before being sent to the server.

[1447] Image Analysis Module

[1448] The server receives the transmitted image and video data and analyzes it using the image analysis module. This analysis mainly uses deep learning-based algorithms such as TensorFlow and OpenCV. As a result of the analysis, dirt and improperly placed objects within the factory are identified. The analysis results are stored in a database and used for subsequent processing.

[1449] Cleaning suggestion module

[1450] Based on the results of the image analysis module, the server proposes efficient cleaning methods and necessary tools. These proposals include specific steps, such as "If oil stains are detected, clean using a specific detergent and cloth." Links to purchase the necessary tools are also provided. This information is then sent to the robot and the user's device.

[1451] Virtual Reality (VR) Module

[1452] The server uses virtual reality technology to simulate the state of the physical space before and after cleaning. In particular, it predicts the state after cleaning and allows users to visually confirm the cleaning results. This virtual reality environment can be experienced by users through VR goggles or a corresponding application.

[1453] Specific examples

[1454] For example, if oil stains are found on the floor of a factory, the robot takes a photo of the area and sends the image to the server. The server then uses an image analysis module to detect the oil stains and a cleaning suggestion module to suggest cleaning using a specified detergent and cloth. This suggestion is sent to the robot, which then carries out the cleaning. After cleaning, the robot takes another image, and a virtual reality module compares the before and after cleaning, allowing the user to check the results using VR goggles.

[1455] Prompt Sentence Examples

[1456] "Analyze these images to identify dirt and unwanted materials in the factory and suggest effective cleaning methods."

[1457] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1458] Step 1:

[1459] As the robot moves around the factory, it uses a camera to take photos and videos. In this step, the robot acquires data from the actual environment. The input is visual data in the physical space, and the output is compressed and formatted image and video data.

[1460] Step 2:

[1461] The terminal receives the data captured by the robot and sends it to the server. The input is compressed and format-converted data, and the output is image and video data uploaded to the server.

[1462] Step 3:

[1463] The data received by the server is input into the image analysis module for analysis. The input is image and video data uploaded to the server, and the output is information on dirt and object placement in the physical space. Specifically, data calculations using a deep learning model detect dirt and identify improper object placement.

[1464] Step 4:

[1465] The server then proposes cleaning areas and methods based on the results of image analysis. The input is dirt and placement information obtained from the image analysis module, and the output is a list of specific cleaning procedures and necessary tools. Specific actions such as "if oil stains are detected, clean using a specific detergent and cloth" are generated.

[1466] Step 5:

[1467] The terminal sends the proposed cleaning content along with a prompt to the robot, which then actually performs the cleaning. The input is the cleaning proposal information from the server, and the output is the performed cleaning task. The robot cleans the physical space using the specified cleaning tools.

[1468] Step 6:

[1469] After cleaning, the robot takes images and videos again and sends them to the server. The input is visual data of the physical space after cleaning, and the output is image and video data uploaded to the server.

[1470] Step 7:

[1471] The server compares image data before and after cleaning and generates the results in a virtual reality environment. The input is image data before and after cleaning, and the output is comparative information before and after cleaning. Specifically, the before-cleaning and after-cleaning states are displayed side by side, visually highlighting the differences.

[1472] Step 8:

[1473] The device provides a virtual reality environment to the user, who then uses VR goggles to check the cleaning results. The input is comparison information from the server before and after cleaning, and the output is the user's visual confirmation experience. The user can wear the VR goggles and check the before and after cleaning status in 3D.

[1474] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1475] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention helps a user understand the current state of the room, implement effective cleaning procedures, recognize emotions during the cleaning process, and virtually experience a sense of accomplishment. Specific embodiments of the present invention are described below.

[1476] System configuration

[1477] The system is broadly divided into five main parts:

[1478] 1. Data Acquisition Module

[1479] 2. Image Analysis Module

[1480] 3. Cleaning suggestion module

[1481] 4. Virtual Reality (VR) Module

[1482] 5. Emotion Recognition Module

[1483] Data Acquisition Module

[1484] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[1485] The device compresses the captured data and prepares it for transmission to the server.

[1486] Image Analysis Module

[1487] The server passes the received data to an image analysis module, which processes it to identify dirty or cluttered areas in the room.

[1488] Image analysis uses deep learning algorithms to detect specific objects and conditions (e.g., dirt, stains, improper placement).

[1489] Cleaning suggestion module

[1490] The server provides a means to suggest effective cleaning methods based on the results from the image analysis module.

[1491] The suggestions include specific cleaning procedures and essential items.

[1492] The device notifies the user of this information and displays it in a concise, easy-to-understand format.

[1493] Virtual Reality (VR) Module

[1494] Run the VR module to simulate the state of the room after the server has finished cleaning.

[1495] Data before and after cleaning is compared, allowing users to visually experience the difference in a VR environment.

[1496] The device supports users to experience virtual environments using VR goggles and adaptive apps.

[1497] Emotion Recognition Module

[1498] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[1499] The server analyzes the acquired data and recognizes the user's emotions in real time.

[1500] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[1501] The terminal notifies the user of these advices.

[1502] Specific examples

[1503] For example, a user takes a photo of their living room and uploads it to their device through the application. This data is sent to the server and fed into the image analysis module. Image analysis reveals that there is a coffee stain on the carpet.

[1504] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[1505] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[1506] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[1507] In this way, the present invention achieves improved cleaning efficiency and motivation through comprehensive support, including emotion recognition technology.

[1508] The processing flow will be explained below.

[1509] Step 1:

[1510] The user launches a dedicated application and takes photos and videos of the room.

[1511] Step 2:

[1512] The terminal compresses the captured image or video data and prepares it for transmission to the server.

[1513] Step 3:

[1514] The terminal transmits the prepared data to the server.

[1515] Step 4:

[1516] The server passes the received data to the image analysis module.

[1517] Step 5:

[1518] The server's image analysis module uses deep learning algorithms to detect dirt, debris, and improper placement of objects in images or videos.

[1519] Step 6:

[1520] The server stores the analysis results in a database and identifies the optimal cleaning method for each stain and location.

[1521] Step 7:

[1522] The server generates a message proposing the specified cleaning method and necessary cleaning tools and transmits it to the terminal.

[1523] Step 8:

[1524] The device will notify the user of the received message and display the cleaning area, specific cleaning method, and a link to purchase recommended cleaning supplies.

[1525] Step 9:

[1526] The user starts cleaning the room according to the presented cleaning method.

[1527] Step 10:

[1528] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[1529] Step 11:

[1530] The server uses an emotion recognition module to analyze the acquired data and recognize the user's emotions in real time.

[1531] Step 12:

[1532] The server generates appropriate advice or encouraging messages according to the user's emotional state and sends them to the terminal.

[1533] Step 13:

[1534] The device notifies the user of advice and messages received and displays suggestions to increase motivation.

[1535] Step 14:

[1536] After the user has completed cleaning, they can take photos or videos again and upload the new data to the device.

[1537] Step 15:

[1538] The device sends the post-cleaning data to the server.

[1539] Step 16:

[1540] The server compares the data before and after cleaning and generates a virtual reality environment of the state of the room after cleaning.

[1541] Step 17:

[1542] The virtual reality data generated by the server is sent to the terminal.

[1543] Step 18:

[1544] The device uses VR goggles and a compatible app to allow users to visually experience the room after cleaning.

[1545] Step 19:

[1546] Through virtual reality, users can experience a sense of accomplishment after cleaning the room, which increases their motivation.

[1547] Example 2

[1548] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1549] In recent years, cleaning a room has become a burden for individuals with busy lifestyles. Maintaining motivation and choosing an efficient cleaning method are particularly difficult. Furthermore, conventional cleaning methods offer limited means for visually confirming the results of cleaning, making it difficult to achieve a sense of accomplishment. Given this background, there is a demand for a system that allows users to efficiently clean a room, maintain their motivation, and visually experience the results of cleaning.

[1550] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring images or videos of the room taken by the user; means for compressing the acquired images or videos and transmitting them to the server; means for analyzing the images or videos received by the server and identifying stains and the location of objects in the room using a deep learning algorithm; means for suggesting areas to be cleaned and cleaning methods based on the identified stains and locations; means for notifying and displaying the suggestions to the user in an easy-to-understand format; means for generating a post-cleaning state of the room using virtual reality technology and visually presenting it to the user; means for acquiring the user's facial expression, voice, and behavior data, analyzing the data, recognizing emotions, and providing a cleaning method according to the user's emotional state and advice for improving motivation based on the recognition results; and means for notifying the user of the advice based on the acquired emotional data. This not only improves cleaning efficiency but also maintains the user's motivation and enhances the sense of accomplishment after cleaning.

[1551] "User" refers to an individual who uses the system to improve cleaning efficiency and check the effectiveness.

[1552] A "terminal" refers to a smartphone or dedicated device used by a user, which captures images and videos of the room and sends them to a server.

[1553] "Server" refers to a central processing unit that receives data, analyzes it, makes recommendations, and generates the virtual reality environment.

[1554] "Image analysis" refers to the process of using deep learning algorithms to identify dirt and object placement in a room.

[1555] A "deep learning algorithm" is a type of machine learning used to perform complex image analysis and automatically detect specific objects or conditions.

[1556] "Cleaning suggestions" refers to the process of suggesting effective cleaning locations and cleaning methods to the user based on the analyzed data.

[1557] "Virtual reality technology" refers to a simulation technology that allows users to visually experience the state of a room before and after cleaning.

[1558] "Emotion recognition" refers to the process of analyzing a user's facial, vocal, and behavioral data to identify their emotional state.

[1559] "Advice" refers to guidance and recommendations on cleaning methods and motivation provided to users based on emotion recognition results.

[1560] "Compression" refers to a data processing technique for reducing the volume of captured image or video data.

[1561] "Notification" refers to the process by which a device communicates suggestions or advice to a user.

[1562] MODE FOR CARRYING OUT THE INVENTION

[1563] The present invention is a system for efficiently cleaning a room, which combines image analysis, virtual reality technology, and emotion recognition technology. The system of the present invention supports a user in understanding the current state of the room, implementing effective cleaning procedures, and virtually experiencing a sense of accomplishment while recognizing emotions during the cleaning process. Specific embodiments of the present invention are described below.

[1564] The system consists of five main parts:

[1565] 1. Data Acquisition Module

[1566] 2. Image Analysis Module

[1567] 3. Cleaning suggestion module

[1568] 4. Virtual Reality (VR) Module

[1569] 5. Emotion Recognition Module

[1570] Data Acquisition Module

[1571] User: Take photos and videos of the room using a smartphone or dedicated device application.

[1572] Terminal: Compresses the captured data and prepares it for transmission to the server.

[1573] Specifically, the user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the device records the detailed state of the room. The device then uses compression algorithms such as JPEG or H.264 to reduce the data size and sends it to the server.

[1574] Image Analysis Module

[1575] Server: Passes the received data to the image analysis module, which uses deep learning algorithms to identify dirty and cluttered areas in the room.

[1576] The server receives the data via HTTP or WebSocket, saves it to disk, and then forwards it to the analysis module, which uses machine learning frameworks such as TensorFlow and PyTorch to analyze it. The model detects dust particles and blemishes in the image and outputs their location information.

[1577] Cleaning suggestion module

[1578] Server: Based on the results from the image analysis module, it proposes effective cleaning methods.

[1579] Terminal: Informs the user of the suggestions and displays them in a concise and easy-to-understand format.

[1580] Specifically, the server evaluates the analysis results and pulls out appropriate cleaning methods from the database. For example, it generates suggestions such as "A specific detergent and brush are needed to remove stains from the carpet." These suggestions and links are then notified and displayed to the user via their device.

[1581] Virtual Reality (VR) Module

[1582] Server: Runs the VR module to simulate the state of the room after cleaning is completed.

[1583] Device: Helps users experience virtual environments using VR goggles and compatible apps.

[1584] By comparing image data before and after cleaning, a 3D model of the clean room is generated. This model is sent to the device as a data file for the VR app, and the user can visually experience the state of the room after cleaning by wearing VR goggles.

[1585] Emotion Recognition Module

[1586] Terminal: Captures the user's facial expression, voice, and behavioral data and sends it to the emotion recognition module.

[1587] Server: Analyzes the acquired data and recognizes the user's emotions in real time. Based on the emotion recognition results, it provides advice on cleaning methods and motivation that correspond to the user's emotional state.

[1588] Terminal: Informs the user of these suggestions.

[1589] Specifically, the device collects the user's facial expressions and voice through a camera and microphone and transmits them to a server in real time. An emotion recognition algorithm uses Python's OpenCV and deep learning to classify emotions, and if a user shows a tired expression, it generates advice such as "Take a break" or "The next cleaning step is light work."

[1590] Specific examples

[1591] For example, a user can take a photo of their living room and upload it to their device through the application. This data is then sent to the server and fed into the image analysis module, which can then detect a coffee stain on the carpet.

[1592] The server then suggests using a specific detergent and brush to remove the stain from the carpet, and provides links to purchase these items. The device notifies and displays these suggestions and links to the user.

[1593] While cleaning, the device captures the user's facial expressions and voice to analyze their emotions. For example, if the user looks tired, the server will provide advice such as "Take a break" or "The next cleaning step is light work."

[1594] After the user has finished cleaning, they take photos and videos again and upload them using the same process. The server compares the data before and after cleaning and generates a VR environment of the room after cleaning. The device then provides this virtual experience to the user, allowing them to visually experience the results of the cleaning.

[1595] Recommended prompt sentence examples

[1596] "Take a photo of your living room and upload it to the app. If it finds a coffee stain on your carpet, it will suggest a specific cleaner and brush to use."

[1597] In this way, the present invention achieves increased cleaning efficiency and motivation through comprehensive support including emotion recognition technology.

[1598] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1599] Step 1: Data Acquisition

[1600] The user takes photos and videos of the room using an application on a smartphone or dedicated device.

[1601] Input: Current state of the room (photo or video)

[1602] Data processing: Capture images or videos using the camera function of your smartphone or device.

[1603] Output: Captured photo or video data

[1604] Specific operation: The user presses the camera button on the app to capture a picture of the entire room. By taking multiple photos and videos, the detailed condition of the room is recorded.

[1605] Step 2: Compress data and prepare for transmission

[1606] The device compresses the captured data and prepares it for transmission to the server.

[1607] Input: Photo or video data taken

[1608] Data processing: Compressing the data using compression algorithms such as JPEG or H.264.

[1609] Output: Compressed photo or video data

[1610] Specific operation: The device compresses the captured data in the specified format and prepares to send it to the server.

[1611] Step 3: Receiving data

[1612] The server receives the compressed data sent from the terminal.

[1613] Input: Compressed photo or video data

[1614] Data processing: Receive data using HTTP or WebSocket.

[1615] Output: Received photo or video data

[1616] Specific operation: The server receives and stores data at the specified port or address.

[1617] Step 4: Image analysis

[1618] The server passes the received data to an image analysis module, which processes it to identify areas of dirt or clutter.

[1619] Input: Received photo or video data

[1620] Data Computing: Uses deep learning algorithms to detect dirt and clutter.

[1621] Output: Identification of dirty and cluttered areas (including location information)

[1622] Specific operation: The server uses machine learning frameworks such as TensorFlow and PyTorch, and the model detects dust, blemishes, etc. in the image and outputs their location information.

[1623] Step 5: Generate cleaning suggestions

[1624] The server suggests effective cleaning methods based on the results from the image analysis module.

[1625] Input: Identification of dirty and cluttered areas

[1626] Data calculation: Evaluate the analysis results and extract the appropriate cleaning method from the database.

[1627] Output: Cleaning suggestions

[1628] Specific behavior: The server retrieves cleaning methods from a database and generates suggestions, such as "a specific detergent is needed to remove stains from the carpet."

[1629] Step 6: Proposal Notification

[1630] The device will notify the user of the suggestions and display them in an easy-to-understand format.

[1631] Input: Cleaning suggestions

[1632] Data processing: Convert the proposal content into a notification format.

[1633] Output: User notification

[1634] What it does: The device will communicate suggestions to the user via push notifications or in-app displays, such as "Use the following items to remove stains from your carpet."

[1635] Step 7: Acquire data after cleaning

[1636] After the user has finished cleaning, they take photos and videos again and send them to the server.

[1637] Input: Photo or video data after cleaning

[1638] Data processing: Images or videos are taken using the camera function of a smartphone or device and compressed.

[1639] Output: Compressed cleaned photo or video data

[1640] Specific operation: The user takes a photo of the room after cleaning and sends it back to the server.

[1641] Step 8: Virtual Reality (VR) Generation

[1642] The server compares the data before and after cleaning and generates a virtual reality representation of the state of the room after cleaning.

[1643] Input: Photos or video data before and after cleaning

[1644] Data calculation: Compare image data before and after cleaning to generate a virtual reality environment.

[1645] Output: Virtual reality environment data

[1646] Specific operation: Based on the comparison results, the server generates a 3D model of the clean room and sends it to the terminal as VR data.

[1647] Step 9: Experience the virtual environment

[1648] The device supports users in experiencing virtual environments using VR goggles and compatible apps.

[1649] Input: Virtual reality environment data

[1650] Data processing: Converting VR data into a format that can be displayed on VR goggles.

[1651] Output: Virtual reality experience to the user

[1652] Specific operation: The device reads the transmitted VR data and displays it to the user through the VR goggles, allowing the user to visually experience the state before and after cleaning in a virtual reality environment.

[1653] Step 10: Acquire emotion data

[1654] The terminal acquires the user's facial expression, voice, and behavior data and sends them to the emotion recognition module.

[1655] Input: User's facial expression, voice, and behavior data

[1656] Data processing: Collect data using cameras and microphones.

[1657] Output: Obtained emotion data

[1658] Specific operation: The device uses a camera and microphone to record the user's facial expressions, voice, and actions in real time and transmits them to the server.

[1659] Step 11: Sentiment Analysis

[1660] The server analyzes the acquired emotional data and recognizes the user's emotions in real time.

[1661] Input: Acquired emotion data

[1662] Data Computing: Emotion recognition using deep learning algorithms.

[1663] Output: User's emotional state

[1664] Specific behavior: Emotion recognition algorithms analyze facial, voice, and behavioral data to classify the user's emotional state.

[1665] Step 12: Emotionally adaptive advice

[1666] Based on the emotion recognition results, the server provides advice on cleaning methods and motivation improvement according to the user's emotional state.

[1667] Input: User's emotional state

[1668] Data calculation: Generate appropriate advice according to emotional state.

[1669] Output: Personalized advice

[1670] Specific actions: For example, if the user is tired, advice such as "take a break" or "the next task is light work" is generated.

[1671] Step 13: Advice Notification

[1672] The terminal notifies the user of the advice.

[1673] Input: Personalized advice

[1674] Data processing: Convert the advice content into a notification format.

[1675] Output: Advice notice to user

[1676] Specific behavior: The device will communicate advice to the user via push notifications or in-app displays.

[1677] (Application example 2)

[1678] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1679] Conventional cleaning systems not only detect dirt and obstacles, but also lack the ability to propose efficient cleaning methods based on the detection results, or support for maintaining the motivation of cleaning staff. Effective cleaning management is also difficult because there are limited ways to visually check the status after cleaning. Especially in industrial environments, rapid detection and processing of dirt and obstacles is important, and methods for reducing fatigue and stress for cleaning staff are also required.

[1680] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images or videos of a dirty environment, means for analyzing the acquired images or videos to detect dirt and obstacles in the environment, means for proposing cleaning areas and cleaning methods based on the detected dirt and obstacles, means for generating a post-cleaning state using virtual reality technology, means for providing the generated virtual reality environment to the user, means for analyzing the facial expressions and voice of the cleaning staff and recognizing their emotions, and means for providing the user with advice on cleaning methods and motivation based on the recognized emotions. This enables effective cleaning management, reducing the burden on the staff and improving the efficiency and accuracy of cleaning.

[1681] "Images or videos of a dirty environment" means visual information, including dirt or obstacles present in a particular location or object, obtained using a camera or sensor.

[1682] "Means for analyzing captured images or videos and detecting environmental dirt or obstacles" refers to technology that uses deep learning algorithms and image processing technology to identify specific dirt or obstacles from captured visual information.

[1683] "Means for suggesting cleaning areas and cleaning methods" refers to technology that suggests optimal cleaning procedures and necessary tools to the user based on detected dirt and obstacles.

[1684] "Means for generating the post-cleaning state using virtual reality technology" refers to technology that compares the state before and after cleaning and reproduces the ideal post-cleaning environment in a virtual reality environment.

[1685] "Means for providing a generated virtual reality environment to a user" refers to technology that allows a user to visually experience the generated virtual environment using virtual reality goggles or other display devices.

[1686] "Means for analyzing the facial expressions and voice of cleaning staff and recognizing their emotions" refers to technology that acquires the facial expressions and speech of cleaning staff and analyzes that data to identify the emotional state of the staff.

[1687] "Means for providing users with advice on cleaning methods and motivation based on recognized emotions" refers to technology that provides users with appropriate changes to their cleaning methods and encouraging messages based on analyzed emotional data.

[1688] The system of the present invention aims to improve the efficiency of cleaning work in a factory environment and to increase the motivation of the person in charge. Specific embodiments of the present invention will be described below.

[1689] System Configuration

[1690] This system is composed of various modules, each of which is explained below.

[1691] Data Acquisition Module

[1692] The server uses a robot equipped with a camera to capture images or video of the environment to be cleaned. The robot collects visual data as it patrols the factory, which is then compressed and prepared for transmission to the server.

[1693] Image Analysis Module

[1694] The server analyzes the captured image or video data using deep learning algorithms, such as TensorFlow and Keras software, to identify dirt and obstacles within the factory.

[1695] Cleaning Suggestion Module

[1696] The server then proposes optimal cleaning methods based on the image analysis results, including specific instructions on the cleaning agents, brushes, and other cleaning equipment to be used. The proposal results are then sent to the user via their device.

[1697] Virtual Reality (VR) Module

[1698] The server uses virtual reality technology to generate the post-cleaning environment. It compares data before and after cleaning and simulates the effect in the VR environment. Users can experience the generated virtual environment using VR goggles.

[1699] Emotion Recognition Module

[1700] The device captures the facial expressions and voice of the cleaner and sends them to an emotion recognition module. The server then analyzes the facial and voice data using deep learning algorithms to identify the cleaner's emotional state. The software used includes a Keras model that supports emotion recognition.

[1701] Based on the emotion recognition results, the server can suggest changes to cleaning methods or provide encouraging messages to the user. For example, if fatigue is detected, the server can provide advice such as "Take a break" or "Try a lighter task next time."

[1702] Specific examples

[1703] In a specific scenario, a robot detects dirt in a factory and the server suggests appropriate cleaning methods. After the cleaning is complete, the user can visually check the cleaning results using VR goggles. Furthermore, the server analyzes the operator's facial expressions and suggests timely breaks if fatigue is detected.

[1704] Prompt Sentence Examples

[1705] I am developing a highly automated factory cleaning robot. This robot uses cameras to detect dirt in the factory and suggests the best cleaning method. It also simulates the cleaning state using VR goggles and analyzes the operator's emotions to provide real-time feedback. This system incorporates image analysis, emotion recognition, and virtual reality technology using deep learning algorithms. I would like your detailed feedback and suggestions for improvement based on this project.

[1706] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1707] Step 1:

[1708] The user starts the robot to acquire environmental data. Specifically, the camera mounted on the robot is used to acquire images and videos of the factory interior. Libraries such as OpenCV are used to acquire this data. The input is real-time video of the factory interior, and the output is a file that stores this in image or video data format.

[1709] Step 2:

[1710] The device compresses the acquired data and prepares it for transmission to the server. Specifically, it converts image and video files into a format suitable for communication and compresses them. The input is the image and video data generated in step 1, and the output is a compressed data file.

[1711] Step 3:

[1712] The server receives the compressed data and passes it to the image analysis module. The server decodes the received data and analyzes it using deep learning algorithms. This processing uses TensorFlow and Keras. The input is the compressed data file, and the output is information about detected dirt and obstacles.

[1713] Step 4:

[1714] The server proposes the optimal cleaning method based on the results of image analysis. Based on the analyzed data, it recommends the cleaning tools, detergents, and cleaning procedures to be used. This information is generated by the cleaning suggestion module. The input is the image analysis results, and the output is a detailed cleaning procedure proposal.

[1715] Step 5:

[1716] The device notifies the user of the suggested cleaning method by displaying or reading out the required tools and cleaning steps to the user using the device's display or audio output. The input is the cleaning suggestion obtained from the server, and the output is a visual or audio notification to the user.

[1717] Step 6:

[1718] The user performs the cleaning as suggested and takes photos and videos during the process. The device sends these data to the server. The input is the image and video data after cleaning, and the output is compressed data sent to the server.

[1719] Step 7:

[1720] The server compares the data before and after cleaning and uses virtual reality technology to generate an ideal post-cleaning environment, allowing users to visually experience the cleaning results. The input is the data before and after cleaning, and the output is the generated virtual environment data.

[1721] Step 8:

[1722] The device captures the facial expressions and voice of the cleaner and sends them to the emotion recognition module. The facial expression and voice data are analyzed using emotion recognition algorithms (such as Keras). The input is the cleaner's facial expression and voice data, and the output is the analyzed emotional state.

[1723] Step 9:

[1724] The server then provides the user with advice on how to change their cleaning method or improve their motivation based on the analyzed emotional data. Specifically, it suggests taking a break as needed or showing the next cleaning step. The input is the analyzed emotional state, and the output is advice for the user.

[1725] Step 10:

[1726] The user receives advice via the terminal and continues cleaning. In this way, the system achieves effective cleaning management and reduces the burden on the person in charge.

[1727] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1728] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1729] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1730] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1731] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1732] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1733] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1734] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1735] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1736] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1737] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1738] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1739] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1740] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1741] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1742] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1743] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1744] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1745] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1746] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1747] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1748] The following is further disclosed regarding the above embodiment.

[1749] (Claim 1)

[1750] a means for capturing images or video of the room;

[1751] A means for analyzing images or videos to detect dirt and object placement in a room;

[1752] A means for suggesting cleaning locations and cleaning methods based on the detected dirt and its arrangement;

[1753] A means for generating the state of the room after cleaning using virtual reality technology;

[1754] means for providing the generated virtual reality environment to a user;

[1755] A system including:

[1756] (Claim 2)

[1757] 10. The system of claim 1, further comprising means for identifying tools needed for cleaning and providing the user with a link to purchase the tools.

[1758] (Claim 3)

[1759] 10. The system of claim 1, further comprising means for recording the state after cleaning and providing a comparison of before and after cleaning using virtual reality technology.

[1760] "Example 1"

[1761] (Claim 1)

[1762] A means for a user to capture images or video of a room;

[1763] A terminal has a means for compressing image or video data, converting the format, and adding metadata;

[1764] means for the terminal to transmit data to a server;

[1765] A means for the server to identify dirty and cluttered areas in the room using an image analysis model;

[1766] A means for the server to store the analysis results in a database;

[1767] A means for the server to suggest a cleaning method based on the analysis results;

[1768] A means for the terminal to notify the user of a cleaning method;

[1769] A means for the server to compare data before and after cleaning and generate the state of the room after cleaning using virtual reality technology;

[1770] a means for providing the generated virtual reality environment to a user;

[1771] A system including:

[1772] (Claim 2)

[1773] 10. The system of claim 1, further comprising means for identifying tools needed for cleaning and providing the user with a link to purchase the tools.

[1774] (Claim 3)

[1775] 10. The system of claim 1, further comprising means for recording the state after cleaning and providing a comparison of before and after cleaning using virtual reality technology.

[1776] "Application Example 1"

[1777] (Claim 1)

[1778] a means for capturing images or video of a room or other physical space;

[1779] A means for analyzing images or videos to detect dirt and object placement in a physical space;

[1780] A means for suggesting cleaning locations and cleaning methods based on the detected dirt and its arrangement;

[1781] A means for generating the state of the physical space after cleaning using virtual reality technology;

[1782] means for providing the generated virtual reality environment to a user;

[1783] A means for the robot to take images or videos in the physical space and transmit them to a server;

[1784] robotic means for performing cleaning according to the provided suggestions;

[1785] A means to visually confirm the effectiveness of cleaning by comparing images before and after cleaning,

[1786] A system including:

[1787] (Claim 2)

[1788] 10. The system of claim 1, further comprising means for identifying equipment needed for cleaning and providing the user with a link to purchase the equipment.

[1789] (Claim 3)

[1790] 10. The system of claim 1, further comprising means for recording the state after cleaning and providing a comparison of before and after cleaning using virtual reality technology.

[1791] "Example 2: Combining Emotion Engines"

[1792] (Claim 1)

[1793] A means for acquiring images or videos of a room taken by a user;

[1794] means for compressing the captured images or videos and transmitting them to a server;

[1795] A means for the server to analyze the received images or videos and use a deep learning algorithm to identify dirt and object placement in the room;

[1796] A means for suggesting cleaning locations and cleaning methods based on the identified stains and their locations;

[1797] a means for informing and displaying the suggestions to the user in an easy-to-understand format;

[1798] A means for generating a state of the room after cleaning using virtual reality technology and visually providing the state to the user;

[1799] A means for acquiring and analyzing facial expression, voice, and behavioral data of a user to recognize emotions, and providing advice on cleaning methods and motivation improvement according to the user's emotional state based on the recognition results;

[1800] means for notifying a user of advice based on the acquired emotion data;

[1801] A system including:

[1802] (Claim 2)

[1803] 10. The system of claim 1, further comprising means for identifying equipment needed for cleaning and providing a link to the user to purchase the equipment.

[1804] (Claim 3)

[1805] 10. The system of claim 1, further comprising means for recording before and after cleaning conditions and providing a before and after cleaning comparison using virtual reality technology.

[1806] "Application example 2 when combining emotion engines"

[1807] (Claim 1)

[1808] a means for capturing images or video of the contaminated environment;

[1809] A means for analyzing the acquired images or videos to detect dirt or obstacles in the environment;

[1810] A means for suggesting cleaning locations and cleaning methods based on detected dirt and obstacles;

[1811] A means for generating a post-cleaning state using virtual reality technology;

[1812] means for providing the generated virtual reality environment to a user;

[1813] A means of analyzing the facial expressions and voices of cleaning staff and recognizing their emotions,

[1814] a means for providing the user with cleaning instructions and motivational advice based on the recognized emotions;

[1815] A system including:

[1816] (Claim 2)

[1817] 10. The system of claim 1, further comprising means for identifying tools needed for cleaning and providing the user with a link to purchase the tools.

[1818] (Claim 3)

[1819] 10. The system of claim 1, further comprising means for recording the state after cleaning and providing a comparison of before and after cleaning using virtual reality technology. [Explanation of symbols]

[1820] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing images or video of the room; A means for analyzing images or videos to detect dirt and object placement in a room; A means for suggesting cleaning locations and cleaning methods based on the detected dirt and its arrangement; A means for generating the state of the room after cleaning using virtual reality technology; means for providing the generated virtual reality environment to a user; A system including:

2. The system of claim 1 further comprising means for identifying tools needed for cleaning and providing the user with a link to purchase the tools.

3. 10. The system of claim 1, further comprising means for recording the state after cleaning and providing a comparison of before and after cleaning using virtual reality technology.

Images