System

Abstract

A system is provided.SOLUTION: Means for photographing an initial state of a room, means for storing a photographed image of the initial state of the room in a storage device, means for periodically photographing the entire room, means for uploading a newly photographed image of the room to the storage device, means for comparing the newly photographed image of the room with the image of the initial state, means for digitizing a degree of messiness of the room, and means for generating a notification for notifying a timing of cleaning when the digitized degree of messiness exceeds a Iga value. A system comprising: means for transmitting advice to a user terminal; means for receiving a request for advice on cleaning from a user; and means for providing appropriate advice on cleaning based on the received request.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People who tend to neglect cleaning their rooms due to the hustle and bustle of daily life, or who are not good at tidying up, have difficulty judging the level of clutter and are unable to clean at the appropriate time. This makes it difficult to maintain the cleanliness of their rooms. Furthermore, they often feel burdened because they do not know how to specifically proceed with cleaning and tidying up. The purpose of this invention is to solve such problems and provide a system for efficiently maintaining room cleanliness. [Means for solving the problem]

[0005] This invention is a system including means for photographing an initial state of a room, means for saving the photographed image of the room in the initial state in a storage device, means for periodically photographing the entire room, means for uploading newly photographed images of the room to the storage device, means for comparing the newly photographed image of the room with the image of the initial state, means for quantifying the degree of clutter in the room, means for generating and transmitting to a user terminal a notification informing the user that it is time to clean when the quantified degree of clutter exceeds a threshold, means for accepting requests for cleaning advice from a user, and means for providing appropriate cleaning advice based on the accepted request. This system allows a user to effectively maintain the cleanliness of their room and also receive specific cleaning advice.

[0006] The "means for photographing the initial state of the room" is a device or function for photographing the clean state of the room with a camera.

[0007] The "means for saving the photographed image of the room in its initial state in a storage device" is a device or function for saving the photographed image in a storage device as digital data.

[0008] The "means for periodically photographing the entire room" is a device or function for automatically photographing the entire room at set time intervals.

[0009] The "means for uploading newly captured images of the room to a storage device" refers to a device or function for sending newly acquired images to a cloud or local storage device and storing them therein.

[0010] The "means for comparing a newly captured image of the room with an image of the initial state" refers to a device or function for comparing the latest image with the initial image through computational processing and detecting the differences therebetween.

[0011] The "means for quantifying the degree of messiness of a room" is a device or function for expressing the degree of messiness of a room as a quantitative number based on the comparison results.

[0012] "Means for generating a notification informing the user that it is time to clean when the quantified level of clutter exceeds a threshold and sending it to the user's terminal" refers to a device or function that generates a notification to inform the user of the need to clean when the level of clutter exceeds a set standard and sends it to the user's terminal.

[0013] The "means for accepting a cleaning advice request from a user" is a device or function that allows the system to receive a request from a user inquiring about a cleaning method.

[0014] The "means for providing appropriate cleaning advice based on the received request" is a device or function for processing an advice request from a user and providing the user with appropriate cleaning methods and tips in response. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that automatically determines the messiness of a room and prompts a user to clean at an appropriate time. Specific embodiments of the system will be described below.

[0037] System Configuration

[0038] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[0039] Registering the initial state of the room

[0040] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0041] Regularly taking photos of the room

[0042] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0043] Analyzing images and determining clutter levels

[0044] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0045] Sending cleaning notifications

[0046] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[0047] Providing cleaning advice

[0048] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[0049] Specific examples

[0050] Registering the initial state

[0051] User: Clean up the living room and take a photo with your smartphone camera.

[0052] Device: Uploads captured images to the cloud server.

[0053] Server: Save the image and set it as a clean reference image.

[0054] Regular imaging and analysis

[0055] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0056] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[0057] Notifications when clutter exceeds thresholds

[0058] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0059] Device: Show the user the message "The living room is messy. Time to clean!"

[0060] Providing cleaning advice

[0061] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[0062] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[0063] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[0064] The processing flow will be explained below.

[0065] Registering the initial state of the room

[0066] Step 1:

[0067] User: Clean up the living room and take a picture with the camera.

[0068] The user takes an image of the room using a smartphone or a camera on the device.

[0069] Step 2:

[0070] Device: Uploads captured images to the cloud server.

[0071] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[0072] Step 3:

[0073] Server: Saves the uploaded image and sets it as the reference image.

[0074] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[0075] Regularly photograph and analyze the room conditions

[0076] Step 4:

[0077] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[0078] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[0079] Step 5:

[0080] Device: Automatically take a photo of the living room at the set time.

[0081] The camera will automatically start up and capture the entire room.

[0082] Step 6:

[0083] Device: Uploads newly captured images to the cloud server.

[0084] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[0085] Step 7:

[0086] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[0087] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[0088] Step 8:

[0089] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[0090] A comparison algorithm quantifies the clutter level of a room.

[0091] Sending cleaning notifications

[0092] Step 9:

[0093] Server: Compare the calculated clutter level with a threshold.

[0094] Check whether the quantified clutter level exceeds a set threshold.

[0095] Step 10:

[0096] Server: Generates cleaning notification if threshold is exceeded.

[0097] A notification message is generated and the message content and recipient information are added.

[0098] Step 11:

[0099] Server: Sends notifications to the user device.

[0100] Use the push notification system to send notification messages to the user's device.

[0101] Step 12:

[0102] Terminal: Show cleaning notification to user.

[0103] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[0104] Providing cleaning advice

[0105] Step 13:

[0106] User: Uses the device's chat function to ask about cleaning tips.

[0107] The user opens the chat screen, types a question, and sends it to the server.

[0108] Step 14:

[0109] Server: Receives questions from users and searches for appropriate advice.

[0110] The server searches the database for relevant cleaning advice and prepares a response.

[0111] Step 15:

[0112] Server: Sends advice to users via chat function.

[0113] Appropriate advice content is sent to the terminal as a chat message.

[0114] Step 16:

[0115] User: Clean according to the advice provided.

[0116] The user performs cleaning based on the advice received.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] In modern life, efficiently cleaning a room amidst busy daily routines is a difficult task. In particular, accurately understanding the state of clutter in a room and cleaning at the optimal time can be a burden for many people. Conventional methods require users to check the state of the room themselves and determine the need for cleaning each time, which is a very time-consuming operation. Therefore, there is a need for a system that solves these problems and allows users to keep their rooms clean efficiently and comfortably.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes a means for quantifying the degree of messiness of a room, a means for using image processing AI to recognize objects, detect dust and dirt, and analyze changes in the placement of objects, a means for notifying the user of the need for cleaning through a notification function, and a means for providing advice in response to the user's questions through a chat function, thereby enabling the user to automatically grasp the degree of messiness of the room and clean at the appropriate time.

[0122] The "initial state of the room" indicates a state in which the room is neat and tidy, and is an image that serves as a reference for evaluating the degree of messiness in the future.

[0123] The "means for periodically photographing the entire room" is a function for automatically acquiring images of the room at set time intervals.

[0124] "Image of initial state" refers to an image of the room's initial state that is captured and stored on a cloud server.

[0125] "Quantifying the degree of clutter" is the process of analyzing the clutter level of a room and expressing it as a number according to certain standards.

[0126] "Notification when threshold is exceeded" means generating and sending a notification to the user informing them of the need to clean when the quantified clutter level exceeds a set reference value.

[0127] "Image processing AI" is an artificial intelligence technology that analyzes captured images to recognize objects, detect dirt, and determine changes in the placement of objects.

[0128] "Object recognition" is a technology that identifies objects in an image and determines their location and type.

[0129] "Dust and dirt detection" is a technology that identifies dust and dirt in an image and confirms their presence.

[0130] The "notification function" is a function that sends messages to notify the user of specific events or actions.

[0131] The "chat function" is a function that allows users to communicate with the server in real time via text messages.

[0132] "Providing cleaning advice" refers to suggesting optimal cleaning methods and ideas in response to the user's questions and requests.

[0133] The present invention is a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time, and specific embodiments of the system will be described below.

[0134] System Configuration

[0135] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[0136] Registering the initial state of the room

[0137] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0138] Regularly taking photos of the room

[0139] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0140] Analyzing images and determining clutter levels

[0141] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0142] Sending cleaning notifications

[0143] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[0144] Providing cleaning advice

[0145] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[0146] Specific examples

[0147] Registering the initial state

[0148] User: Clean up the living room and take a photo with your smartphone camera.

[0149] Device: Uploads captured images to the cloud server.

[0150] Server: Save the image and set it as a clean reference image.

[0151] Regular imaging and analysis

[0152] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0153] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[0154] Notifications when clutter exceeds thresholds

[0155] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0156] Device: Show the user the message "The living room is messy. Time to clean!"

[0157] Providing cleaning advice

[0158] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[0159] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[0160] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[0161] Prompt Sentence Examples

[0162] Below is an example of a prompt sentence to input into the image processing AI model.

[0163] Compare the initial image with the new room image and quantify the clutter level. Evaluate based on object recognition, dust / dirt detection, and changes in object placement, and generate a notification if the clutter level exceeds a threshold.

[0164] These embodiments allow the user to automatically understand the messiness of the room and clean it at the appropriate time.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1:

[0167] Initial image registration

[0168] User: Prepare a clean room. Specifically, clean the room and make sure everything is neat and tidy. In this state, start the smartphone camera and take a picture of the entire room.

[0169] Input: Image of the initial room state

[0170] Terminal: Uploads images taken by the user to the cloud server.

[0171] Output: Images uploaded to the cloud server

[0172] Server: Receives the uploaded image and saves it in a storage device as an "initial image."

[0173] Output: Saved initial image

[0174] Step 2:

[0175] Regular imaging

[0176] Device: Automatically activate the camera at a specified time according to a pre-set schedule, for example, 10:00 AM every day.

[0177] Input: Specified schedule

[0178] Terminal: Take a photo of the entire room again to obtain a new image of the room.

[0179] Output: Newly acquired room image

[0180] Step 3:

[0181] Uploading an image

[0182] Device: Newly acquired images are automatically uploaded to the cloud server.

[0183] Input: A newly acquired image of a room.

[0184] Output: New image uploaded to the cloud server

[0185] Server: Receives new images sent from the device and temporarily stores them in a storage device.

[0186] Output: New image saved

[0187] Step 4:

[0188] Determining the level of clutter

[0189] Server: Starts the process of comparing the new image with the initial image.

[0190] Input: New image, initial image

[0191] Server: Uses image processing AI to recognize objects, detect dust and dirt, and analyze changes in object placement.

[0192] Data processing: object recognition, dust and dirt detection, analysis of changes in object placement

[0193] Server: Based on the analysis results, the degree of clutter is quantified and it is determined whether this number exceeds a preset threshold.

[0194] Data calculation: Quantifying clutter level and determining threshold

[0195] Output: Numerical value of clutter level and threshold exceedance judgment

[0196] Step 5:

[0197] Sending cleaning notifications

[0198] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0199] Input: Threshold exceedance judgment

[0200] Server: Sends notifications to the user's device.

[0201] Output: Cleaning timing notification

[0202] Device: Receives a notification from the server and displays the message "Your room is messy. Time to clean!" to the user.

[0203] Output: The message that is displayed to the user

[0204] Step 6:

[0205] Providing cleaning advice

[0206] User: If necessary, use the chat function on the device to type in a cleaning question. For example, "How can I efficiently clean the dust off the floor?"

[0207] Input: User question

[0208] Terminal: Sends the user's question to the cloud server.

[0209] Output: Questions sent to the cloud server

[0210] Server: Searches for pre-registered cleaning advice information based on the question.

[0211] Data processing: Search for advice information based on the question

[0212] Server: Creates optimal advice as a text message and sends it to the user's device.

[0213] Output: Advice in the form of a text message

[0214] Terminal: Receives advice from the server and displays it to the user. For example, it displays a message such as "It would be a good idea to use a microfiber cloth."

[0215] Output: An advisory message that is displayed to the user.

[0216] This is the specific process flow of this system. By using this system, users can automatically determine the messiness of their room and clean it at the appropriate time.

[0217] (Application example 1)

[0218] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0219] Regular cleaning is essential to maintaining cleanliness in physical stores. However, store staff often miss cleaning opportunities or are unfamiliar with the optimal cleaning methods. This can result in a deterioration of the store environment, lower customer satisfaction, and hygiene issues. Therefore, there is a need for a system that can automatically and efficiently manage cleaning in physical stores and prompt cleaning at the appropriate times.

[0220] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0221] In this invention, the server includes a means for periodically taking images with a camera installed in a specific area of ​​the store and uploading them to a cloud server, a means for an image processing AI on the cloud server to evaluate the degree of clutter and generate a notification as needed and send it to a user terminal, a means for accepting cleaning advice requests from users, and a means for providing appropriate cleaning advice based on the accepted request. This makes it possible to constantly monitor the environment within the store and clean at the appropriate time.

[0222] The "initial state" is a reference image that shows the ideal clean state of a room or store.

[0223] A "storage device" is a device that stores data, and includes cloud servers and the like.

[0224] A "cloud server" is a server that stores and processes data via the Internet.

[0225] "Image processing AI" is an artificial intelligence technology that analyzes images and extracts information for specific purposes.

[0226] "Camera" means a device that captures images or videos, including those built into smartphones and dedicated fixed cameras.

[0227] "Clutter level" is a numerical indication of the cleanliness and tidiness of a room or store.

[0228] A "threshold" is a reference value set for determining a specific condition.

[0229] A "notification" is a message that notifies the user of specific information.

[0230] A "user terminal" is a device that is directly operated by a user, and includes smartphones and tablets.

[0231] An "advice request" is a request by a user for specific information or advice.

[0232] "Cleaning advice" means providing information on cleaning methods and cleaning tips.

[0233] A "specific area" is a specific area within the store that is to be cleaned.

[0234] This invention is a system for streamlining cleaning management in brick-and-mortar stores, and uses a user terminal, a cloud server, and a camera as key components. This system is designed to determine the clutter level of a room and prompt cleaning at the appropriate time. Specific embodiments of the system are described below.

[0235] First, store staff take a photo of a specific area of ​​the store after cleaning and upload the image to a cloud server as an "initial image." This image shows the ideal cleanliness of the store and serves as a standard for future comparisons.

[0236] The system is then scheduled to periodically capture images of specific areas of the store. For example, every day at 2:00 p.m., the camera automatically captures images of the area and uploads the newly captured images to a cloud server, where they are analyzed using image processing AI.

[0237] The cloud server compares the newly uploaded image with the initial image and quantifies it based on object recognition and changes in clutter level. If this quantification exceeds a pre-defined threshold, the cloud server generates a notification and sends a message to the user device, such as "A specific area is cluttered. Time to clean!"

[0238] The user device also has a chat function, allowing store staff to ask questions about cleaning methods. For example, by entering a prompt such as, "Please tell me how to improve the cleaning efficiency of the cash register counter," appropriate cleaning advice is provided from the cloud server's knowledge base. The server then sends specific advice to the device, such as, "It is effective to disinfect the cash register counter once a week."

[0239] Hardware and software used

[0240] Hardware:

[0241] Camera: A camera built into a smartphone or a fixed camera installed in a store.

[0242] User devices: smartphones, tablet devices.

[0243] software:

[0244] Cloud server: Stores and processes data.

[0245] Image processing AI: Analyzes images and quantifies the degree of clutter.

[0246] Notification system: Sends messages to user terminals.

[0247] Chatbot: Provides cleaning advice in response to user questions.

[0248] Specific examples

[0249] Store staff clean the cash register area every day at 8:00 a.m., then take a photo of the area with their smartphone and upload it to a cloud server.

[0250] The system takes an up-to-date photo of the checkout area every day at 2 p.m. and uploads it to a cloud server. Image processing AI evaluates the clutter level and sends notifications as necessary.

[0251] Staff will clean the area again based on the notification they receive.

[0252] Prompt Sentence Examples

[0253] "Please tell me how to improve the cleaning efficiency of the cash register counter."

[0254] "What is the best way to respond if the seating area is messy?"

[0255] This system allows physical stores to maintain a clean environment at all times, contributing to improved customer satisfaction and hygiene.

[0256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0257] Step 1:

[0258] A user cleans a specific area of ​​the store and takes a picture of that area with a camera. The input is an image taken by the camera, which is uploaded to a cloud server. The server stores the received image in a storage device as an "initial state image." This sets a reference for the initial state.

[0259] Step 2:

[0260] The device periodically takes photos of specific areas of the store according to a set schedule, and the new images are input and uploaded to a cloud server, which uses image processing AI to analyze the new images and generate data for object recognition and clutter assessment.

[0261] Step 3:

[0262] The cloud server uses image processing AI to compare the new image with the initial image. The input is the new image and the initial image, which are compared to quantify the degree of clutter. This process outputs the clutter level as a number.

[0263] Step 4:

[0264] The server determines whether the numerically-quantified clutter level exceeds a set threshold. The input is the numerically-quantified clutter level, which is evaluated. If the threshold is exceeded, the server generates a notification informing the user that it is time to clean and sends it to the user's device. This notification is then displayed on the user's device.

[0265] Step 5:

[0266] After receiving the notification, the user uses the chat function on the device to request cleaning advice from the server. The input is a question (prompt sentence) from the user. The server searches for appropriate advice from a pre-registered knowledge base and generates this information.

[0267] Step 6:

[0268] The server sends the generated cleaning advice to the user's device. The input is the knowledge base information in the server and the user's question, and the advice generated based on this is output. The advice is displayed on the user's device, and the physical store is cleaned efficiently based on this.

[0269] Following these steps will help keep your store environment clean, leading to more efficient cleaning and improved customer satisfaction.

[0270] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0271] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0272] System Configuration

[0273] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[0274] Registering the initial state of the room

[0275] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0276] Regularly taking photos of the room

[0277] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0278] Analyzing images and determining clutter levels

[0279] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0280] Emotion Engine Operation

[0281] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[0282] Sending cleaning notifications

[0283] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[0284] Providing cleaning advice

[0285] If a user has trouble with cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "How can I efficiently clean the dust off the floor?" the server might send advice such as "It would be good to use a microfiber cloth" to the device. The emotion engine can also provide specific advice for reducing stress by taking into account the user's stress level.

[0286] Specific examples

[0287] Registering the initial state

[0288] User: Clean up the living room and take a photo with your smartphone camera.

[0289] Device: Uploads captured images to the cloud server.

[0290] Server: Save the image and set it as a clean reference image.

[0291] Regular imaging and analysis

[0292] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0293] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[0294] Notifications when clutter exceeds thresholds

[0295] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0296] Terminal: Shows the user a message saying "The living room is messy. Time to clean!" The message may change depending on the emotion recognized by the emotion engine.

[0297] Providing cleaning advice

[0298] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[0299] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it." The emotion engine provides specific advice for reducing stress according to the user's stress level.

[0300] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[0301] The processing flow will be explained below.

[0302] Registering the initial state of the room

[0303] Step 1:

[0304] User: Clean up the living room and take a picture with the camera.

[0305] The user takes an image of the room using a smartphone or a camera on the device.

[0306] Step 2:

[0307] Device: Uploads captured images to the cloud server.

[0308] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[0309] Step 3:

[0310] Server: Saves the uploaded image and sets it as the reference image.

[0311] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[0312] Regularly photograph and analyze the room conditions

[0313] Step 4:

[0314] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[0315] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[0316] Step 5:

[0317] Device: Automatically take a photo of the living room at the set time.

[0318] The camera will automatically start up and capture the entire room.

[0319] Step 6:

[0320] Device: Uploads newly captured images to the cloud server.

[0321] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[0322] Step 7:

[0323] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[0324] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[0325] Step 8:

[0326] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[0327] A comparison algorithm quantifies the clutter level of a room.

[0328] Sending cleaning notifications

[0329] Step 9:

[0330] Server: Compare the calculated clutter level with a threshold.

[0331] Check whether the quantified clutter level exceeds a set threshold.

[0332] Step 10:

[0333] Server: Generates cleaning notification if threshold is exceeded.

[0334] A notification message is generated and the message content and recipient information are added.

[0335] Step 11:

[0336] Server: Sends notifications to the user device.

[0337] Use the push notification system to send notification messages to the user's device.

[0338] Step 12:

[0339] Terminal: Show cleaning notification to user.

[0340] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[0341] Emotion Engine Operation

[0342] Step 13:

[0343] Terminal: Collects the user's facial expressions and voice.

[0344] The device's camera and microphone are used to capture the user's facial expressions and voice data.

[0345] Step 14:

[0346] Server: The emotion engine analyzes the acquired data and recognizes the user's emotion.

[0347] The emotion engine uses image and audio analysis to identify the user's emotions (e.g., stress, joy, fatigue).

[0348] Step 15:

[0349] Server: Adaptively change the content of cleaning notifications based on the user's emotions.

[0350] If the user is tired, an encouraging message such as "Let's clean up and refresh ourselves" is generated.

[0351] Providing cleaning advice

[0352] Step 16:

[0353] User: Uses the device's chat function to ask about cleaning tips.

[0354] The user opens the chat screen, types a question, and sends it to the server.

[0355] Step 17:

[0356] Server: Receives questions from users and searches for appropriate advice.

[0357] The server searches the database for relevant cleaning advice and prepares a response.

[0358] Step 18:

[0359] Server: Sends advice to users via chat function.

[0360] Appropriate advice is sent to the device as a chat message. The emotion engine provides specific advice for reducing stress according to the user's stress level.

[0361] Step 19:

[0362] User: Clean according to the advice provided.

[0363] The user performs cleaning based on the advice received.

[0364] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[0365] Example 2

[0366] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0367] In today's busy living environment, there is a need for a system that helps users clean their rooms at appropriate times. However, typical cleaning support systems send uniform notifications without considering the user's emotional state, which can be annoying for users. Furthermore, they lack the functionality to provide specific cleaning advice, and therefore do not provide sufficient support for users to effectively organize their rooms. To solve this problem, a system is needed that recognizes the user's emotions, adaptively changes cleaning notifications, and provides specific cleaning advice.

[0368] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0369] In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state using image processing means, means for performing object recognition and quantifying the degree of messiness of the room, means for generating a notification informing the user that it is time to clean when the quantified degree of messiness exceeds a threshold and sending the notification to the user terminal, means for recognizing the user's emotions, means for adaptively changing the content of the notification based on the user's emotions, means for receiving a request for cleaning advice from the user, and means for providing appropriate cleaning advice based on the received request. This allows the user to clean their room efficiently, receive cleaning notifications at appropriate times, and further allows for flexible responses according to the user's emotional state.

[0370] The "means for capturing an image of the initial state of the room" refers to a device or method for capturing an image of the room in its tidy state.

[0371] The "means for saving the captured image of the room in its initial state in a storage device" refers to a device or process for saving the captured image data of the initial state.

[0372] "Means for periodically photographing the entire room" refers to a device or method for automatically or manually photographing the entire room based on a pre-set schedule.

[0373] "Means for uploading newly captured images of the room to a storage device" refers to a device or process that transfers the latest captured image data to a cloud or other storage system.

[0374] "Means for comparing a newly captured image of the room with an image of the initial state using image processing means" refers to an algorithm or device for comparing an image of the initial state with the latest image and detecting differences.

[0375] "Means for performing object recognition and quantifying the degree of clutter in a room" refers to a process or device that uses image processing technology to analyze the presence and placement of objects, and then expresses the degree of clutter as a number based on the results.

[0376] "Means for generating and sending to a user device a notification informing the user that it is time to clean when the quantified clutter level exceeds a threshold" refers to a system or method for creating and sending to a user device a notification encouraging cleaning when the clutter level exceeds a pre-set threshold.

[0377] "Means for recognizing user emotions" refers to technology or devices that analyze a user's facial expressions, voice, text input, etc. to determine the user's emotional state.

[0378] "Means for adaptively changing notification content based on user emotion" refers to a system or process for changing the content or tone of a notification based on a recognized user emotion.

[0379] "Means for accepting cleaning advice requests from users" refers to an interface or method for accepting input from users asking questions or requesting advice about cleaning.

[0380] "Means for providing appropriate cleaning advice based on a received request" refers to a system or method for providing appropriate cleaning advice based on information and knowledge prepared in advance in response to a received user request.

[0381] MODE FOR CARRYING OUT THE INVENTION

[0382] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0383] System Configuration

[0384] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[0385] Registering the initial state of the room

[0386] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0387] Examples:

[0388] User: Tidy up the living room and take a photo with their smartphone camera.

[0389] Device: Upload the captured images to the cloud server.

[0390] Server: Save the image and set it as a clean reference image.

[0391] Regularly taking photos of the room

[0392] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0393] Examples:

[0394] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0395] Analyzing images and determining clutter levels

[0396] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. Software used here includes TensorFlow and OpenCV. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a preset threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0397] Examples:

[0398] Server: Image processing AI compares the new image with the initial image and calculates the degree of clutter.

[0399] Server: When the clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[0400] Emotion Engine Operation

[0401] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[0402] Examples:

[0403] Server: Adaptively change the content of notifications for users based on emotions recognized by the emotion engine.

[0404] Sending cleaning notifications

[0405] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[0406] Examples:

[0407] Device: Displays received notifications to the user and plays notifications and alarms as needed.

[0408] Providing cleaning advice

[0409] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. The emotion engine also takes the user's stress level into consideration and provides specific advice for stress reduction.

[0410] Examples:

[0411] User: Uses the chat function on their device to ask the server for cleaning tips.

[0412] Server: Responds to the user's question with advice such as "It may be effective to remove the cushion cover before washing."

[0413] Prompt Sentence Examples

[0414] Example prompt for initial registration:

[0415] After you have tidied up your living room, take a picture with your smartphone camera and upload the image to the cloud server.

[0416] Cleaning notification prompt example:

[0417] Generate a cleaning notification message when the emotion engine recognizes that the user is tired. Example: Clean up and refresh yourself.

[0418] Example cleaning advice prompts:

[0419] If a user asks how to efficiently dust the floor, provide appropriate advice taking into account the results of the sentiment engine. For example: Using a microfiber cloth is a good idea.

[0420] In this way, the system allows users to clean their rooms efficiently, receive timely notifications, and respond flexibly to the user's emotions.

[0421] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0422] Step 1:

[0423] User: Take a picture of the initial state of the room with the camera.

[0424] Input: A clean image of a room.

[0425] Output: Pristine image data.

[0426] Specific operation: The user opens the camera app on their smartphone, clicks the "Register initial state" button, and takes a photo of the room.

[0427] Step 2:

[0428] Device: Upload the captured initial image to the cloud server.

[0429] Input: Pristine image data.

[0430] Output: Image data stored on a cloud server.

[0431] What it does: It establishes an internet connection to instantly upload images taken by the smartphone to a cloud server.

[0432] Step 3:

[0433] Server: Saves the received image data in its initial state in a storage device.

[0434] Input: Image data uploaded to the cloud server.

[0435] Output: Pristine image data stored in a database.

[0436] Specific operation: The cloud server receives the image data and stores it in a database.

[0437] Step 4:

[0438] Device: Automatically activate the camera at a set time (e.g., 10:00 AM every day) and capture images of the room.

[0439] Input: Scheduled timing.

[0440] Output: Image data of the new room.

[0441] Specific operation: The device will start the camera at 10:00 AM every day and automatically take pictures of the room.

[0442] Step 5:

[0443] On your device: Upload any new images you take to the cloud server.

[0444] Input: Image data of the new room.

[0445] Output: New image data stored on the cloud server.

[0446] Specific operation: Data transfer is performed to upload newly captured images to the cloud server.

[0447] Step 6:

[0448] Server: Compares the new image with the initial image using image processing techniques.

[0449] Input: New image data and initial image data.

[0450] Output: Image comparison results.

[0451] How it works: The cloud server uses image processing AI to compare the new image with the initial image using tools such as TensorFlow and OpenCV.

[0452] Step 7:

[0453] Server: Performs object recognition and quantifies the degree of clutter in a room.

[0454] Input: Image comparison results.

[0455] Output: Quantified clutter level.

[0456] What it does: It uses an object recognition algorithm to quantify the level of clutter based on the placement of objects, the presence of dust and dirt, etc.

[0457] Step 8:

[0458] Server: When the quantified clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[0459] Input: Quantified clutter level.

[0460] Output: Notification that it's time to clean.

[0461] Specific behavior: When the clutter level exceeds a threshold, the server creates a notification and sends it to the user's device.

[0462] Step 9:

[0463] Server: Recognizes user emotions.

[0464] Input: User facial, voice, and text input data.

[0465] Output: User emotion recognition results.

[0466] Specific operation: Using the emotion engine, facial expression recognition and voice analysis are performed to determine the user's emotions.

[0467] Step 10:

[0468] Server: Adaptively change notification content based on user emotions.

[0469] Input: User emotion recognition results.

[0470] Output: Notification content tailored based on sentiment.

[0471] Specific behavior: Depending on the recognized user emotion, for example, if it is determined that the user is "tired," a message such as "Let's clean up and refresh" is generated.

[0472] Step 11:

[0473] Terminal: Displays notifications sent from the cloud server to the user.

[0474] Input: Notification from the cloud server.

[0475] Output: A notification message that will be displayed on the user's terminal.

[0476] Specific behavior: The device displays the received notification to the user and plays a notification sound or alarm if necessary.

[0477] Step 12:

[0478] User: Uses the chat function to ask the server for cleaning tips.

[0479] Input: The question asked by the user.

[0480] Output: Query data to the server.

[0481] Specific behavior: The user opens the chat function on their device and types a question about cleaning.

[0482] Step 13:

[0483] Server: Provides appropriate cleaning advice based on the user's request.

[0484] Input: The question asked by the user.

[0485] Output: Cleaning advice message.

[0486] Specific operation: The server generates specific advice for the user's question based on pre-registered information and replies via the chat function.

[0487] (Application example 2)

[0488] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0489] In conventional virtual stores, managers had to manually check the level of clutter and tidiness within the store and periodically tidy up. However, this process required time and effort, and it was difficult for the manager to respond appropriately based on their emotions and the situation. Furthermore, in order to improve the user experience, a system was needed that could automatically determine the level of tidiness in the virtual space and perform efficient maintenance.

[0490] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state, means for quantifying the degree of clutter in the room, means for generating a notification informing the user that cleaning is necessary when the quantified degree of clutter exceeds a threshold and sending the notification to the user terminal, means for accepting a request for cleaning advice from a user, means for providing appropriate cleaning advice based on the accepted request, means for recognizing the user's emotions and changing the content of the notification based on the emotions, and means for analyzing the organization of structures in the virtual space. This enables the server to automatically determine the degree of clutter in a virtual store, efficiently issue maintenance notifications, and appropriately respond to the manager's emotions and circumstances.

[0491] The "initial state of the room" is an image of a tidy and tidy room taken when the user first registers with the system.

[0492] A "storage device" is a hardware component of a computer for persistently storing data.

[0493] The "means for taking regular photographs" refers to a function or device that automatically takes photographs of the room based on a set schedule.

[0494] A "newly captured image of the room" is an image that is periodically captured and shows the latest state of the room.

[0495] "Means for uploading images to a storage device" refers to a process or system that sends captured images to a storage device such as a cloud or server.

[0496] The "level of clutter in a room" is an index that quantifies the degree of clutter, dirt, etc., compared to the initial state of the room.

[0497] "Means of quantifying" refers to algorithms or programs that use image processing technology to express the tidiness of a room as a number.

[0498] A "user terminal" is a device such as a smartphone, tablet, or PC that a user uses to access the system.

[0499] "Means for accepting requests for cleaning advice" refers to a function or chat system that allows users to send questions about cleaning methods and tips.

[0500] The "means for providing appropriate cleaning advice" refers to algorithms and databases that provide appropriate cleaning methods in response to user questions.

[0501] "Means for recognizing emotions and changing notification content based on emotions" refers to a system that analyzes emotions from the user's facial expressions and voice and adjusts the content of notifications accordingly.

[0502] A "virtual space" is a virtual space reproduced on a computer, and is a digital environment in which users can interact.

[0503] The "means for analyzing the organization of structures" is a system that analyzes the arrangement of items and objects placed in a virtual space and evaluates their organization.

[0504] System configuration

[0505] The system of this invention is composed of a user terminal, a cloud server, a camera, and an emotion engine. The user terminal is a device such as a smartphone, tablet, or PC, and the camera may be built into these devices or an external camera may be used. The cloud server is equipped with an image processing AI and an emotion engine.

[0506] Registering the initial state

[0507] Users organize their virtual store and take a photo of it. This "initial image" represents the ideal state and is uploaded to a cloud server and stored in a storage device. This serves as a baseline for future comparison.

[0508] Regular status checks

[0509] The system is scheduled to periodically take photos of the entire virtual store. The user's device automatically starts the camera at the set time, acquires new images, and uploads them to the cloud server. This schedule can be freely adjusted to suit the user's convenience.

[0510] Analyzing images and determining clutter levels

[0511] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to let the user know it's time to clean.

[0512] Emotion Engine Operation

[0513] The emotion engine recognizes the user's emotions by analyzing their facial expressions, voice data, and input data. For example, if the user is tired, it generates a gentle message such as, "Your virtual store is messy. Why don't you take a break and then tidy it up?"

[0514] Providing notice and advice

[0515] The user's device receives notifications sent from the cloud server and displays the message to the user. If the user has a question about cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information. The emotion engine also works in conjunction with this to recommend advice tailored to the user's emotions and stress level.

[0516] Specific examples

[0517] Initial state registration:

[0518] User: Organize a virtual store and take photos with their smartphone camera.

[0519] Device: Upload the captured images to the cloud server.

[0520] Server: Save the image and set it as a clean reference image.

[0521] Regular imaging and analysis:

[0522] Device: Take a photo of the virtual store every day at 10:00 AM and upload the new image to the cloud server.

[0523] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[0524] Example notification:

[0525] If the clutter level exceeds a threshold, the server generates a notification saying, "Your virtual store is messy. Let's tidy it up." and sends it to the user's device. Depending on the emotion recognized by the emotion engine, the notification text is changed to something like, "Why don't you take a short break and then tidy up?"

[0526] Examples of cleaning advice provided:

[0527] When a user uses the chat function on their device to ask, "How should I organize my store?", the server will provide advice such as, "It would be a good idea to gather all your important documents and put them in one place." If the emotion engine determines that the user is feeling stressed, it will recommend a message such as, "Take a relaxing break, then organize your store in an orderly manner."

[0528] Example prompt for a generative AI model:

[0529] "Based on an image, determine how messy a room is. Also, generate a notification message based on the user's current emotion."

[0530] As a result, this system efficiently organizes the virtual store and provides flexible support that can also respond to the user's emotions.

[0531] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0532] Step 1:

[0533] Registering the initial state

[0534] Input: The user organizes the virtual store and takes a photo of the store.

[0535] What happens: A user uses the camera on their smartphone or tablet to capture an ideal image of the store.

[0536] Output: The captured "initial image" is generated.

[0537] Data processing: The initial image is uploaded from the user's device to the cloud server and saved in a storage device.

[0538] Example of operation: A user presses the "initial state registration" button on a smartphone app, takes a picture, and sends it to the cloud server.

[0539] Step 2:

[0540] Periodic image acquisition

[0541] Input: A recurring schedule.

[0542] Specific operation: The scheduling function of the user device automatically starts the camera at the set time (e.g., 10:00 a.m. every day) and captures images of the store.

[0543] Output: The newly captured image is generated.

[0544] Data processing: The captured images are automatically uploaded to a cloud server.

[0545] Example of operation: Every day at 10:00 AM, the smartphone camera automatically starts up, takes pictures of the virtual store, and sends them to the cloud server.

[0546] Step 3:

[0547] Compare images and assess clutter levels

[0548] Input: Initial image, newly captured image.

[0549] How it works: The server uses image processing AI to compare the initial image with the newly captured image.

[0550] Output: A quantified result of clutter level.

[0551] Data calculation: Image processing AI analyzes object recognition, dirt detection, and changes in object placement to quantify the degree of clutter.

[0552] How it works: The server takes the uploaded image, runs an image processing algorithm to analyze the differences between each pixel, and generates a clutter score.

[0553] Step 4:

[0554] Generate and send cleaning notifications

[0555] Input: Quantified clutter level.

[0556] What happens: The server checks if clutter exceeds a set threshold.

[0557] Output: Notification message to let you know when it's time to clean.

[0558] Data calculation: If the clutter level exceeds a threshold, the server generates a notification message.

[0559] Example of operation: If the clutter level exceeds 60%, the cloud server generates a message saying, "The virtual store is messy. Please tidy it up," and sends it to the user's device.

[0560] Step 5:

[0561] Changing notification content with emotion engine

[0562] Input: User facial and voice data.

[0563] Specific operation: The emotion engine analyzes the user's emotions and adaptively changes the notification message.

[0564] Output: Customized notification message depending on the emotion.

[0565] Data calculation: The emotion engine analyzes the user's input data and determines and classifies emotions.

[0566] Example of how it works: If the user is determined to be tired, the message changes to "How about taking a break and then tidying up?"

[0567] Step 6:

[0568] Providing cleaning advice

[0569] Input: A user's request for cleaning advice.

[0570] Specific operation: Use the chat function from the user's device to send a question to the server.

[0571] Output: A good advice message on how to clean.

[0572] Data processing: The cloud server selects and provides the most appropriate cleaning advice from a pre-registered cleaning advice database.

[0573] Example of how it works: A user asks, "How can I organize my store efficiently?" and the server sends the advice, "It would be a good idea to collect all your important documents and keep them in one place."

[0574] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0575] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0576] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0577] [Second embodiment]

[0578] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0579] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0580] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0581] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0582] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0583] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0584] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0585] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0586] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0587] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0588] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0589] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0590] The present invention is a system that automatically determines the messiness of a room and prompts a user to clean at an appropriate time. Specific embodiments of the system will be described below.

[0591] System Configuration

[0592] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[0593] Registering the initial state of the room

[0594] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0595] Regularly taking photos of the room

[0596] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0597] Analyzing images and determining clutter levels

[0598] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0599] Sending cleaning notifications

[0600] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[0601] Providing cleaning advice

[0602] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[0603] Specific examples

[0604] Registering the initial state

[0605] User: Clean up the living room and take a photo with your smartphone camera.

[0606] Device: Uploads captured images to the cloud server.

[0607] Server: Save the image and set it as a clean reference image.

[0608] Regular imaging and analysis

[0609] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0610] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[0611] Notifications when clutter exceeds thresholds

[0612] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0613] Device: Show the user the message "The living room is messy. Time to clean!"

[0614] Providing cleaning advice

[0615] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[0616] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[0617] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[0618] The processing flow will be explained below.

[0619] Registering the initial state of the room

[0620] Step 1:

[0621] User: Clean up the living room and take a picture with the camera.

[0622] The user takes an image of the room using a smartphone or a camera on the device.

[0623] Step 2:

[0624] Device: Uploads captured images to the cloud server.

[0625] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[0626] Step 3:

[0627] Server: Saves the uploaded image and sets it as the reference image.

[0628] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[0629] Regularly photograph and analyze the room conditions

[0630] Step 4:

[0631] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[0632] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[0633] Step 5:

[0634] Device: Automatically take a photo of the living room at the set time.

[0635] The camera will automatically start up and capture the entire room.

[0636] Step 6:

[0637] Device: Uploads newly captured images to the cloud server.

[0638] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[0639] Step 7:

[0640] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[0641] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[0642] Step 8:

[0643] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[0644] A comparison algorithm quantifies the clutter level of a room.

[0645] Sending cleaning notifications

[0646] Step 9:

[0647] Server: Compare the calculated clutter level with a threshold.

[0648] Check whether the quantified clutter level exceeds a set threshold.

[0649] Step 10:

[0650] Server: Generates cleaning notification if threshold is exceeded.

[0651] A notification message is generated and the message content and recipient information are added.

[0652] Step 11:

[0653] Server: Sends notifications to the user device.

[0654] Use the push notification system to send notification messages to the user's device.

[0655] Step 12:

[0656] Terminal: Show cleaning notification to user.

[0657] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[0658] Providing cleaning advice

[0659] Step 13:

[0660] User: Uses the device's chat function to ask about cleaning tips.

[0661] The user opens the chat screen, types a question, and sends it to the server.

[0662] Step 14:

[0663] Server: Receives questions from users and searches for appropriate advice.

[0664] The server searches the database for relevant cleaning advice and prepares a response.

[0665] Step 15:

[0666] Server: Sends advice to users via chat function.

[0667] Appropriate advice content is sent to the terminal as a chat message.

[0668] Step 16:

[0669] User: Clean according to the advice provided.

[0670] The user performs cleaning based on the advice received.

[0671] Example 1

[0672] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0673] In modern life, efficiently cleaning a room amidst busy daily routines is a difficult task. In particular, accurately understanding the state of clutter in a room and cleaning at the optimal time can be a burden for many people. Conventional methods require users to check the state of the room themselves and determine the need for cleaning each time, which is a very time-consuming operation. Therefore, there is a need for a system that solves these problems and allows users to keep their rooms clean efficiently and comfortably.

[0674] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0675] In this invention, the server includes a means for quantifying the degree of messiness of a room, a means for using image processing AI to recognize objects, detect dust and dirt, and analyze changes in the placement of objects, a means for notifying the user of the need for cleaning through a notification function, and a means for providing advice in response to the user's questions through a chat function, thereby enabling the user to automatically grasp the degree of messiness of the room and clean at the appropriate time.

[0676] The "initial state of the room" indicates a state in which the room is neat and tidy, and is an image that serves as a reference for evaluating the degree of messiness in the future.

[0677] The "means for periodically photographing the entire room" is a function for automatically acquiring images of the room at set time intervals.

[0678] "Image of initial state" refers to an image of the room's initial state that is captured and stored on a cloud server.

[0679] "Quantifying the degree of clutter" is the process of analyzing the clutter level of a room and expressing it as a number according to certain standards.

[0680] "Notification when threshold is exceeded" means generating and sending a notification to the user informing them of the need to clean when the quantified clutter level exceeds a set reference value.

[0681] "Image processing AI" is an artificial intelligence technology that analyzes captured images to recognize objects, detect dirt, and determine changes in the placement of objects.

[0682] "Object recognition" is a technology that identifies objects in an image and determines their location and type.

[0683] "Dust and dirt detection" is a technology that identifies dust and dirt in an image and confirms their presence.

[0684] The "notification function" is a function that sends messages to notify the user of specific events or actions.

[0685] The "chat function" is a function that allows users to communicate with the server in real time via text messages.

[0686] "Providing cleaning advice" refers to suggesting optimal cleaning methods and ideas in response to the user's questions and requests.

[0687] The present invention is a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time, and specific embodiments of the system will be described below.

[0688] System Configuration

[0689] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[0690] Registering the initial state of the room

[0691] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0692] Regularly taking photos of the room

[0693] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0694] Analyzing images and determining clutter levels

[0695] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0696] Sending cleaning notifications

[0697] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[0698] Providing cleaning advice

[0699] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[0700] Specific examples

[0701] Registering the initial state

[0702] User: Clean up the living room and take a photo with your smartphone camera.

[0703] Device: Uploads captured images to the cloud server.

[0704] Server: Save the image and set it as a clean reference image.

[0705] Regular imaging and analysis

[0706] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0707] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[0708] Notifications when clutter exceeds thresholds

[0709] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0710] Device: Show the user the message "The living room is messy. Time to clean!"

[0711] Providing cleaning advice

[0712] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[0713] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[0714] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[0715] Prompt Sentence Examples

[0716] Below is an example of a prompt sentence to input into the image processing AI model.

[0717] Compare the initial image with the new room image and quantify the clutter level. Evaluate based on object recognition, dust / dirt detection, and changes in object placement, and generate a notification if the clutter level exceeds a threshold.

[0718] These embodiments allow the user to automatically understand the messiness of the room and clean it at the appropriate time.

[0719] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0720] Step 1:

[0721] Initial image registration

[0722] User: Prepare a clean room. Specifically, clean the room and make sure everything is neat and tidy. In this state, start the smartphone camera and take a picture of the entire room.

[0723] Input: Image of the initial room state

[0724] Terminal: Uploads images taken by the user to the cloud server.

[0725] Output: Images uploaded to the cloud server

[0726] Server: Receives the uploaded image and saves it in a storage device as an "initial image."

[0727] Output: Saved initial image

[0728] Step 2:

[0729] Regular imaging

[0730] Device: Automatically activate the camera at a specified time according to a pre-set schedule, for example, 10:00 AM every day.

[0731] Input: Specified schedule

[0732] Terminal: Take a photo of the entire room again to obtain a new image of the room.

[0733] Output: Newly acquired room image

[0734] Step 3:

[0735] Uploading an image

[0736] Device: Newly acquired images are automatically uploaded to the cloud server.

[0737] Input: A newly acquired image of a room.

[0738] Output: New image uploaded to the cloud server

[0739] Server: Receives new images sent from the device and temporarily stores them in a storage device.

[0740] Output: New image saved

[0741] Step 4:

[0742] Determining the level of clutter

[0743] Server: Starts the process of comparing the new image with the initial image.

[0744] Input: New image, initial image

[0745] Server: Uses image processing AI to recognize objects, detect dust and dirt, and analyze changes in object placement.

[0746] Data processing: object recognition, dust and dirt detection, analysis of changes in object placement

[0747] Server: Based on the analysis results, the degree of clutter is quantified and it is determined whether this number exceeds a preset threshold.

[0748] Data calculation: Quantifying clutter level and determining threshold

[0749] Output: Numerical value of clutter level and threshold exceedance judgment

[0750] Step 5:

[0751] Sending cleaning notifications

[0752] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0753] Input: Threshold exceedance judgment

[0754] Server: Sends notifications to the user's device.

[0755] Output: Cleaning timing notification

[0756] Device: Receives a notification from the server and displays the message "Your room is messy. Time to clean!" to the user.

[0757] Output: The message that is displayed to the user

[0758] Step 6:

[0759] Providing cleaning advice

[0760] User: If necessary, use the chat function on the device to type in a cleaning question. For example, "How can I efficiently clean the dust off the floor?"

[0761] Input: User question

[0762] Terminal: Sends the user's question to the cloud server.

[0763] Output: Questions sent to the cloud server

[0764] Server: Searches for pre-registered cleaning advice information based on the question.

[0765] Data processing: Search for advice information based on the question

[0766] Server: Creates optimal advice as a text message and sends it to the user's device.

[0767] Output: Advice in the form of a text message

[0768] Terminal: Receives advice from the server and displays it to the user. For example, it displays a message such as "It would be a good idea to use a microfiber cloth."

[0769] Output: An advisory message that is displayed to the user.

[0770] This is the specific process flow of this system. By using this system, users can automatically determine the messiness of their room and clean it at the appropriate time.

[0771] (Application example 1)

[0772] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0773] Regular cleaning is essential to maintaining cleanliness in physical stores. However, store staff often miss cleaning opportunities or are unfamiliar with the optimal cleaning methods. This can result in a deterioration of the store environment, lower customer satisfaction, and hygiene issues. Therefore, there is a need for a system that can automatically and efficiently manage cleaning in physical stores and prompt cleaning at the appropriate times.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0775] In this invention, the server includes a means for periodically taking images with a camera installed in a specific area of ​​the store and uploading them to a cloud server, a means for an image processing AI on the cloud server to evaluate the degree of clutter and generate a notification as needed and send it to a user terminal, a means for accepting cleaning advice requests from users, and a means for providing appropriate cleaning advice based on the accepted request. This makes it possible to constantly monitor the environment within the store and clean at the appropriate time.

[0776] The "initial state" is a reference image that shows the ideal clean state of a room or store.

[0777] A "storage device" is a device that stores data, and includes cloud servers and the like.

[0778] A "cloud server" is a server that stores and processes data via the Internet.

[0779] "Image processing AI" is an artificial intelligence technology that analyzes images and extracts information for specific purposes.

[0780] "Camera" means a device that captures images or videos, including those built into smartphones and dedicated fixed cameras.

[0781] "Clutter level" is a numerical indication of the cleanliness and tidiness of a room or store.

[0782] A "threshold" is a reference value set for determining a specific condition.

[0783] A "notification" is a message that notifies the user of specific information.

[0784] A "user terminal" is a device that is directly operated by a user, and includes smartphones and tablets.

[0785] An "advice request" is a request by a user for specific information or advice.

[0786] "Cleaning advice" means providing information on cleaning methods and cleaning tips.

[0787] A "specific area" is a specific area within the store that is to be cleaned.

[0788] This invention is a system for streamlining cleaning management in brick-and-mortar stores, and uses a user terminal, a cloud server, and a camera as key components. This system is designed to determine the clutter level of a room and prompt cleaning at the appropriate time. Specific embodiments of the system are described below.

[0789] First, store staff take a photo of a specific area of ​​the store after cleaning and upload the image to a cloud server as an "initial image." This image shows the ideal cleanliness of the store and serves as a standard for future comparisons.

[0790] The system is then scheduled to periodically capture images of specific areas of the store. For example, every day at 2:00 p.m., the camera automatically captures images of the area and uploads the newly captured images to a cloud server, where they are analyzed using image processing AI.

[0791] The cloud server compares the newly uploaded image with the initial image and quantifies it based on object recognition and changes in clutter level. If this quantification exceeds a pre-defined threshold, the cloud server generates a notification and sends a message to the user device, such as "A specific area is cluttered. Time to clean!"

[0792] The user device also has a chat function, allowing store staff to ask questions about cleaning methods. For example, by entering a prompt such as, "Please tell me how to improve the cleaning efficiency of the cash register counter," appropriate cleaning advice is provided from the cloud server's knowledge base. The server then sends specific advice to the device, such as, "It is effective to disinfect the cash register counter once a week."

[0793] Hardware and software used

[0794] Hardware:

[0795] Camera: A camera built into a smartphone or a fixed camera installed in a store.

[0796] User devices: smartphones, tablet devices.

[0797] software:

[0798] Cloud server: Stores and processes data.

[0799] Image processing AI: Analyzes images and quantifies the degree of clutter.

[0800] Notification system: Sends messages to user terminals.

[0801] Chatbot: Provides cleaning advice in response to user questions.

[0802] Specific examples

[0803] Store staff clean the cash register area every day at 8:00 a.m., then take a photo of the area with their smartphone and upload it to a cloud server.

[0804] The system takes an up-to-date photo of the checkout area every day at 2 p.m. and uploads it to a cloud server. Image processing AI evaluates the clutter level and sends notifications as necessary.

[0805] Staff will clean the area again based on the notification they receive.

[0806] Prompt Sentence Examples

[0807] "Please tell me how to improve the cleaning efficiency of the cash register counter."

[0808] "What is the best way to respond if the seating area is messy?"

[0809] This system allows physical stores to maintain a clean environment at all times, contributing to improved customer satisfaction and hygiene.

[0810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0811] Step 1:

[0812] A user cleans a specific area of ​​the store and takes a picture of that area with a camera. The input is an image taken by the camera, which is uploaded to a cloud server. The server stores the received image in a storage device as an "initial state image." This sets a reference for the initial state.

[0813] Step 2:

[0814] The device periodically takes photos of specific areas of the store according to a set schedule, and the new images are input and uploaded to a cloud server, which uses image processing AI to analyze the new images and generate data for object recognition and clutter assessment.

[0815] Step 3:

[0816] The cloud server uses image processing AI to compare the new image with the initial image. The input is the new image and the initial image, which are compared to quantify the degree of clutter. This process outputs the clutter level as a number.

[0817] Step 4:

[0818] The server determines whether the numerically-quantified clutter level exceeds a set threshold. The input is the numerically-quantified clutter level, which is evaluated. If the threshold is exceeded, the server generates a notification informing the user that it is time to clean and sends it to the user's device. This notification is then displayed on the user's device.

[0819] Step 5:

[0820] After receiving the notification, the user uses the chat function on the device to request cleaning advice from the server. The input is a question (prompt sentence) from the user. The server searches for appropriate advice from a pre-registered knowledge base and generates this information.

[0821] Step 6:

[0822] The server sends the generated cleaning advice to the user's device. The input is the knowledge base information in the server and the user's question, and the advice generated based on this is output. The advice is displayed on the user's device, and the physical store is cleaned efficiently based on this.

[0823] Following these steps will help keep your store environment clean, leading to more efficient cleaning and improved customer satisfaction.

[0824] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0825] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0826] System Configuration

[0827] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[0828] Registering the initial state of the room

[0829] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0830] Regularly taking photos of the room

[0831] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0832] Analyzing images and determining clutter levels

[0833] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0834] Emotion Engine Operation

[0835] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[0836] Sending cleaning notifications

[0837] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[0838] Providing cleaning advice

[0839] If a user has trouble with cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "How can I efficiently clean the dust off the floor?" the server might send advice such as "It would be good to use a microfiber cloth" to the device. The emotion engine can also provide specific advice for reducing stress by taking into account the user's stress level.

[0840] Specific examples

[0841] Registering the initial state

[0842] User: Clean up the living room and take a photo with your smartphone camera.

[0843] Device: Uploads captured images to the cloud server.

[0844] Server: Save the image and set it as a clean reference image.

[0845] Regular imaging and analysis

[0846] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0847] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[0848] Notifications when clutter exceeds thresholds

[0849] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[0850] Terminal: Shows the user a message saying "The living room is messy. Time to clean!" The message may change depending on the emotion recognized by the emotion engine.

[0851] Providing cleaning advice

[0852] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[0853] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it." The emotion engine provides specific advice for reducing stress according to the user's stress level.

[0854] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[0855] The processing flow will be explained below.

[0856] Registering the initial state of the room

[0857] Step 1:

[0858] User: Clean up the living room and take a picture with the camera.

[0859] The user takes an image of the room using a smartphone or a camera on the device.

[0860] Step 2:

[0861] Device: Uploads captured images to the cloud server.

[0862] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[0863] Step 3:

[0864] Server: Saves the uploaded image and sets it as the reference image.

[0865] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[0866] Regularly photograph and analyze the room conditions

[0867] Step 4:

[0868] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[0869] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[0870] Step 5:

[0871] Device: Automatically take a photo of the living room at the set time.

[0872] The camera will automatically start up and capture the entire room.

[0873] Step 6:

[0874] Device: Uploads newly captured images to the cloud server.

[0875] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[0876] Step 7:

[0877] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[0878] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[0879] Step 8:

[0880] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[0881] A comparison algorithm quantifies the clutter level of a room.

[0882] Sending cleaning notifications

[0883] Step 9:

[0884] Server: Compare the calculated clutter level with a threshold.

[0885] Check whether the quantified clutter level exceeds a set threshold.

[0886] Step 10:

[0887] Server: Generates cleaning notification if threshold is exceeded.

[0888] A notification message is generated and the message content and recipient information are added.

[0889] Step 11:

[0890] Server: Sends notifications to the user device.

[0891] Use the push notification system to send notification messages to the user's device.

[0892] Step 12:

[0893] Terminal: Show cleaning notification to user.

[0894] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[0895] Emotion Engine Operation

[0896] Step 13:

[0897] Terminal: Collects the user's facial expressions and voice.

[0898] The device's camera and microphone are used to capture the user's facial expressions and voice data.

[0899] Step 14:

[0900] Server: The emotion engine analyzes the acquired data and recognizes the user's emotion.

[0901] The emotion engine uses image and audio analysis to identify the user's emotions (e.g., stress, joy, fatigue).

[0902] Step 15:

[0903] Server: Adaptively change the content of cleaning notifications based on the user's emotions.

[0904] If the user is tired, an encouraging message such as "Let's clean up and refresh ourselves" is generated.

[0905] Providing cleaning advice

[0906] Step 16:

[0907] User: Uses the device's chat function to ask about cleaning tips.

[0908] The user opens the chat screen, types a question, and sends it to the server.

[0909] Step 17:

[0910] Server: Receives questions from users and searches for appropriate advice.

[0911] The server searches the database for relevant cleaning advice and prepares a response.

[0912] Step 18:

[0913] Server: Sends advice to users via chat function.

[0914] Appropriate advice is sent to the device as a chat message. The emotion engine provides specific advice for reducing stress according to the user's stress level.

[0915] Step 19:

[0916] User: Clean according to the advice provided.

[0917] The user performs cleaning based on the advice received.

[0918] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[0919] Example 2

[0920] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0921] In today's busy living environment, there is a need for a system that helps users clean their rooms at appropriate times. However, typical cleaning support systems send uniform notifications without considering the user's emotional state, which can be annoying for users. Furthermore, they lack the functionality to provide specific cleaning advice, and therefore do not provide sufficient support for users to effectively organize their rooms. To solve this problem, a system is needed that recognizes the user's emotions, adaptively changes cleaning notifications, and provides specific cleaning advice.

[0922] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0923] In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state using image processing means, means for performing object recognition and quantifying the degree of messiness of the room, means for generating a notification informing the user that it is time to clean when the quantified degree of messiness exceeds a threshold and sending the notification to the user terminal, means for recognizing the user's emotions, means for adaptively changing the content of the notification based on the user's emotions, means for receiving a request for cleaning advice from the user, and means for providing appropriate cleaning advice based on the received request. This allows the user to clean their room efficiently, receive cleaning notifications at appropriate times, and further allows for flexible responses according to the user's emotional state.

[0924] The "means for capturing an image of the initial state of the room" refers to a device or method for capturing an image of the room in its tidy state.

[0925] The "means for saving the captured image of the room in its initial state in a storage device" refers to a device or process for saving the captured image data of the initial state.

[0926] "Means for periodically photographing the entire room" refers to a device or method for automatically or manually photographing the entire room based on a pre-set schedule.

[0927] "Means for uploading newly captured images of the room to a storage device" refers to a device or process that transfers the latest captured image data to a cloud or other storage system.

[0928] "Means for comparing a newly captured image of the room with an image of the initial state using image processing means" refers to an algorithm or device for comparing an image of the initial state with the latest image and detecting differences.

[0929] "Means for performing object recognition and quantifying the degree of clutter in a room" refers to a process or device that uses image processing technology to analyze the presence and placement of objects, and then expresses the degree of clutter as a number based on the results.

[0930] "Means for generating and sending to a user device a notification informing the user that it is time to clean when the quantified clutter level exceeds a threshold" refers to a system or method for creating and sending to a user device a notification encouraging cleaning when the clutter level exceeds a pre-set threshold.

[0931] "Means for recognizing user emotions" refers to technology or devices that analyze a user's facial expressions, voice, text input, etc. to determine the user's emotional state.

[0932] "Means for adaptively changing notification content based on user emotion" refers to a system or process for changing the content or tone of a notification based on a recognized user emotion.

[0933] "Means for accepting cleaning advice requests from users" refers to an interface or method for accepting input from users asking questions or requesting advice about cleaning.

[0934] "Means for providing appropriate cleaning advice based on a received request" refers to a system or method for providing appropriate cleaning advice based on information and knowledge prepared in advance in response to a received user request.

[0935] MODE FOR CARRYING OUT THE INVENTION

[0936] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[0937] System Configuration

[0938] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[0939] Registering the initial state of the room

[0940] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[0941] Examples:

[0942] User: Tidy up the living room and take a photo with their smartphone camera.

[0943] Device: Upload the captured images to the cloud server.

[0944] Server: Save the image and set it as a clean reference image.

[0945] Regularly taking photos of the room

[0946] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[0947] Examples:

[0948] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[0949] Analyzing images and determining clutter levels

[0950] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. Software used here includes TensorFlow and OpenCV. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a preset threshold, the server generates a notification to the user's device informing them that it's time to clean.

[0951] Examples:

[0952] Server: Image processing AI compares the new image with the initial image and calculates the degree of clutter.

[0953] Server: When the clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[0954] Emotion Engine Operation

[0955] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[0956] Examples:

[0957] Server: Adaptively change the content of notifications for users based on emotions recognized by the emotion engine.

[0958] Sending cleaning notifications

[0959] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[0960] Examples:

[0961] Device: Displays received notifications to the user and plays notifications and alarms as needed.

[0962] Providing cleaning advice

[0963] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. The emotion engine also takes the user's stress level into consideration and provides specific advice for stress reduction.

[0964] Examples:

[0965] User: Uses the chat function on their device to ask the server for cleaning tips.

[0966] Server: Responds to the user's question with advice such as "It may be effective to remove the cushion cover before washing."

[0967] Prompt Sentence Examples

[0968] Example prompt for initial registration:

[0969] After you have tidied up your living room, take a picture with your smartphone camera and upload the image to the cloud server.

[0970] Cleaning notification prompt example:

[0971] Generate a cleaning notification message when the emotion engine recognizes that the user is tired. Example: Clean up and refresh yourself.

[0972] Example cleaning advice prompts:

[0973] If a user asks how to efficiently dust the floor, provide appropriate advice taking into account the results of the sentiment engine. For example: Using a microfiber cloth is a good idea.

[0974] In this way, the system allows users to clean their rooms efficiently, receive timely notifications, and respond flexibly to the user's emotions.

[0975] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0976] Step 1:

[0977] User: Take a picture of the initial state of the room with the camera.

[0978] Input: A clean image of a room.

[0979] Output: Pristine image data.

[0980] Specific operation: The user opens the camera app on their smartphone, clicks the "Register initial state" button, and takes a photo of the room.

[0981] Step 2:

[0982] Device: Upload the captured initial image to the cloud server.

[0983] Input: Pristine image data.

[0984] Output: Image data stored on a cloud server.

[0985] What it does: It establishes an internet connection to instantly upload images taken by the smartphone to a cloud server.

[0986] Step 3:

[0987] Server: Saves the received image data in its initial state in a storage device.

[0988] Input: Image data uploaded to the cloud server.

[0989] Output: Pristine image data stored in a database.

[0990] Specific operation: The cloud server receives the image data and stores it in a database.

[0991] Step 4:

[0992] Device: Automatically activate the camera at a set time (e.g., 10:00 AM every day) and capture images of the room.

[0993] Input: Scheduled timing.

[0994] Output: Image data of the new room.

[0995] Specific operation: The device will start the camera at 10:00 AM every day and automatically take pictures of the room.

[0996] Step 5:

[0997] On your device: Upload any new images you take to the cloud server.

[0998] Input: Image data of the new room.

[0999] Output: New image data stored on the cloud server.

[1000] Specific operation: Data transfer is performed to upload newly captured images to the cloud server.

[1001] Step 6:

[1002] Server: Compares the new image with the initial image using image processing techniques.

[1003] Input: New image data and initial image data.

[1004] Output: Image comparison results.

[1005] How it works: The cloud server uses image processing AI to compare the new image with the initial image using tools such as TensorFlow and OpenCV.

[1006] Step 7:

[1007] Server: Performs object recognition and quantifies the degree of clutter in a room.

[1008] Input: Image comparison results.

[1009] Output: Quantified clutter level.

[1010] What it does: It uses an object recognition algorithm to quantify the level of clutter based on the placement of objects, the presence of dust and dirt, etc.

[1011] Step 8:

[1012] Server: When the quantified clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[1013] Input: Quantified clutter level.

[1014] Output: Notification that it's time to clean.

[1015] Specific behavior: When the clutter level exceeds a threshold, the server creates a notification and sends it to the user's device.

[1016] Step 9:

[1017] Server: Recognizes user emotions.

[1018] Input: User facial, voice, and text input data.

[1019] Output: User emotion recognition results.

[1020] Specific operation: Using the emotion engine, facial expression recognition and voice analysis are performed to determine the user's emotions.

[1021] Step 10:

[1022] Server: Adaptively change notification content based on user emotions.

[1023] Input: User emotion recognition results.

[1024] Output: Notification content tailored based on sentiment.

[1025] Specific behavior: Depending on the recognized user emotion, for example, if it is determined that the user is "tired," a message such as "Let's clean up and refresh" is generated.

[1026] Step 11:

[1027] Terminal: Displays notifications sent from the cloud server to the user.

[1028] Input: Notification from the cloud server.

[1029] Output: A notification message that will be displayed on the user's terminal.

[1030] Specific behavior: The device displays the received notification to the user and plays a notification sound or alarm if necessary.

[1031] Step 12:

[1032] User: Uses the chat function to ask the server for cleaning tips.

[1033] Input: The question asked by the user.

[1034] Output: Query data to the server.

[1035] Specific behavior: The user opens the chat function on their device and types a question about cleaning.

[1036] Step 13:

[1037] Server: Provides appropriate cleaning advice based on the user's request.

[1038] Input: The question asked by the user.

[1039] Output: Cleaning advice message.

[1040] Specific operation: The server generates specific advice for the user's question based on pre-registered information and replies via the chat function.

[1041] (Application example 2)

[1042] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1043] In conventional virtual stores, managers had to manually check the level of clutter and tidiness within the store and periodically tidy up. However, this process required time and effort, and it was difficult for the manager to respond appropriately based on their emotions and the situation. Furthermore, in order to improve the user experience, a system was needed that could automatically determine the level of tidiness in the virtual space and perform efficient maintenance.

[1044] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state, means for quantifying the degree of clutter in the room, means for generating a notification informing the user that cleaning is necessary when the quantified degree of clutter exceeds a threshold and sending the notification to the user terminal, means for accepting a request for cleaning advice from a user, means for providing appropriate cleaning advice based on the accepted request, means for recognizing the user's emotions and changing the content of the notification based on the emotions, and means for analyzing the organization of structures in the virtual space. This enables the server to automatically determine the degree of clutter in a virtual store, efficiently issue maintenance notifications, and appropriately respond to the manager's emotions and circumstances.

[1045] The "initial state of the room" is an image of a tidy and tidy room taken when the user first registers with the system.

[1046] A "storage device" is a hardware component of a computer for persistently storing data.

[1047] The "means for taking regular photographs" refers to a function or device that automatically takes photographs of the room based on a set schedule.

[1048] A "newly captured image of the room" is an image that is periodically captured and shows the latest state of the room.

[1049] "Means for uploading images to a storage device" refers to a process or system that sends captured images to a storage device such as a cloud or server.

[1050] The "level of clutter in a room" is an index that quantifies the degree of clutter, dirt, etc., compared to the initial state of the room.

[1051] "Means of quantifying" refers to algorithms or programs that use image processing technology to express the tidiness of a room as a number.

[1052] A "user terminal" is a device such as a smartphone, tablet, or PC that a user uses to access the system.

[1053] "Means for accepting requests for cleaning advice" refers to a function or chat system that allows users to send questions about cleaning methods and tips.

[1054] The "means for providing appropriate cleaning advice" refers to algorithms and databases that provide appropriate cleaning methods in response to user questions.

[1055] "Means for recognizing emotions and changing notification content based on emotions" refers to a system that analyzes emotions from the user's facial expressions and voice and adjusts the content of notifications accordingly.

[1056] A "virtual space" is a virtual space reproduced on a computer, and is a digital environment in which users can interact.

[1057] The "means for analyzing the organization of structures" is a system that analyzes the arrangement of items and objects placed in a virtual space and evaluates their organization.

[1058] System configuration

[1059] The system of this invention is composed of a user terminal, a cloud server, a camera, and an emotion engine. The user terminal is a device such as a smartphone, tablet, or PC, and the camera may be built into these devices or an external camera may be used. The cloud server is equipped with an image processing AI and an emotion engine.

[1060] Registering the initial state

[1061] Users organize their virtual store and take a photo of it. This "initial image" represents the ideal state and is uploaded to a cloud server and stored in a storage device. This serves as a baseline for future comparison.

[1062] Regular status checks

[1063] The system is scheduled to periodically take photos of the entire virtual store. The user's device automatically starts the camera at the set time, acquires new images, and uploads them to the cloud server. This schedule can be freely adjusted to suit the user's convenience.

[1064] Analyzing images and determining clutter levels

[1065] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to let the user know it's time to clean.

[1066] Emotion Engine Operation

[1067] The emotion engine recognizes the user's emotions by analyzing their facial expressions, voice data, and input data. For example, if the user is tired, it generates a gentle message such as, "Your virtual store is messy. Why don't you take a break and then tidy it up?"

[1068] Providing notice and advice

[1069] The user's device receives notifications sent from the cloud server and displays the message to the user. If the user has a question about cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information. The emotion engine also works in conjunction with this to recommend advice tailored to the user's emotions and stress level.

[1070] Specific examples

[1071] Initial state registration:

[1072] User: Organize a virtual store and take photos with their smartphone camera.

[1073] Device: Upload the captured images to the cloud server.

[1074] Server: Save the image and set it as a clean reference image.

[1075] Regular imaging and analysis:

[1076] Device: Take a photo of the virtual store every day at 10:00 AM and upload the new image to the cloud server.

[1077] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1078] Example notification:

[1079] If the clutter level exceeds a threshold, the server generates a notification saying, "Your virtual store is messy. Let's tidy it up." and sends it to the user's device. Depending on the emotion recognized by the emotion engine, the notification text is changed to something like, "Why don't you take a short break and then tidy up?"

[1080] Examples of cleaning advice provided:

[1081] When a user uses the chat function on their device to ask, "How should I organize my store?", the server will provide advice such as, "It would be a good idea to gather all your important documents and put them in one place." If the emotion engine determines that the user is feeling stressed, it will recommend a message such as, "Take a relaxing break, then organize your store in an orderly manner."

[1082] Example prompt for a generative AI model:

[1083] "Based on an image, determine how messy a room is. Also, generate a notification message based on the user's current emotion."

[1084] As a result, this system efficiently organizes the virtual store and provides flexible support that can also respond to the user's emotions.

[1085] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1086] Step 1:

[1087] Registering the initial state

[1088] Input: The user organizes the virtual store and takes a photo of the store.

[1089] What happens: A user uses the camera on their smartphone or tablet to capture an ideal image of the store.

[1090] Output: The captured "initial image" is generated.

[1091] Data processing: The initial image is uploaded from the user's device to the cloud server and saved in a storage device.

[1092] Example of operation: A user presses the "initial state registration" button on a smartphone app, takes a picture, and sends it to the cloud server.

[1093] Step 2:

[1094] Periodic image acquisition

[1095] Input: A recurring schedule.

[1096] Specific operation: The scheduling function of the user device automatically starts the camera at the set time (e.g., 10:00 a.m. every day) and captures images of the store.

[1097] Output: The newly captured image is generated.

[1098] Data processing: The captured images are automatically uploaded to a cloud server.

[1099] Example of operation: Every day at 10:00 AM, the smartphone camera automatically starts up, takes pictures of the virtual store, and sends them to the cloud server.

[1100] Step 3:

[1101] Compare images and assess clutter levels

[1102] Input: Initial image, newly captured image.

[1103] How it works: The server uses image processing AI to compare the initial image with the newly captured image.

[1104] Output: A quantified result of clutter level.

[1105] Data calculation: Image processing AI analyzes object recognition, dirt detection, and changes in object placement to quantify the degree of clutter.

[1106] How it works: The server takes the uploaded image, runs an image processing algorithm to analyze the differences between each pixel, and generates a clutter score.

[1107] Step 4:

[1108] Generate and send cleaning notifications

[1109] Input: Quantified clutter level.

[1110] What happens: The server checks if clutter exceeds a set threshold.

[1111] Output: Notification message to let you know when it's time to clean.

[1112] Data calculation: If the clutter level exceeds a threshold, the server generates a notification message.

[1113] Example of operation: If the clutter level exceeds 60%, the cloud server generates a message saying, "The virtual store is messy. Please tidy it up," and sends it to the user's device.

[1114] Step 5:

[1115] Changing notification content with emotion engine

[1116] Input: User facial and voice data.

[1117] Specific operation: The emotion engine analyzes the user's emotions and adaptively changes the notification message.

[1118] Output: Customized notification message depending on the emotion.

[1119] Data calculation: The emotion engine analyzes the user's input data and determines and classifies emotions.

[1120] Example of how it works: If the user is determined to be tired, the message changes to "How about taking a break and then tidying up?"

[1121] Step 6:

[1122] Providing cleaning advice

[1123] Input: A user's request for cleaning advice.

[1124] Specific operation: Use the chat function from the user's device to send a question to the server.

[1125] Output: A good advice message on how to clean.

[1126] Data processing: The cloud server selects and provides the most appropriate cleaning advice from a pre-registered cleaning advice database.

[1127] Example of how it works: A user asks, "How can I organize my store efficiently?" and the server sends the advice, "It would be a good idea to collect all your important documents and keep them in one place."

[1128] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1129] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1130] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1131] [Third embodiment]

[1132] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1133] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1134] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1135] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1136] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1137] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1138] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1139] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1140] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1141] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1142] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1143] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1144] The present invention is a system that automatically determines the messiness of a room and prompts a user to clean at an appropriate time. Specific embodiments of the system will be described below.

[1145] System Configuration

[1146] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[1147] Registering the initial state of the room

[1148] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[1149] Regularly taking photos of the room

[1150] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[1151] Analyzing images and determining clutter levels

[1152] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[1153] Sending cleaning notifications

[1154] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[1155] Providing cleaning advice

[1156] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[1157] Specific examples

[1158] Registering the initial state

[1159] User: Clean up the living room and take a photo with your smartphone camera.

[1160] Device: Uploads captured images to the cloud server.

[1161] Server: Save the image and set it as a clean reference image.

[1162] Regular imaging and analysis

[1163] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[1164] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1165] Notifications when clutter exceeds thresholds

[1166] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1167] Device: Show the user the message "The living room is messy. Time to clean!"

[1168] Providing cleaning advice

[1169] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[1170] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[1171] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[1172] The processing flow will be explained below.

[1173] Registering the initial state of the room

[1174] Step 1:

[1175] User: Clean up the living room and take a picture with the camera.

[1176] The user takes an image of the room using a smartphone or a camera on the device.

[1177] Step 2:

[1178] Device: Uploads captured images to the cloud server.

[1179] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[1180] Step 3:

[1181] Server: Saves the uploaded image and sets it as the reference image.

[1182] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[1183] Regularly photograph and analyze the room conditions

[1184] Step 4:

[1185] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[1186] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[1187] Step 5:

[1188] Device: Automatically take a photo of the living room at the set time.

[1189] The camera will automatically start up and capture the entire room.

[1190] Step 6:

[1191] Device: Uploads newly captured images to the cloud server.

[1192] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[1193] Step 7:

[1194] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[1195] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[1196] Step 8:

[1197] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[1198] A comparison algorithm quantifies the clutter level of a room.

[1199] Sending cleaning notifications

[1200] Step 9:

[1201] Server: Compare the calculated clutter level with a threshold.

[1202] Check whether the quantified clutter level exceeds a set threshold.

[1203] Step 10:

[1204] Server: Generates cleaning notification if threshold is exceeded.

[1205] A notification message is generated and the message content and recipient information are added.

[1206] Step 11:

[1207] Server: Sends notifications to the user device.

[1208] Use the push notification system to send notification messages to the user's device.

[1209] Step 12:

[1210] Terminal: Show cleaning notification to user.

[1211] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[1212] Providing cleaning advice

[1213] Step 13:

[1214] User: Uses the device's chat function to ask about cleaning tips.

[1215] The user opens the chat screen, types a question, and sends it to the server.

[1216] Step 14:

[1217] Server: Receives questions from users and searches for appropriate advice.

[1218] The server searches the database for relevant cleaning advice and prepares a response.

[1219] Step 15:

[1220] Server: Sends advice to users via chat function.

[1221] Appropriate advice content is sent to the terminal as a chat message.

[1222] Step 16:

[1223] User: Clean according to the advice provided.

[1224] The user performs cleaning based on the advice received.

[1225] Example 1

[1226] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1227] In modern life, efficiently cleaning a room amidst busy daily routines is a difficult task. In particular, accurately understanding the state of clutter in a room and cleaning at the optimal time can be a burden for many people. Conventional methods require users to check the state of the room themselves and determine the need for cleaning each time, which is a very time-consuming operation. Therefore, there is a need for a system that solves these problems and allows users to keep their rooms clean efficiently and comfortably.

[1228] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1229] In this invention, the server includes a means for quantifying the degree of messiness of a room, a means for using image processing AI to recognize objects, detect dust and dirt, and analyze changes in the placement of objects, a means for notifying the user of the need for cleaning through a notification function, and a means for providing advice in response to the user's questions through a chat function, thereby enabling the user to automatically grasp the degree of messiness of the room and clean at the appropriate time.

[1230] The "initial state of the room" indicates a state in which the room is neat and tidy, and is an image that serves as a reference for evaluating the degree of messiness in the future.

[1231] The "means for periodically photographing the entire room" is a function for automatically acquiring images of the room at set time intervals.

[1232] "Image of initial state" refers to an image of the room's initial state that is captured and stored on a cloud server.

[1233] "Quantifying the degree of clutter" is the process of analyzing the clutter level of a room and expressing it as a number according to certain standards.

[1234] "Notification when threshold is exceeded" means generating and sending a notification to the user informing them of the need to clean when the quantified clutter level exceeds a set reference value.

[1235] "Image processing AI" is an artificial intelligence technology that analyzes captured images to recognize objects, detect dirt, and determine changes in the placement of objects.

[1236] "Object recognition" is a technology that identifies objects in an image and determines their location and type.

[1237] "Dust and dirt detection" is a technology that identifies dust and dirt in an image and confirms their presence.

[1238] The "notification function" is a function that sends messages to notify the user of specific events or actions.

[1239] The "chat function" is a function that allows users to communicate with the server in real time via text messages.

[1240] "Providing cleaning advice" refers to suggesting optimal cleaning methods and ideas in response to the user's questions and requests.

[1241] The present invention is a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time, and specific embodiments of the system will be described below.

[1242] System Configuration

[1243] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[1244] Registering the initial state of the room

[1245] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[1246] Regularly taking photos of the room

[1247] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[1248] Analyzing images and determining clutter levels

[1249] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[1250] Sending cleaning notifications

[1251] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[1252] Providing cleaning advice

[1253] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[1254] Specific examples

[1255] Registering the initial state

[1256] User: Clean up the living room and take a photo with your smartphone camera.

[1257] Device: Uploads captured images to the cloud server.

[1258] Server: Save the image and set it as a clean reference image.

[1259] Regular imaging and analysis

[1260] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[1261] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1262] Notifications when clutter exceeds thresholds

[1263] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1264] Device: Show the user the message "The living room is messy. Time to clean!"

[1265] Providing cleaning advice

[1266] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[1267] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[1268] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[1269] Prompt Sentence Examples

[1270] Below is an example of a prompt sentence to input into the image processing AI model.

[1271] Compare the initial image with the new room image and quantify the clutter level. Evaluate based on object recognition, dust / dirt detection, and changes in object placement, and generate a notification if the clutter level exceeds a threshold.

[1272] These embodiments allow the user to automatically understand the messiness of the room and clean it at the appropriate time.

[1273] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1274] Step 1:

[1275] Initial image registration

[1276] User: Prepare a clean room. Specifically, clean the room and make sure everything is neat and tidy. In this state, start the smartphone camera and take a picture of the entire room.

[1277] Input: Image of the initial room state

[1278] Terminal: Uploads images taken by the user to the cloud server.

[1279] Output: Images uploaded to the cloud server

[1280] Server: Receives the uploaded image and saves it in a storage device as an "initial image."

[1281] Output: Saved initial image

[1282] Step 2:

[1283] Regular imaging

[1284] Device: Automatically activate the camera at a specified time according to a pre-set schedule, for example, 10:00 AM every day.

[1285] Input: Specified schedule

[1286] Terminal: Take a photo of the entire room again to obtain a new image of the room.

[1287] Output: Newly acquired room image

[1288] Step 3:

[1289] Uploading an image

[1290] Device: Newly acquired images are automatically uploaded to the cloud server.

[1291] Input: A newly acquired image of a room.

[1292] Output: New image uploaded to the cloud server

[1293] Server: Receives new images sent from the device and temporarily stores them in a storage device.

[1294] Output: New image saved

[1295] Step 4:

[1296] Determining the level of clutter

[1297] Server: Starts the process of comparing the new image with the initial image.

[1298] Input: New image, initial image

[1299] Server: Uses image processing AI to recognize objects, detect dust and dirt, and analyze changes in object placement.

[1300] Data processing: object recognition, dust and dirt detection, analysis of changes in object placement

[1301] Server: Based on the analysis results, the degree of clutter is quantified and it is determined whether this number exceeds a preset threshold.

[1302] Data calculation: Quantifying clutter level and determining threshold

[1303] Output: Numerical value of clutter level and threshold exceedance judgment

[1304] Step 5:

[1305] Sending cleaning notifications

[1306] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1307] Input: Threshold exceedance judgment

[1308] Server: Sends notifications to the user's device.

[1309] Output: Cleaning timing notification

[1310] Device: Receives a notification from the server and displays the message "Your room is messy. Time to clean!" to the user.

[1311] Output: The message that is displayed to the user

[1312] Step 6:

[1313] Providing cleaning advice

[1314] User: If necessary, use the chat function on the device to type in a cleaning question. For example, "How can I efficiently clean the dust off the floor?"

[1315] Input: User question

[1316] Terminal: Sends the user's question to the cloud server.

[1317] Output: Questions sent to the cloud server

[1318] Server: Searches for pre-registered cleaning advice information based on the question.

[1319] Data processing: Search for advice information based on the question

[1320] Server: Creates optimal advice as a text message and sends it to the user's device.

[1321] Output: Advice in the form of a text message

[1322] Terminal: Receives advice from the server and displays it to the user. For example, it displays a message such as "It would be a good idea to use a microfiber cloth."

[1323] Output: An advisory message that is displayed to the user.

[1324] This is the specific process flow of this system. By using this system, users can automatically determine the messiness of their room and clean it at the appropriate time.

[1325] (Application example 1)

[1326] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1327] Regular cleaning is essential to maintaining cleanliness in physical stores. However, store staff often miss cleaning opportunities or are unfamiliar with the optimal cleaning methods. This can result in a deterioration of the store environment, lower customer satisfaction, and hygiene issues. Therefore, there is a need for a system that can automatically and efficiently manage cleaning in physical stores and prompt cleaning at the appropriate times.

[1328] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1329] In this invention, the server includes a means for periodically taking images with a camera installed in a specific area of ​​the store and uploading them to a cloud server, a means for an image processing AI on the cloud server to evaluate the degree of clutter and generate a notification as needed and send it to a user terminal, a means for accepting cleaning advice requests from users, and a means for providing appropriate cleaning advice based on the accepted request. This makes it possible to constantly monitor the environment within the store and clean at the appropriate time.

[1330] The "initial state" is a reference image that shows the ideal clean state of a room or store.

[1331] A "storage device" is a device that stores data, and includes cloud servers and the like.

[1332] A "cloud server" is a server that stores and processes data via the Internet.

[1333] "Image processing AI" is an artificial intelligence technology that analyzes images and extracts information for specific purposes.

[1334] "Camera" means a device that captures images or videos, including those built into smartphones and dedicated fixed cameras.

[1335] "Clutter level" is a numerical indication of the cleanliness and tidiness of a room or store.

[1336] A "threshold" is a reference value set for determining a specific condition.

[1337] A "notification" is a message that notifies the user of specific information.

[1338] A "user terminal" is a device that is directly operated by a user, and includes smartphones and tablets.

[1339] An "advice request" is a request by a user for specific information or advice.

[1340] "Cleaning advice" means providing information on cleaning methods and cleaning tips.

[1341] A "specific area" is a specific area within the store that is to be cleaned.

[1342] This invention is a system for streamlining cleaning management in brick-and-mortar stores, and uses a user terminal, a cloud server, and a camera as key components. This system is designed to determine the clutter level of a room and prompt cleaning at the appropriate time. Specific embodiments of the system are described below.

[1343] First, store staff take a photo of a specific area of ​​the store after cleaning and upload the image to a cloud server as an "initial image." This image shows the ideal cleanliness of the store and serves as a standard for future comparisons.

[1344] The system is then scheduled to periodically capture images of specific areas of the store. For example, every day at 2:00 p.m., the camera automatically captures images of the area and uploads the newly captured images to a cloud server, where they are analyzed using image processing AI.

[1345] The cloud server compares the newly uploaded image with the initial image and quantifies it based on object recognition and changes in clutter level. If this quantification exceeds a pre-defined threshold, the cloud server generates a notification and sends a message to the user device, such as "A specific area is cluttered. Time to clean!"

[1346] The user device also has a chat function, allowing store staff to ask questions about cleaning methods. For example, by entering a prompt such as, "Please tell me how to improve the cleaning efficiency of the cash register counter," appropriate cleaning advice is provided from the cloud server's knowledge base. The server then sends specific advice to the device, such as, "It is effective to disinfect the cash register counter once a week."

[1347] Hardware and software used

[1348] Hardware:

[1349] Camera: A camera built into a smartphone or a fixed camera installed in a store.

[1350] User devices: smartphones, tablet devices.

[1351] software:

[1352] Cloud server: Stores and processes data.

[1353] Image processing AI: Analyzes images and quantifies the degree of clutter.

[1354] Notification system: Sends messages to user terminals.

[1355] Chatbot: Provides cleaning advice in response to user questions.

[1356] Specific examples

[1357] Store staff clean the cash register area every day at 8:00 a.m., then take a photo of the area with their smartphone and upload it to a cloud server.

[1358] The system takes an up-to-date photo of the checkout area every day at 2 p.m. and uploads it to a cloud server. Image processing AI evaluates the clutter level and sends notifications as necessary.

[1359] Staff will clean the area again based on the notification they receive.

[1360] Prompt Sentence Examples

[1361] "Please tell me how to improve the cleaning efficiency of the cash register counter."

[1362] "What is the best way to respond if the seating area is messy?"

[1363] This system allows physical stores to maintain a clean environment at all times, contributing to improved customer satisfaction and hygiene.

[1364] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1365] Step 1:

[1366] A user cleans a specific area of ​​the store and takes a picture of that area with a camera. The input is an image taken by the camera, which is uploaded to a cloud server. The server stores the received image in a storage device as an "initial state image." This sets a reference for the initial state.

[1367] Step 2:

[1368] The device periodically takes photos of specific areas of the store according to a set schedule, and the new images are input and uploaded to a cloud server, which uses image processing AI to analyze the new images and generate data for object recognition and clutter assessment.

[1369] Step 3:

[1370] The cloud server uses image processing AI to compare the new image with the initial image. The input is the new image and the initial image, which are compared to quantify the degree of clutter. This process outputs the clutter level as a number.

[1371] Step 4:

[1372] The server determines whether the numerically-quantified clutter level exceeds a set threshold. The input is the numerically-quantified clutter level, which is evaluated. If the threshold is exceeded, the server generates a notification informing the user that it is time to clean and sends it to the user's device. This notification is then displayed on the user's device.

[1373] Step 5:

[1374] After receiving the notification, the user uses the chat function on the device to request cleaning advice from the server. The input is a question (prompt sentence) from the user. The server searches for appropriate advice from a pre-registered knowledge base and generates this information.

[1375] Step 6:

[1376] The server sends the generated cleaning advice to the user's device. The input is the knowledge base information in the server and the user's question, and the advice generated based on this is output. The advice is displayed on the user's device, and the physical store is cleaned efficiently based on this.

[1377] Following these steps will help keep your store environment clean, leading to more efficient cleaning and improved customer satisfaction.

[1378] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1379] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[1380] System Configuration

[1381] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[1382] Registering the initial state of the room

[1383] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[1384] Regularly taking photos of the room

[1385] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[1386] Analyzing images and determining clutter levels

[1387] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[1388] Emotion Engine Operation

[1389] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[1390] Sending cleaning notifications

[1391] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[1392] Providing cleaning advice

[1393] If a user has trouble with cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "How can I efficiently clean the dust off the floor?" the server might send advice such as "It would be good to use a microfiber cloth" to the device. The emotion engine can also provide specific advice for reducing stress by taking into account the user's stress level.

[1394] Specific examples

[1395] Registering the initial state

[1396] User: Clean up the living room and take a photo with your smartphone camera.

[1397] Device: Uploads captured images to the cloud server.

[1398] Server: Save the image and set it as a clean reference image.

[1399] Regular imaging and analysis

[1400] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[1401] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1402] Notifications when clutter exceeds thresholds

[1403] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1404] Terminal: Shows the user a message saying "The living room is messy. Time to clean!" The message may change depending on the emotion recognized by the emotion engine.

[1405] Providing cleaning advice

[1406] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[1407] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it." The emotion engine provides specific advice for reducing stress according to the user's stress level.

[1408] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[1409] The processing flow will be explained below.

[1410] Registering the initial state of the room

[1411] Step 1:

[1412] User: Clean up the living room and take a picture with the camera.

[1413] The user takes an image of the room using a smartphone or a camera on the device.

[1414] Step 2:

[1415] Device: Uploads captured images to the cloud server.

[1416] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[1417] Step 3:

[1418] Server: Saves the uploaded image and sets it as the reference image.

[1419] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[1420] Regularly photograph and analyze the room conditions

[1421] Step 4:

[1422] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[1423] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[1424] Step 5:

[1425] Device: Automatically take a photo of the living room at the set time.

[1426] The camera will automatically start up and capture the entire room.

[1427] Step 6:

[1428] Device: Uploads newly captured images to the cloud server.

[1429] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[1430] Step 7:

[1431] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[1432] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[1433] Step 8:

[1434] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[1435] A comparison algorithm quantifies the clutter level of a room.

[1436] Sending cleaning notifications

[1437] Step 9:

[1438] Server: Compare the calculated clutter level with a threshold.

[1439] Check whether the quantified clutter level exceeds a set threshold.

[1440] Step 10:

[1441] Server: Generates cleaning notification if threshold is exceeded.

[1442] A notification message is generated and the message content and recipient information are added.

[1443] Step 11:

[1444] Server: Sends notifications to the user device.

[1445] Use the push notification system to send notification messages to the user's device.

[1446] Step 12:

[1447] Terminal: Show cleaning notification to user.

[1448] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[1449] Emotion Engine Operation

[1450] Step 13:

[1451] Terminal: Collects the user's facial expressions and voice.

[1452] The device's camera and microphone are used to capture the user's facial expressions and voice data.

[1453] Step 14:

[1454] Server: The emotion engine analyzes the acquired data and recognizes the user's emotion.

[1455] The emotion engine uses image and audio analysis to identify the user's emotions (e.g., stress, joy, fatigue).

[1456] Step 15:

[1457] Server: Adaptively change the content of cleaning notifications based on the user's emotions.

[1458] If the user is tired, an encouraging message such as "Let's clean up and refresh ourselves" is generated.

[1459] Providing cleaning advice

[1460] Step 16:

[1461] User: Uses the device's chat function to ask about cleaning tips.

[1462] The user opens the chat screen, types a question, and sends it to the server.

[1463] Step 17:

[1464] Server: Receives questions from users and searches for appropriate advice.

[1465] The server searches the database for relevant cleaning advice and prepares a response.

[1466] Step 18:

[1467] Server: Sends advice to users via chat function.

[1468] Appropriate advice is sent to the device as a chat message. The emotion engine provides specific advice for reducing stress according to the user's stress level.

[1469] Step 19:

[1470] User: Clean according to the advice provided.

[1471] The user performs cleaning based on the advice received.

[1472] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[1473] Example 2

[1474] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1475] In today's busy living environment, there is a need for a system that helps users clean their rooms at appropriate times. However, typical cleaning support systems send uniform notifications without considering the user's emotional state, which can be annoying for users. Furthermore, they lack the functionality to provide specific cleaning advice, and therefore do not provide sufficient support for users to effectively organize their rooms. To solve this problem, a system is needed that recognizes the user's emotions, adaptively changes cleaning notifications, and provides specific cleaning advice.

[1476] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1477] In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state using image processing means, means for performing object recognition and quantifying the degree of messiness of the room, means for generating a notification informing the user that it is time to clean when the quantified degree of messiness exceeds a threshold and sending the notification to the user terminal, means for recognizing the user's emotions, means for adaptively changing the content of the notification based on the user's emotions, means for receiving a request for cleaning advice from the user, and means for providing appropriate cleaning advice based on the received request. This allows the user to clean their room efficiently, receive cleaning notifications at appropriate times, and further allows for flexible responses according to the user's emotional state.

[1478] The "means for capturing an image of the initial state of the room" refers to a device or method for capturing an image of the room in its tidy state.

[1479] The "means for saving the captured image of the room in its initial state in a storage device" refers to a device or process for saving the captured image data of the initial state.

[1480] "Means for periodically photographing the entire room" refers to a device or method for automatically or manually photographing the entire room based on a pre-set schedule.

[1481] "Means for uploading newly captured images of the room to a storage device" refers to a device or process that transfers the latest captured image data to a cloud or other storage system.

[1482] "Means for comparing a newly captured image of the room with an image of the initial state using image processing means" refers to an algorithm or device for comparing an image of the initial state with the latest image and detecting differences.

[1483] "Means for performing object recognition and quantifying the degree of clutter in a room" refers to a process or device that uses image processing technology to analyze the presence and placement of objects, and then expresses the degree of clutter as a number based on the results.

[1484] "Means for generating and sending to a user device a notification informing the user that it is time to clean when the quantified clutter level exceeds a threshold" refers to a system or method for creating and sending to a user device a notification encouraging cleaning when the clutter level exceeds a pre-set threshold.

[1485] "Means for recognizing user emotions" refers to technology or devices that analyze a user's facial expressions, voice, text input, etc. to determine the user's emotional state.

[1486] "Means for adaptively changing notification content based on user emotion" refers to a system or process for changing the content or tone of a notification based on a recognized user emotion.

[1487] "Means for accepting cleaning advice requests from users" refers to an interface or method for accepting input from users asking questions or requesting advice about cleaning.

[1488] "Means for providing appropriate cleaning advice based on a received request" refers to a system or method for providing appropriate cleaning advice based on information and knowledge prepared in advance in response to a received user request.

[1489] MODE FOR CARRYING OUT THE INVENTION

[1490] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[1491] System Configuration

[1492] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[1493] Registering the initial state of the room

[1494] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[1495] Examples:

[1496] User: Tidy up the living room and take a photo with their smartphone camera.

[1497] Device: Upload the captured images to the cloud server.

[1498] Server: Save the image and set it as a clean reference image.

[1499] Regularly taking photos of the room

[1500] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[1501] Examples:

[1502] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[1503] Analyzing images and determining clutter levels

[1504] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. Software used here includes TensorFlow and OpenCV. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a preset threshold, the server generates a notification to the user's device informing them that it's time to clean.

[1505] Examples:

[1506] Server: Image processing AI compares the new image with the initial image and calculates the degree of clutter.

[1507] Server: When the clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[1508] Emotion Engine Operation

[1509] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[1510] Examples:

[1511] Server: Adaptively change the content of notifications for users based on emotions recognized by the emotion engine.

[1512] Sending cleaning notifications

[1513] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[1514] Examples:

[1515] Device: Displays received notifications to the user and plays notifications and alarms as needed.

[1516] Providing cleaning advice

[1517] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. The emotion engine also takes the user's stress level into consideration and provides specific advice for stress reduction.

[1518] Examples:

[1519] User: Uses the chat function on their device to ask the server for cleaning tips.

[1520] Server: Responds to the user's question with advice such as "It may be effective to remove the cushion cover before washing."

[1521] Prompt Sentence Examples

[1522] Example prompt for initial registration:

[1523] After you have tidied up your living room, take a picture with your smartphone camera and upload the image to the cloud server.

[1524] Cleaning notification prompt example:

[1525] Generate a cleaning notification message when the emotion engine recognizes that the user is tired. Example: Clean up and refresh yourself.

[1526] Example cleaning advice prompts:

[1527] If a user asks how to efficiently dust the floor, provide appropriate advice taking into account the results of the sentiment engine. For example: Using a microfiber cloth is a good idea.

[1528] In this way, the system allows users to clean their rooms efficiently, receive timely notifications, and respond flexibly to the user's emotions.

[1529] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1530] Step 1:

[1531] User: Take a picture of the initial state of the room with the camera.

[1532] Input: A clean image of a room.

[1533] Output: Pristine image data.

[1534] Specific operation: The user opens the camera app on their smartphone, clicks the "Register initial state" button, and takes a photo of the room.

[1535] Step 2:

[1536] Device: Upload the captured initial image to the cloud server.

[1537] Input: Pristine image data.

[1538] Output: Image data stored on a cloud server.

[1539] What it does: It establishes an internet connection to instantly upload images taken by the smartphone to a cloud server.

[1540] Step 3:

[1541] Server: Saves the received image data in its initial state in a storage device.

[1542] Input: Image data uploaded to the cloud server.

[1543] Output: Pristine image data stored in a database.

[1544] Specific operation: The cloud server receives the image data and stores it in a database.

[1545] Step 4:

[1546] Device: Automatically activate the camera at a set time (e.g., 10:00 AM every day) and capture images of the room.

[1547] Input: Scheduled timing.

[1548] Output: Image data of the new room.

[1549] Specific operation: The device will start the camera at 10:00 AM every day and automatically take pictures of the room.

[1550] Step 5:

[1551] On your device: Upload any new images you take to the cloud server.

[1552] Input: Image data of the new room.

[1553] Output: New image data stored on the cloud server.

[1554] Specific operation: Data transfer is performed to upload newly captured images to the cloud server.

[1555] Step 6:

[1556] Server: Compares the new image with the initial image using image processing techniques.

[1557] Input: New image data and initial image data.

[1558] Output: Image comparison results.

[1559] How it works: The cloud server uses image processing AI to compare the new image with the initial image using tools such as TensorFlow and OpenCV.

[1560] Step 7:

[1561] Server: Performs object recognition and quantifies the degree of clutter in a room.

[1562] Input: Image comparison results.

[1563] Output: Quantified clutter level.

[1564] What it does: It uses an object recognition algorithm to quantify the level of clutter based on the placement of objects, the presence of dust and dirt, etc.

[1565] Step 8:

[1566] Server: When the quantified clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[1567] Input: Quantified clutter level.

[1568] Output: Notification that it's time to clean.

[1569] Specific behavior: When the clutter level exceeds a threshold, the server creates a notification and sends it to the user's device.

[1570] Step 9:

[1571] Server: Recognizes user emotions.

[1572] Input: User facial, voice, and text input data.

[1573] Output: User emotion recognition results.

[1574] Specific operation: Using the emotion engine, facial expression recognition and voice analysis are performed to determine the user's emotions.

[1575] Step 10:

[1576] Server: Adaptively change notification content based on user emotions.

[1577] Input: User emotion recognition results.

[1578] Output: Notification content tailored based on sentiment.

[1579] Specific behavior: Depending on the recognized user emotion, for example, if it is determined that the user is "tired," a message such as "Let's clean up and refresh" is generated.

[1580] Step 11:

[1581] Terminal: Displays notifications sent from the cloud server to the user.

[1582] Input: Notification from the cloud server.

[1583] Output: A notification message that will be displayed on the user's terminal.

[1584] Specific behavior: The device displays the received notification to the user and plays a notification sound or alarm if necessary.

[1585] Step 12:

[1586] User: Uses the chat function to ask the server for cleaning tips.

[1587] Input: The question asked by the user.

[1588] Output: Query data to the server.

[1589] Specific behavior: The user opens the chat function on their device and types a question about cleaning.

[1590] Step 13:

[1591] Server: Provides appropriate cleaning advice based on the user's request.

[1592] Input: The question asked by the user.

[1593] Output: Cleaning advice message.

[1594] Specific operation: The server generates specific advice for the user's question based on pre-registered information and replies via the chat function.

[1595] (Application example 2)

[1596] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1597] In conventional virtual stores, managers had to manually check the level of clutter and tidiness within the store and periodically tidy up. However, this process required time and effort, and it was difficult for the manager to respond appropriately based on their emotions and the situation. Furthermore, in order to improve the user experience, a system was needed that could automatically determine the level of tidiness in the virtual space and perform efficient maintenance.

[1598] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state, means for quantifying the degree of clutter in the room, means for generating a notification informing the user that cleaning is necessary when the quantified degree of clutter exceeds a threshold and sending the notification to the user terminal, means for accepting a request for cleaning advice from a user, means for providing appropriate cleaning advice based on the accepted request, means for recognizing the user's emotions and changing the content of the notification based on the emotions, and means for analyzing the organization of structures in the virtual space. This enables the server to automatically determine the degree of clutter in a virtual store, efficiently issue maintenance notifications, and appropriately respond to the manager's emotions and circumstances.

[1599] The "initial state of the room" is an image of a tidy and tidy room taken when the user first registers with the system.

[1600] A "storage device" is a hardware component of a computer for persistently storing data.

[1601] The "means for taking regular photographs" refers to a function or device that automatically takes photographs of the room based on a set schedule.

[1602] A "newly captured image of the room" is an image that is periodically captured and shows the latest state of the room.

[1603] "Means for uploading images to a storage device" refers to a process or system that sends captured images to a storage device such as a cloud or server.

[1604] The "level of clutter in a room" is an index that quantifies the degree of clutter, dirt, etc., compared to the initial state of the room.

[1605] "Means of quantifying" refers to algorithms or programs that use image processing technology to express the tidiness of a room as a number.

[1606] A "user terminal" is a device such as a smartphone, tablet, or PC that a user uses to access the system.

[1607] "Means for accepting requests for cleaning advice" refers to a function or chat system that allows users to send questions about cleaning methods and tips.

[1608] The "means for providing appropriate cleaning advice" refers to algorithms and databases that provide appropriate cleaning methods in response to user questions.

[1609] "Means for recognizing emotions and changing notification content based on emotions" refers to a system that analyzes emotions from the user's facial expressions and voice and adjusts the content of notifications accordingly.

[1610] A "virtual space" is a virtual space reproduced on a computer, and is a digital environment in which users can interact.

[1611] The "means for analyzing the organization of structures" is a system that analyzes the arrangement of items and objects placed in a virtual space and evaluates their organization.

[1612] System configuration

[1613] The system of this invention is composed of a user terminal, a cloud server, a camera, and an emotion engine. The user terminal is a device such as a smartphone, tablet, or PC, and the camera may be built into these devices or an external camera may be used. The cloud server is equipped with an image processing AI and an emotion engine.

[1614] Registering the initial state

[1615] Users organize their virtual store and take a photo of it. This "initial image" represents the ideal state and is uploaded to a cloud server and stored in a storage device. This serves as a baseline for future comparison.

[1616] Regular status checks

[1617] The system is scheduled to periodically take photos of the entire virtual store. The user's device automatically starts the camera at the set time, acquires new images, and uploads them to the cloud server. This schedule can be freely adjusted to suit the user's convenience.

[1618] Analyzing images and determining clutter levels

[1619] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to let the user know it's time to clean.

[1620] Emotion Engine Operation

[1621] The emotion engine recognizes the user's emotions by analyzing their facial expressions, voice data, and input data. For example, if the user is tired, it generates a gentle message such as, "Your virtual store is messy. Why don't you take a break and then tidy it up?"

[1622] Providing notice and advice

[1623] The user's device receives notifications sent from the cloud server and displays the message to the user. If the user has a question about cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information. The emotion engine also works in conjunction with this to recommend advice tailored to the user's emotions and stress level.

[1624] Specific examples

[1625] Initial state registration:

[1626] User: Organize a virtual store and take photos with their smartphone camera.

[1627] Device: Upload the captured images to the cloud server.

[1628] Server: Save the image and set it as a clean reference image.

[1629] Regular imaging and analysis:

[1630] Device: Take a photo of the virtual store every day at 10:00 AM and upload the new image to the cloud server.

[1631] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1632] Example notification:

[1633] If the clutter level exceeds a threshold, the server generates a notification saying, "Your virtual store is messy. Let's tidy it up." and sends it to the user's device. Depending on the emotion recognized by the emotion engine, the notification text is changed to something like, "Why don't you take a short break and then tidy up?"

[1634] Examples of cleaning advice provided:

[1635] When a user uses the chat function on their device to ask, "How should I organize my store?", the server will provide advice such as, "It would be a good idea to gather all your important documents and put them in one place." If the emotion engine determines that the user is feeling stressed, it will recommend a message such as, "Take a relaxing break, then organize your store in an orderly manner."

[1636] Example prompt for a generative AI model:

[1637] "Based on an image, determine how messy a room is. Also, generate a notification message based on the user's current emotion."

[1638] As a result, this system efficiently organizes the virtual store and provides flexible support that can also respond to the user's emotions.

[1639] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1640] Step 1:

[1641] Registering the initial state

[1642] Input: The user organizes the virtual store and takes a photo of the store.

[1643] What happens: A user uses the camera on their smartphone or tablet to capture an ideal image of the store.

[1644] Output: The captured "initial image" is generated.

[1645] Data processing: The initial image is uploaded from the user's device to the cloud server and saved in a storage device.

[1646] Example of operation: A user presses the "initial state registration" button on a smartphone app, takes a picture, and sends it to the cloud server.

[1647] Step 2:

[1648] Periodic image acquisition

[1649] Input: A recurring schedule.

[1650] Specific operation: The scheduling function of the user device automatically starts the camera at the set time (e.g., 10:00 a.m. every day) and captures images of the store.

[1651] Output: The newly captured image is generated.

[1652] Data processing: The captured images are automatically uploaded to a cloud server.

[1653] Example of operation: Every day at 10:00 AM, the smartphone camera automatically starts up, takes pictures of the virtual store, and sends them to the cloud server.

[1654] Step 3:

[1655] Compare images and assess clutter levels

[1656] Input: Initial image, newly captured image.

[1657] How it works: The server uses image processing AI to compare the initial image with the newly captured image.

[1658] Output: A quantified result of clutter level.

[1659] Data calculation: Image processing AI analyzes object recognition, dirt detection, and changes in object placement to quantify the degree of clutter.

[1660] How it works: The server takes the uploaded image, runs an image processing algorithm to analyze the differences between each pixel, and generates a clutter score.

[1661] Step 4:

[1662] Generate and send cleaning notifications

[1663] Input: Quantified clutter level.

[1664] What happens: The server checks if clutter exceeds a set threshold.

[1665] Output: Notification message to let you know when it's time to clean.

[1666] Data calculation: If the clutter level exceeds a threshold, the server generates a notification message.

[1667] Example of operation: If the clutter level exceeds 60%, the cloud server generates a message saying, "The virtual store is messy. Please tidy it up," and sends it to the user's device.

[1668] Step 5:

[1669] Changing notification content with emotion engine

[1670] Input: User facial and voice data.

[1671] Specific operation: The emotion engine analyzes the user's emotions and adaptively changes the notification message.

[1672] Output: Customized notification message depending on the emotion.

[1673] Data calculation: The emotion engine analyzes the user's input data and determines and classifies emotions.

[1674] Example of how it works: If the user is determined to be tired, the message changes to "How about taking a break and then tidying up?"

[1675] Step 6:

[1676] Providing cleaning advice

[1677] Input: A user's request for cleaning advice.

[1678] Specific operation: Use the chat function from the user's device to send a question to the server.

[1679] Output: A good advice message on how to clean.

[1680] Data processing: The cloud server selects and provides the most appropriate cleaning advice from a pre-registered cleaning advice database.

[1681] Example of how it works: A user asks, "How can I organize my store efficiently?" and the server sends the advice, "It would be a good idea to collect all your important documents and keep them in one place."

[1682] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1683] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1684] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1685] [Fourth embodiment]

[1686] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1687] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1688] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1689] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1690] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1691] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1692] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1693] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1694] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1695] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1696] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1697] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1698] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1699] The present invention is a system that automatically determines the messiness of a room and prompts a user to clean at an appropriate time. Specific embodiments of the system will be described below.

[1700] System Configuration

[1701] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[1702] Registering the initial state of the room

[1703] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[1704] Regularly taking photos of the room

[1705] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[1706] Analyzing images and determining clutter levels

[1707] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[1708] Sending cleaning notifications

[1709] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[1710] Providing cleaning advice

[1711] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[1712] Specific examples

[1713] Registering the initial state

[1714] User: Clean up the living room and take a photo with your smartphone camera.

[1715] Device: Uploads captured images to the cloud server.

[1716] Server: Save the image and set it as a clean reference image.

[1717] Regular imaging and analysis

[1718] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[1719] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1720] Notifications when clutter exceeds thresholds

[1721] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1722] Device: Show the user the message "The living room is messy. Time to clean!"

[1723] Providing cleaning advice

[1724] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[1725] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[1726] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[1727] The processing flow will be explained below.

[1728] Registering the initial state of the room

[1729] Step 1:

[1730] User: Clean up the living room and take a picture with the camera.

[1731] The user takes an image of the room using a smartphone or a camera on the device.

[1732] Step 2:

[1733] Device: Uploads captured images to the cloud server.

[1734] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[1735] Step 3:

[1736] Server: Saves the uploaded image and sets it as the reference image.

[1737] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[1738] Regularly photograph and analyze the room conditions

[1739] Step 4:

[1740] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[1741] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[1742] Step 5:

[1743] Device: Automatically take a photo of the living room at the set time.

[1744] The camera will automatically start up and capture the entire room.

[1745] Step 6:

[1746] Device: Uploads newly captured images to the cloud server.

[1747] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[1748] Step 7:

[1749] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[1750] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[1751] Step 8:

[1752] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[1753] A comparison algorithm quantifies the clutter level of a room.

[1754] Sending cleaning notifications

[1755] Step 9:

[1756] Server: Compare the calculated clutter level with a threshold.

[1757] Check whether the quantified clutter level exceeds a set threshold.

[1758] Step 10:

[1759] Server: Generates cleaning notification if threshold is exceeded.

[1760] A notification message is generated and the message content and recipient information are added.

[1761] Step 11:

[1762] Server: Sends notifications to the user device.

[1763] Use the push notification system to send notification messages to the user's device.

[1764] Step 12:

[1765] Terminal: Show cleaning notification to user.

[1766] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[1767] Providing cleaning advice

[1768] Step 13:

[1769] User: Uses the device's chat function to ask about cleaning tips.

[1770] The user opens the chat screen, types a question, and sends it to the server.

[1771] Step 14:

[1772] Server: Receives questions from users and searches for appropriate advice.

[1773] The server searches the database for relevant cleaning advice and prepares a response.

[1774] Step 15:

[1775] Server: Sends advice to users via chat function.

[1776] Appropriate advice content is sent to the terminal as a chat message.

[1777] Step 16:

[1778] User: Clean according to the advice provided.

[1779] The user performs cleaning based on the advice received.

[1780] Example 1

[1781] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1782] In modern life, efficiently cleaning a room amidst busy daily routines is a difficult task. In particular, accurately understanding the state of clutter in a room and cleaning at the optimal time can be a burden for many people. Conventional methods require users to check the state of the room themselves and determine the need for cleaning each time, which is a very time-consuming operation. Therefore, there is a need for a system that solves these problems and allows users to keep their rooms clean efficiently and comfortably.

[1783] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1784] In this invention, the server includes a means for quantifying the degree of messiness of a room, a means for using image processing AI to recognize objects, detect dust and dirt, and analyze changes in the placement of objects, a means for notifying the user of the need for cleaning through a notification function, and a means for providing advice in response to the user's questions through a chat function, thereby enabling the user to automatically grasp the degree of messiness of the room and clean at the appropriate time.

[1785] The "initial state of the room" indicates a state in which the room is neat and tidy, and is an image that serves as a reference for evaluating the degree of messiness in the future.

[1786] The "means for periodically photographing the entire room" is a function for automatically acquiring images of the room at set time intervals.

[1787] "Image of initial state" refers to an image of the room's initial state that is captured and stored on a cloud server.

[1788] "Quantifying the degree of clutter" is the process of analyzing the clutter level of a room and expressing it as a number according to certain standards.

[1789] "Notification when threshold is exceeded" means generating and sending a notification to the user informing them of the need to clean when the quantified clutter level exceeds a set reference value.

[1790] "Image processing AI" is an artificial intelligence technology that analyzes captured images to recognize objects, detect dirt, and determine changes in the placement of objects.

[1791] "Object recognition" is a technology that identifies objects in an image and determines their location and type.

[1792] "Dust and dirt detection" is a technology that identifies dust and dirt in an image and confirms their presence.

[1793] The "notification function" is a function that sends messages to notify the user of specific events or actions.

[1794] The "chat function" is a function that allows users to communicate with the server in real time via text messages.

[1795] "Providing cleaning advice" refers to suggesting optimal cleaning methods and ideas in response to the user's questions and requests.

[1796] The present invention is a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time, and specific embodiments of the system will be described below.

[1797] System Configuration

[1798] This system is based on a user device, a cloud server, and a camera. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with image processing AI, which determines the level of clutter. The camera may be built into the user device or a dedicated camera device.

[1799] Registering the initial state of the room

[1800] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[1801] Regularly taking photos of the room

[1802] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[1803] Analyzing images and determining clutter levels

[1804] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[1805] Sending cleaning notifications

[1806] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. It's time to clean!" This notification function allows the user to clean at the appropriate time.

[1807] Providing cleaning advice

[1808] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "What is the best way to efficiently clean dust off the floor?" the server might send advice such as "It would be a good idea to use a microfiber cloth" to the device.

[1809] Specific examples

[1810] Registering the initial state

[1811] User: Clean up the living room and take a photo with your smartphone camera.

[1812] Device: Uploads captured images to the cloud server.

[1813] Server: Save the image and set it as a clean reference image.

[1814] Regular imaging and analysis

[1815] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[1816] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1817] Notifications when clutter exceeds thresholds

[1818] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1819] Device: Show the user the message "The living room is messy. Time to clean!"

[1820] Providing cleaning advice

[1821] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[1822] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it."

[1823] In this way, the system provides support to users to efficiently clean their rooms and maintain a clean environment.

[1824] Prompt Sentence Examples

[1825] Below is an example of a prompt sentence to input into the image processing AI model.

[1826] Compare the initial image with the new room image and quantify the clutter level. Evaluate based on object recognition, dust / dirt detection, and changes in object placement, and generate a notification if the clutter level exceeds a threshold.

[1827] These embodiments allow the user to automatically understand the messiness of the room and clean it at the appropriate time.

[1828] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1829] Step 1:

[1830] Initial image registration

[1831] User: Prepare a clean room. Specifically, clean the room and make sure everything is neat and tidy. In this state, start the smartphone camera and take a picture of the entire room.

[1832] Input: Image of the initial room state

[1833] Terminal: Uploads images taken by the user to the cloud server.

[1834] Output: Images uploaded to the cloud server

[1835] Server: Receives the uploaded image and saves it in a storage device as an "initial image."

[1836] Output: Saved initial image

[1837] Step 2:

[1838] Regular imaging

[1839] Device: Automatically activate the camera at a specified time according to a pre-set schedule, for example, 10:00 AM every day.

[1840] Input: Specified schedule

[1841] Terminal: Take a photo of the entire room again to obtain a new image of the room.

[1842] Output: Newly acquired room image

[1843] Step 3:

[1844] Uploading an image

[1845] Device: Newly acquired images are automatically uploaded to the cloud server.

[1846] Input: A newly acquired image of a room.

[1847] Output: New image uploaded to the cloud server

[1848] Server: Receives new images sent from the device and temporarily stores them in a storage device.

[1849] Output: New image saved

[1850] Step 4:

[1851] Determining the level of clutter

[1852] Server: Starts the process of comparing the new image with the initial image.

[1853] Input: New image, initial image

[1854] Server: Uses image processing AI to recognize objects, detect dust and dirt, and analyze changes in object placement.

[1855] Data processing: object recognition, dust and dirt detection, analysis of changes in object placement

[1856] Server: Based on the analysis results, the degree of clutter is quantified and it is determined whether this number exceeds a preset threshold.

[1857] Data calculation: Quantifying clutter level and determining threshold

[1858] Output: Numerical value of clutter level and threshold exceedance judgment

[1859] Step 5:

[1860] Sending cleaning notifications

[1861] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1862] Input: Threshold exceedance judgment

[1863] Server: Sends notifications to the user's device.

[1864] Output: Cleaning timing notification

[1865] Device: Receives a notification from the server and displays the message "Your room is messy. Time to clean!" to the user.

[1866] Output: The message that is displayed to the user

[1867] Step 6:

[1868] Providing cleaning advice

[1869] User: If necessary, use the chat function on the device to type in a cleaning question. For example, "How can I efficiently clean the dust off the floor?"

[1870] Input: User question

[1871] Terminal: Sends the user's question to the cloud server.

[1872] Output: Questions sent to the cloud server

[1873] Server: Searches for pre-registered cleaning advice information based on the question.

[1874] Data processing: Search for advice information based on the question

[1875] Server: Creates optimal advice as a text message and sends it to the user's device.

[1876] Output: Advice in the form of a text message

[1877] Terminal: Receives advice from the server and displays it to the user. For example, it displays a message such as "It would be a good idea to use a microfiber cloth."

[1878] Output: An advisory message that is displayed to the user.

[1879] This is the specific process flow of this system. By using this system, users can automatically determine the messiness of their room and clean it at the appropriate time.

[1880] (Application example 1)

[1881] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1882] Regular cleaning is essential to maintaining cleanliness in physical stores. However, store staff often miss cleaning opportunities or are unfamiliar with the optimal cleaning methods. This can result in a deterioration of the store environment, lower customer satisfaction, and hygiene issues. Therefore, there is a need for a system that can automatically and efficiently manage cleaning in physical stores and prompt cleaning at the appropriate times.

[1883] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1884] In this invention, the server includes a means for periodically taking images with a camera installed in a specific area of ​​the store and uploading them to a cloud server, a means for an image processing AI on the cloud server to evaluate the degree of clutter and generate a notification as needed and send it to a user terminal, a means for accepting cleaning advice requests from users, and a means for providing appropriate cleaning advice based on the accepted request. This makes it possible to constantly monitor the environment within the store and clean at the appropriate time.

[1885] The "initial state" is a reference image that shows the ideal clean state of a room or store.

[1886] A "storage device" is a device that stores data, and includes cloud servers and the like.

[1887] A "cloud server" is a server that stores and processes data via the Internet.

[1888] "Image processing AI" is an artificial intelligence technology that analyzes images and extracts information for specific purposes.

[1889] "Camera" means a device that captures images or videos, including those built into smartphones and dedicated fixed cameras.

[1890] "Clutter level" is a numerical indication of the cleanliness and tidiness of a room or store.

[1891] A "threshold" is a reference value set for determining a specific condition.

[1892] A "notification" is a message that notifies the user of specific information.

[1893] A "user terminal" is a device that is directly operated by a user, and includes smartphones and tablets.

[1894] An "advice request" is a request by a user for specific information or advice.

[1895] "Cleaning advice" means providing information on cleaning methods and cleaning tips.

[1896] A "specific area" is a specific area within the store that is to be cleaned.

[1897] This invention is a system for streamlining cleaning management in brick-and-mortar stores, and uses a user terminal, a cloud server, and a camera as key components. This system is designed to determine the clutter level of a room and prompt cleaning at the appropriate time. Specific embodiments of the system are described below.

[1898] First, store staff take a photo of a specific area of ​​the store after cleaning and upload the image to a cloud server as an "initial image." This image shows the ideal cleanliness of the store and serves as a standard for future comparisons.

[1899] The system is then scheduled to periodically capture images of specific areas of the store. For example, every day at 2:00 p.m., the camera automatically captures images of the area and uploads the newly captured images to a cloud server, where they are analyzed using image processing AI.

[1900] The cloud server compares the newly uploaded image with the initial image and quantifies it based on object recognition and changes in clutter level. If this quantification exceeds a pre-defined threshold, the cloud server generates a notification and sends a message to the user device, such as "A specific area is cluttered. Time to clean!"

[1901] The user device also has a chat function, allowing store staff to ask questions about cleaning methods. For example, by entering a prompt such as, "Please tell me how to improve the cleaning efficiency of the cash register counter," appropriate cleaning advice is provided from the cloud server's knowledge base. The server then sends specific advice to the device, such as, "It is effective to disinfect the cash register counter once a week."

[1902] Hardware and software used

[1903] Hardware:

[1904] Camera: A camera built into a smartphone or a fixed camera installed in a store.

[1905] User devices: smartphones, tablet devices.

[1906] software:

[1907] Cloud server: Stores and processes data.

[1908] Image processing AI: Analyzes images and quantifies the degree of clutter.

[1909] Notification system: Sends messages to user terminals.

[1910] Chatbot: Provides cleaning advice in response to user questions.

[1911] Specific examples

[1912] Store staff clean the cash register area every day at 8:00 a.m., then take a photo of the area with their smartphone and upload it to a cloud server.

[1913] The system takes an up-to-date photo of the checkout area every day at 2 p.m. and uploads it to a cloud server. Image processing AI evaluates the clutter level and sends notifications as necessary.

[1914] Staff will clean the area again based on the notification they receive.

[1915] Prompt Sentence Examples

[1916] "Please tell me how to improve the cleaning efficiency of the cash register counter."

[1917] "What is the best way to respond if the seating area is messy?"

[1918] This system allows physical stores to maintain a clean environment at all times, contributing to improved customer satisfaction and hygiene.

[1919] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1920] Step 1:

[1921] A user cleans a specific area of ​​the store and takes a picture of that area with a camera. The input is an image taken by the camera, which is uploaded to a cloud server. The server stores the received image in a storage device as an "initial state image." This sets a reference for the initial state.

[1922] Step 2:

[1923] The device periodically takes photos of specific areas of the store according to a set schedule, and the new images are input and uploaded to a cloud server, which uses image processing AI to analyze the new images and generate data for object recognition and clutter assessment.

[1924] Step 3:

[1925] The cloud server uses image processing AI to compare the new image with the initial image. The input is the new image and the initial image, which are compared to quantify the degree of clutter. This process outputs the clutter level as a number.

[1926] Step 4:

[1927] The server determines whether the numerically-quantified clutter level exceeds a set threshold. The input is the numerically-quantified clutter level, which is evaluated. If the threshold is exceeded, the server generates a notification informing the user that it is time to clean and sends it to the user's device. This notification is then displayed on the user's device.

[1928] Step 5:

[1929] After receiving the notification, the user uses the chat function on the device to request cleaning advice from the server. The input is a question (prompt sentence) from the user. The server searches for appropriate advice from a pre-registered knowledge base and generates this information.

[1930] Step 6:

[1931] The server sends the generated cleaning advice to the user's device. The input is the knowledge base information in the server and the user's question, and the advice generated based on this is output. The advice is displayed on the user's device, and the physical store is cleaned efficiently based on this.

[1932] Following these steps will help keep your store environment clean, leading to more efficient cleaning and improved customer satisfaction.

[1933] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1934] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[1935] System Configuration

[1936] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[1937] Registering the initial state of the room

[1938] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[1939] Regularly taking photos of the room

[1940] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[1941] Analyzing images and determining clutter levels

[1942] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to the user's device informing them that it's time to clean.

[1943] Emotion Engine Operation

[1944] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[1945] Sending cleaning notifications

[1946] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[1947] Providing cleaning advice

[1948] If a user has trouble with cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. For example, in response to a question such as "How can I efficiently clean the dust off the floor?" the server might send advice such as "It would be good to use a microfiber cloth" to the device. The emotion engine can also provide specific advice for reducing stress by taking into account the user's stress level.

[1949] Specific examples

[1950] Registering the initial state

[1951] User: Clean up the living room and take a photo with your smartphone camera.

[1952] Device: Uploads captured images to the cloud server.

[1953] Server: Save the image and set it as a clean reference image.

[1954] Regular imaging and analysis

[1955] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[1956] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[1957] Notifications when clutter exceeds thresholds

[1958] Server: When clutter levels exceed a threshold, it generates a notification to let you know it's time to clean.

[1959] Terminal: Shows the user a message saying "The living room is messy. Time to clean!" The message may change depending on the emotion recognized by the emotion engine.

[1960] Providing cleaning advice

[1961] User: Uses the chat function on the device to ask the server questions about cleaning tips.

[1962] Server: In response to the user's question, sends advice such as "It may be effective to remove the cushion cover before washing it." The emotion engine provides specific advice for reducing stress according to the user's stress level.

[1963] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[1964] The processing flow will be explained below.

[1965] Registering the initial state of the room

[1966] Step 1:

[1967] User: Clean up the living room and take a picture with the camera.

[1968] The user takes an image of the room using a smartphone or a camera on the device.

[1969] Step 2:

[1970] Device: Uploads captured images to the cloud server.

[1971] The device automatically or at the user's command sends the captured images to a cloud server, where they may be compressed or encoded.

[1972] Step 3:

[1973] Server: Saves the uploaded image and sets it as the reference image.

[1974] The server saves the image in a specific directory and records the information as a "reference image" along with the path in the database.

[1975] Regularly photograph and analyze the room conditions

[1976] Step 4:

[1977] Device: Schedule regular photo shoots (e.g., every day at 10:00 AM).

[1978] The device uses a scheduling function to set the camera to automatically take pictures of the room at a specified time.

[1979] Step 5:

[1980] Device: Automatically take a photo of the living room at the set time.

[1981] The camera will automatically start up and capture the entire room.

[1982] Step 6:

[1983] Device: Uploads newly captured images to the cloud server.

[1984] The captured image is sent from the device to the cloud server, where it is compressed and encoded as needed.

[1985] Step 7:

[1986] Server: Uses image processing AI to compare the newly uploaded image with the reference image.

[1987] The server's image processing AI performs object recognition and detects dust and dirt, and analyzes differences in the images.

[1988] Step 8:

[1989] Server: Based on the comparison results, the degree of clutter in the room is quantified.

[1990] A comparison algorithm quantifies the clutter level of a room.

[1991] Sending cleaning notifications

[1992] Step 9:

[1993] Server: Compare the calculated clutter level with a threshold.

[1994] Check whether the quantified clutter level exceeds a set threshold.

[1995] Step 10:

[1996] Server: Generates cleaning notification if threshold is exceeded.

[1997] A notification message is generated and the message content and recipient information are added.

[1998] Step 11:

[1999] Server: Sends notifications to the user device.

[2000] Use the push notification system to send notification messages to the user's device.

[2001] Step 12:

[2002] Terminal: Show cleaning notification to user.

[2003] The device receives the notification and displays the message to the user: "Your room is messy. Time to clean!"

[2004] Emotion Engine Operation

[2005] Step 13:

[2006] Terminal: Collects the user's facial expressions and voice.

[2007] The device's camera and microphone are used to capture the user's facial expressions and voice data.

[2008] Step 14:

[2009] Server: The emotion engine analyzes the acquired data and recognizes the user's emotion.

[2010] The emotion engine uses image and audio analysis to identify the user's emotions (e.g., stress, joy, fatigue).

[2011] Step 15:

[2012] Server: Adaptively change the content of cleaning notifications based on the user's emotions.

[2013] If the user is tired, an encouraging message such as "Let's clean up and refresh ourselves" is generated.

[2014] Providing cleaning advice

[2015] Step 16:

[2016] User: Uses the device's chat function to ask about cleaning tips.

[2017] The user opens the chat screen, types a question, and sends it to the server.

[2018] Step 17:

[2019] Server: Receives questions from users and searches for appropriate advice.

[2020] The server searches the database for relevant cleaning advice and prepares a response.

[2021] Step 18:

[2022] Server: Sends advice to users via chat function.

[2023] Appropriate advice is sent to the device as a chat message. The emotion engine provides specific advice for reducing stress according to the user's stress level.

[2024] Step 19:

[2025] User: Clean according to the advice provided.

[2026] The user performs cleaning based on the advice received.

[2027] In this way, the system not only helps the user efficiently clean the room and maintain a clean environment, but also responds to the user's emotions and provides appropriate support.

[2028] Example 2

[2029] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2030] In today's busy living environment, there is a need for a system that helps users clean their rooms at appropriate times. However, typical cleaning support systems send uniform notifications without considering the user's emotional state, which can be annoying for users. Furthermore, they lack the functionality to provide specific cleaning advice, and therefore do not provide sufficient support for users to effectively organize their rooms. To solve this problem, a system is needed that recognizes the user's emotions, adaptively changes cleaning notifications, and provides specific cleaning advice.

[2031] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2032] In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state using image processing means, means for performing object recognition and quantifying the degree of messiness of the room, means for generating a notification informing the user that it is time to clean when the quantified degree of messiness exceeds a threshold and sending the notification to the user terminal, means for recognizing the user's emotions, means for adaptively changing the content of the notification based on the user's emotions, means for receiving a request for cleaning advice from the user, and means for providing appropriate cleaning advice based on the received request. This allows the user to clean their room efficiently, receive cleaning notifications at appropriate times, and further allows for flexible responses according to the user's emotional state.

[2033] The "means for capturing an image of the initial state of the room" refers to a device or method for capturing an image of the room in its tidy state.

[2034] The "means for saving the captured image of the room in its initial state in a storage device" refers to a device or process for saving the captured image data of the initial state.

[2035] "Means for periodically photographing the entire room" refers to a device or method for automatically or manually photographing the entire room based on a pre-set schedule.

[2036] "Means for uploading newly captured images of the room to a storage device" refers to a device or process that transfers the latest captured image data to a cloud or other storage system.

[2037] "Means for comparing a newly captured image of the room with an image of the initial state using image processing means" refers to an algorithm or device for comparing an image of the initial state with the latest image and detecting differences.

[2038] "Means for performing object recognition and quantifying the degree of clutter in a room" refers to a process or device that uses image processing technology to analyze the presence and placement of objects, and then expresses the degree of clutter as a number based on the results.

[2039] "Means for generating and sending to a user device a notification informing the user that it is time to clean when the quantified clutter level exceeds a threshold" refers to a system or method for creating and sending to a user device a notification encouraging cleaning when the clutter level exceeds a pre-set threshold.

[2040] "Means for recognizing user emotions" refers to technology or devices that analyze a user's facial expressions, voice, text input, etc. to determine the user's emotional state.

[2041] "Means for adaptively changing notification content based on user emotion" refers to a system or process for changing the content or tone of a notification based on a recognized user emotion.

[2042] "Means for accepting cleaning advice requests from users" refers to an interface or method for accepting input from users asking questions or requesting advice about cleaning.

[2043] "Means for providing appropriate cleaning advice based on a received request" refers to a system or method for providing appropriate cleaning advice based on information and knowledge prepared in advance in response to a received user request.

[2044] MODE FOR CARRYING OUT THE INVENTION

[2045] This invention combines a system that automatically determines the messiness of a room and prompts the user to clean at an appropriate time with an emotion engine that recognizes the user's emotions. Specific embodiments of this system are described below.

[2046] System Configuration

[2047] This system is composed of a user device, a cloud server, a camera, and an emotion engine. The user device is a smartphone or tablet device that is used to take images of the room and send the data to the server. The cloud server is equipped with an image processing AI and an emotion engine, and determines the level of clutter and recognizes the user's emotions. The camera may be built into the user device or a dedicated camera device.

[2048] Registering the initial state of the room

[2049] First, the user takes a photo of the room in its clean state. This image is uploaded to the cloud server as an "initial image," which is then stored in a storage device. The initial image shows the ideal state of the room and serves as a reference point for future comparisons.

[2050] Examples:

[2051] User: Tidy up the living room and take a photo with their smartphone camera.

[2052] Device: Upload the captured images to the cloud server.

[2053] Server: Save the image and set it as a clean reference image.

[2054] Regularly taking photos of the room

[2055] The system is scheduled to periodically capture images of the entire room. The user device automatically starts the camera at the set time and captures images of the room. The newly captured images are then uploaded to the cloud server.

[2056] Examples:

[2057] Device: Take a photo of your living room every day at 10am and upload the new image to a cloud server.

[2058] Analyzing images and determining clutter levels

[2059] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. Software used here includes TensorFlow and OpenCV. This AI quantifies the level of clutter based on object recognition, dust and dirt detection, and changes in object placement. If the quantified clutter level exceeds a preset threshold, the server generates a notification to the user's device informing them that it's time to clean.

[2060] Examples:

[2061] Server: Image processing AI compares the new image with the initial image and calculates the degree of clutter.

[2062] Server: When the clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[2063] Emotion Engine Operation

[2064] The emotion engine analyzes the user's facial expressions, voice, and input data to recognize the user's emotions. Based on the emotion recognition results, the emotion engine adaptively changes the content of the cleaning notification. For example, if the user is tired, it will send an encouraging message such as "Let's clean up and feel refreshed." The emotion engine can also determine the user's stress level and provide cleaning advice to reduce stress.

[2065] Examples:

[2066] Server: Adaptively change the content of notifications for users based on emotions recognized by the emotion engine.

[2067] Sending cleaning notifications

[2068] The user device receives the notification sent from the cloud server and displays a message to the user such as "Your room is messy. Time to clean!" The content of the notification may change based on the user's emotions recognized by the emotion engine.

[2069] Examples:

[2070] Device: Displays received notifications to the user and plays notifications and alarms as needed.

[2071] Providing cleaning advice

[2072] If a user has trouble with cleaning methods, they can use the chat function on their device to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information in response to the user's question. The emotion engine also takes the user's stress level into consideration and provides specific advice for stress reduction.

[2073] Examples:

[2074] User: Uses the chat function on their device to ask the server for cleaning tips.

[2075] Server: Responds to the user's question with advice such as "It may be effective to remove the cushion cover before washing."

[2076] Prompt Sentence Examples

[2077] Example prompt for initial registration:

[2078] After you have tidied up your living room, take a picture with your smartphone camera and upload the image to the cloud server.

[2079] Cleaning notification prompt example:

[2080] Generate a cleaning notification message when the emotion engine recognizes that the user is tired. Example: Clean up and refresh yourself.

[2081] Example cleaning advice prompts:

[2082] If a user asks how to efficiently dust the floor, provide appropriate advice taking into account the results of the sentiment engine. For example: Using a microfiber cloth is a good idea.

[2083] In this way, the system allows users to clean their rooms efficiently, receive timely notifications, and respond flexibly to the user's emotions.

[2084] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2085] Step 1:

[2086] User: Take a picture of the initial state of the room with the camera.

[2087] Input: A clean image of a room.

[2088] Output: Pristine image data.

[2089] Specific operation: The user opens the camera app on their smartphone, clicks the "Register initial state" button, and takes a photo of the room.

[2090] Step 2:

[2091] Device: Upload the captured initial image to the cloud server.

[2092] Input: Pristine image data.

[2093] Output: Image data stored on a cloud server.

[2094] What it does: It establishes an internet connection to instantly upload images taken by the smartphone to a cloud server.

[2095] Step 3:

[2096] Server: Saves the received image data in its initial state in a storage device.

[2097] Input: Image data uploaded to the cloud server.

[2098] Output: Pristine image data stored in a database.

[2099] Specific operation: The cloud server receives the image data and stores it in a database.

[2100] Step 4:

[2101] Device: Automatically activate the camera at a set time (e.g., 10:00 AM every day) and capture images of the room.

[2102] Input: Scheduled timing.

[2103] Output: Image data of the new room.

[2104] Specific operation: The device will start the camera at 10:00 AM every day and automatically take pictures of the room.

[2105] Step 5:

[2106] On your device: Upload any new images you take to the cloud server.

[2107] Input: Image data of the new room.

[2108] Output: New image data stored on the cloud server.

[2109] Specific operation: Data transfer is performed to upload newly captured images to the cloud server.

[2110] Step 6:

[2111] Server: Compares the new image with the initial image using image processing techniques.

[2112] Input: New image data and initial image data.

[2113] Output: Image comparison results.

[2114] How it works: The cloud server uses image processing AI to compare the new image with the initial image using tools such as TensorFlow and OpenCV.

[2115] Step 7:

[2116] Server: Performs object recognition and quantifies the degree of clutter in a room.

[2117] Input: Image comparison results.

[2118] Output: Quantified clutter level.

[2119] What it does: It uses an object recognition algorithm to quantify the level of clutter based on the placement of objects, the presence of dust and dirt, etc.

[2120] Step 8:

[2121] Server: When the quantified clutter level exceeds a threshold, a notification is generated and sent to the user's device to notify them that it is time to clean.

[2122] Input: Quantified clutter level.

[2123] Output: Notification that it's time to clean.

[2124] Specific behavior: When the clutter level exceeds a threshold, the server creates a notification and sends it to the user's device.

[2125] Step 9:

[2126] Server: Recognizes user emotions.

[2127] Input: User facial, voice, and text input data.

[2128] Output: User emotion recognition results.

[2129] Specific operation: Using the emotion engine, facial expression recognition and voice analysis are performed to determine the user's emotions.

[2130] Step 10:

[2131] Server: Adaptively change notification content based on user emotions.

[2132] Input: User emotion recognition results.

[2133] Output: Notification content tailored based on sentiment.

[2134] Specific behavior: Depending on the recognized user emotion, for example, if it is determined that the user is "tired," a message such as "Let's clean up and refresh" is generated.

[2135] Step 11:

[2136] Terminal: Displays notifications sent from the cloud server to the user.

[2137] Input: Notification from the cloud server.

[2138] Output: A notification message that will be displayed on the user's terminal.

[2139] Specific behavior: The device displays the received notification to the user and plays a notification sound or alarm if necessary.

[2140] Step 12:

[2141] User: Uses the chat function to ask the server for cleaning tips.

[2142] Input: The question asked by the user.

[2143] Output: Query data to the server.

[2144] Specific behavior: The user opens the chat function on their device and types a question about cleaning.

[2145] Step 13:

[2146] Server: Provides appropriate cleaning advice based on the user's request.

[2147] Input: The question asked by the user.

[2148] Output: Cleaning advice message.

[2149] Specific operation: The server generates specific advice for the user's question based on pre-registered information and replies via the chat function.

[2150] (Application example 2)

[2151] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2152] In conventional virtual stores, managers had to manually check the level of clutter and tidiness within the store and periodically tidy up. However, this process required time and effort, and it was difficult for the manager to respond appropriately based on their emotions and the situation. Furthermore, in order to improve the user experience, a system was needed that could automatically determine the level of tidiness in the virtual space and perform efficient maintenance.

[2153] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing an image of the room in its initial state, means for saving the captured image of the room in its initial state in a storage device, means for periodically capturing an image of the entire room, means for uploading newly captured images of the room to the storage device, means for comparing the newly captured image of the room with the image of the initial state, means for quantifying the degree of clutter in the room, means for generating a notification informing the user that cleaning is necessary when the quantified degree of clutter exceeds a threshold and sending the notification to the user terminal, means for accepting a request for cleaning advice from a user, means for providing appropriate cleaning advice based on the accepted request, means for recognizing the user's emotions and changing the content of the notification based on the emotions, and means for analyzing the organization of structures in the virtual space. This enables the server to automatically determine the degree of clutter in a virtual store, efficiently issue maintenance notifications, and appropriately respond to the manager's emotions and circumstances.

[2154] The "initial state of the room" is an image of a tidy and tidy room taken when the user first registers with the system.

[2155] A "storage device" is a hardware component of a computer for persistently storing data.

[2156] The "means for taking regular photographs" refers to a function or device that automatically takes photographs of the room based on a set schedule.

[2157] A "newly captured image of the room" is an image that is periodically captured and shows the latest state of the room.

[2158] "Means for uploading images to a storage device" refers to a process or system that sends captured images to a storage device such as a cloud or server.

[2159] The "level of clutter in a room" is an index that quantifies the degree of clutter, dirt, etc., compared to the initial state of the room.

[2160] "Means of quantifying" refers to algorithms or programs that use image processing technology to express the tidiness of a room as a number.

[2161] A "user terminal" is a device such as a smartphone, tablet, or PC that a user uses to access the system.

[2162] "Means for accepting requests for cleaning advice" refers to a function or chat system that allows users to send questions about cleaning methods and tips.

[2163] The "means for providing appropriate cleaning advice" refers to algorithms and databases that provide appropriate cleaning methods in response to user questions.

[2164] "Means for recognizing emotions and changing notification content based on emotions" refers to a system that analyzes emotions from the user's facial expressions and voice and adjusts the content of notifications accordingly.

[2165] A "virtual space" is a virtual space reproduced on a computer, and is a digital environment in which users can interact.

[2166] The "means for analyzing the organization of structures" is a system that analyzes the arrangement of items and objects placed in a virtual space and evaluates their organization.

[2167] System configuration

[2168] The system of this invention is composed of a user terminal, a cloud server, a camera, and an emotion engine. The user terminal is a device such as a smartphone, tablet, or PC, and the camera may be built into these devices or an external camera may be used. The cloud server is equipped with an image processing AI and an emotion engine.

[2169] Registering the initial state

[2170] Users organize their virtual store and take a photo of it. This "initial image" represents the ideal state and is uploaded to a cloud server and stored in a storage device. This serves as a baseline for future comparison.

[2171] Regular status checks

[2172] The system is scheduled to periodically take photos of the entire virtual store. The user's device automatically starts the camera at the set time, acquires new images, and uploads them to the cloud server. This schedule can be freely adjusted to suit the user's convenience.

[2173] Analyzing images and determining clutter levels

[2174] The cloud server uses image processing AI to compare the newly uploaded image with the initial image. This AI quantifies the level of clutter based on object recognition, dirt detection, and changes in object placement. If the quantified clutter level exceeds a pre-set threshold, the server generates a notification to let the user know it's time to clean.

[2175] Emotion Engine Operation

[2176] The emotion engine recognizes the user's emotions by analyzing their facial expressions, voice data, and input data. For example, if the user is tired, it generates a gentle message such as, "Your virtual store is messy. Why don't you take a break and then tidy it up?"

[2177] Providing notice and advice

[2178] The user's device receives notifications sent from the cloud server and displays the message to the user. If the user has a question about cleaning methods, they can use the device's chat function to inquire with the server. The cloud server provides appropriate advice based on pre-registered cleaning advice information. The emotion engine also works in conjunction with this to recommend advice tailored to the user's emotions and stress level.

[2179] Specific examples

[2180] Initial state registration:

[2181] User: Organize a virtual store and take photos with their smartphone camera.

[2182] Device: Upload the captured images to the cloud server.

[2183] Server: Save the image and set it as a clean reference image.

[2184] Regular imaging and analysis:

[2185] Device: Take a photo of the virtual store every day at 10:00 AM and upload the new image to the cloud server.

[2186] Server: Image processing AI compares the new image with the reference image and calculates the degree of clutter.

[2187] Example notification:

[2188] If the clutter level exceeds a threshold, the server generates a notification saying, "Your virtual store is messy. Let's tidy it up." and sends it to the user's device. Depending on the emotion recognized by the emotion engine, the notification text is changed to something like, "Why don't you take a short break and then tidy up?"

[2189] Examples of cleaning advice provided:

[2190] When a user uses the chat function on their device to ask, "How should I organize my store?", the server will provide advice such as, "It would be a good idea to gather all your important documents and put them in one place." If the emotion engine determines that the user is feeling stressed, it will recommend a message such as, "Take a relaxing break, then organize your store in an orderly manner."

[2191] Example prompt for a generative AI model:

[2192] "Based on an image, determine how messy a room is. Also, generate a notification message based on the user's current emotion."

[2193] As a result, this system efficiently organizes the virtual store and provides flexible support that can also respond to the user's emotions.

[2194] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2195] Step 1:

[2196] Registering the initial state

[2197] Input: The user organizes the virtual store and takes a photo of the store.

[2198] What happens: A user uses the camera on their smartphone or tablet to capture an ideal image of the store.

[2199] Output: The captured "initial image" is generated.

[2200] Data processing: The initial image is uploaded from the user's device to the cloud server and saved in a storage device.

[2201] Example of operation: A user presses the "initial state registration" button on a smartphone app, takes a picture, and sends it to the cloud server.

[2202] Step 2:

[2203] Periodic image acquisition

[2204] Input: A recurring schedule.

[2205] Specific operation: The scheduling function of the user device automatically starts the camera at the set time (e.g., 10:00 a.m. every day) and captures images of the store.

[2206] Output: The newly captured image is generated.

[2207] Data processing: The captured images are automatically uploaded to a cloud server.

[2208] Example of operation: Every day at 10:00 AM, the smartphone camera automatically starts up, takes pictures of the virtual store, and sends them to the cloud server.

[2209] Step 3:

[2210] Compare images and assess clutter levels

[2211] Input: Initial image, newly captured image.

[2212] How it works: The server uses image processing AI to compare the initial image with the newly captured image.

[2213] Output: A quantified result of clutter level.

[2214] Data calculation: Image processing AI analyzes object recognition, dirt detection, and changes in object placement to quantify the degree of clutter.

[2215] How it works: The server takes the uploaded image, runs an image processing algorithm to analyze the differences between each pixel, and generates a clutter score.

[2216] Step 4:

[2217] Generate and send cleaning notifications

[2218] Input: Quantified clutter level.

[2219] What happens: The server checks if clutter exceeds a set threshold.

[2220] Output: Notification message to let you know when it's time to clean.

[2221] Data calculation: If the clutter level exceeds a threshold, the server generates a notification message.

[2222] Example of operation: If the clutter level exceeds 60%, the cloud server generates a message saying, "The virtual store is messy. Please tidy it up," and sends it to the user's device.

[2223] Step 5:

[2224] Changing notification content with emotion engine

[2225] Input: User facial and voice data.

[2226] Specific operation: The emotion engine analyzes the user's emotions and adaptively changes the notification message.

[2227] Output: Customized notification message depending on the emotion.

[2228] Data calculation: The emotion engine analyzes the user's input data and determines and classifies emotions.

[2229] Example of how it works: If the user is determined to be tired, the message changes to "How about taking a break and then tidying up?"

[2230] Step 6:

[2231] Providing cleaning advice

[2232] Input: A user's request for cleaning advice.

[2233] Specific operation: Use the chat function from the user's device to send a question to the server.

[2234] Output: A good advice message on how to clean.

[2235] Data processing: The cloud server selects and provides the most appropriate cleaning advice from a pre-registered cleaning advice database.

[2236] Example of how it works: A user asks, "How can I organize my store efficiently?" and the server sends the advice, "It would be a good idea to collect all your important documents and keep them in one place."

[2237] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2238] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2239] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2240] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2241] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2242] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2243] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2244] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2245] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2246] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2247] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2248] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2249] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2250] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2251] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2252] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2253] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2254] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this wa...

Claims

1. A means for photographing the initial state of the room; means for storing the captured image of the initial state of the room in a storage device; A means of periodically photographing the entire room, means for uploading newly captured images of the room to a storage device; a means for comparing the newly captured image of the room with the initial image; A way to quantify the degree of clutter in a room, means for generating a notification to notify the user when it is time to clean when the quantified clutter level exceeds a threshold, and transmitting the notification to the user terminal; A means for accepting a cleaning advice request from a user; A means of providing appropriate cleaning advice based on received requests; A system including:

2. The system of claim 1 further comprising means for uploading the initial image and the newly captured image to a cloud server.

3. 2. The system according to claim 1, wherein a scheduling function is used as a means for periodically taking an image of the entire room.

Images