system

Abstract

We provide a household cleaning assistance system. [Solution] A system including a camera device that periodically receives video data, a video acquisition means that receives the video data in stream format, a video analysis means that preprocesses the received video data and uses an AI model to detect dirt and pests, an analysis result recording means that records the detection results in a database along with a confidence score and includes a timestamp and camera location information, a notification generation means that extracts areas requiring cleaning or attention when the confidence score exceeds a certain threshold and generates an appropriate notification message, a notification transmission means that sends the generated notification message to the user terminal, a user interface means that displays it in real time as a push notification or in-app message, and a chatbot means that receives a question from the user, analyzes it using a natural language processing model, generates a specific answer and displays it via the user interface means.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003] <00了0017>

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern households, especially for dual-income couples who balance work and housework and people with little cleaning know-how, it is difficult to grasp the timing and methods of cleaning, so the living environment is likely to deteriorate. Also, cleaning tasks tend to rely on subjective judgment and are difficult to perform efficiently. Therefore, there is an increasing need for a system that can objectively and efficiently manage cleaning within a household. In addition, it is also a major issue to identify the presence of household pests at an early stage and take appropriate measures. The purpose of the present invention is to provide a household cleaning assistant system for solving these problems.

Means for Solving the Problems

[0005] The present invention provides a system including video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, and chatbot means. Specifically, it periodically receives video data from a camera device installed in the home and analyzes the video data to detect dirt and pests. The detection results are recorded in a database along with a confidence score, and an appropriate notification message is generated as needed. The notification is sent to the user's terminal and displayed in real time. In addition, the user can ask questions about specific cleaning methods and appropriate tools via the chatbot interface. This allows for an objective understanding of the timing and method of cleaning, enabling efficient cleaning.

[0006] "Video acquisition means" refers to a device that periodically receives video data from a camera installed in a home and receives that video data in a stream format.

[0007] "Video analysis means" refers to a device that preprocesses received video data and uses an AI model to detect dirt and pests.

[0008] "Analysis result recording means" refers to a device that records detection results along with a confidence score in a database, and that data also includes a timestamp and camera location information.

[0009] A "notification generation means" refers to a device that extracts areas requiring cleaning or attention when the confidence score exceeds a certain threshold, and generates an appropriate notification message.

[0010] "Notification transmission means" refers to a device that sends a generated notification message to the user's terminal and displays it in real time as a push notification or in-app message.

[0011] "User interface means" refers to an interface that allows a user to interact with a system, input information, and receive notifications.

[0012] A "chatbot" refers to a device that receives questions from users, analyzes them using a natural language processing model, generates specific answers, and displays them through a user interface. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the language used in the following description will be explained.

[0016] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[0035] Camera image acquisition and analysis

[0036] server

[0037] The server periodically receives video data from camera devices installed within the home. Each camera is located in a different place within the home and transmits video in a stream format at regular intervals.

[0038] The received video data is first pre-processed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion.

[0039] The pre-processed video frames are sent to an AI model to detect dirt and pests. The AI ​​model uses deep learning algorithms such as pre-trained convolutional neural networks (CNNs).

[0040] Detection results are recorded in the database along with a confidence score. The data also includes location information and timestamps of the detected area.

[0041] Generating and sending notifications

[0042] server

[0043] The system periodically checks the latest detection results from the database, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention.

[0044] Based on the areas that need cleaning or where pests have been detected, the system generates appropriate notification messages. For example, it might generate a message like, "The kitchen drain is dirty. It's time to clean it."

[0045] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[0046] terminal

[0047] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[0048] Chatbot support

[0049] User

[0050] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[0051] terminal

[0052] Questions from users are sent to the server in real time.

[0053] server

[0054] The server analyzes the received question using a natural language processing model and generates an appropriate answer. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[0055] The generated response is sent back to the user's device and displayed in the chatbot interface.

[0056] Specific example

[0057] For example, suppose a camera installed in the kitchen drain sends video data at 7:00 AM one day. The server receives this video data and analyzes it using an AI model. As a result of the analysis, if dirt is detected in the drain, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server will respond with specific instructions. This allows the user to clean effectively.

[0058] Thus, the present invention makes it possible to clean the home efficiently and effectively, and to maintain a clean living environment on a daily basis.

[0059] The following describes the processing flow.

[0060] Camera image acquisition and analysis

[0061] Step 1: Acquire video

[0062] server

[0063] The server periodically receives video data from each camera installed within the home. Since the video data is transmitted in a stream format, real-time analysis is possible.

[0064] Step 2: Pretreatment

[0065] server

[0066] The received video data is first pre-processed. This pre-processing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[0067] Step 3: Video Analysis

[0068] server

[0069] Pre-processed video frames are input into the AI ​​model. The AI ​​model uses a pre-trained convolutional neural network (CNN) to detect dirt and pests in the images. The detection results are output along with a confidence score.

[0070] Step 4: Saving the analysis results

[0071] server

[0072] The detection results and confidence scores are recorded in a database. The record also includes timestamps of the video frames and camera location information. This stores information that will be useful for later reference.

[0073] Generating and sending notifications

[0074] Step 1: Extract the necessary notifications

[0075] server

[0076] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[0077] Step 2: Generate notification message

[0078] server

[0079] Based on the extracted information, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." is generated.

[0080] Step 3: Sending a notification message

[0081] server

[0082] The generated notification message is sent to the user's device. The notification is sent in real time so that the user can understand it immediately.

[0083] Step 4: Displaying notification messages

[0084] terminal

[0085] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to respond quickly.

[0086] Chatbot support

[0087] Step 1: Enter your question

[0088] User

[0089] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[0090] Step 2: Submit your question

[0091] terminal

[0092] The entered questions are sent to the server in real time.

[0093] Step 3: Analyzing the Question

[0094] server

[0095] The received question is analyzed using a natural language processing model. This analysis helps understand the user's intent and generates an appropriate answer.

[0096] Step 4: Generating the answer

[0097] server

[0098] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0099] Step 5: Submit your response

[0100] server

[0101] The generated response is sent to the user's device.

[0102] Step 6: Display the answer

[0103] terminal

[0104] The received response is displayed in the chatbot interface. The user can then use this displayed information to perform specific cleaning steps.

[0105] This series of processing steps allows users to clean their homes efficiently and effectively. Furthermore, this process helps maintain cleanliness in the household while reducing the user's burden.

[0106] (Example 1)

[0107] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0108] Household cleaning is time-consuming and laborious, and early detection of dirt and pests is particularly difficult. Furthermore, a lack of information on cleaning methods and tools often prevents proper treatment. This contributes to the problem of difficulty in maintaining cleanliness in the home.

[0109] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0110] In this invention, the server includes means for acquiring video, means for pre-processing video data, means for analyzing using an AI model, means for recording analysis results, means for generating notification messages, means for sending notification messages, user interface means, and chatbot means. This makes it possible to detect and notify about dirt and pests in the home at an early stage and provide information to the user quickly.

[0111] "Video acquisition means" refers to a device that periodically receives video data from a recording device installed in a home.

[0112] "Means for pre-processing video data" refers to a device or function that performs processing such as noise reduction, resolution conversion, and color space conversion on received video data.

[0113] "Methods of analysis using AI models" refers to devices or functions that analyze pre-processed video data using artificial intelligence technologies such as convolutional neural networks (CNNs) to detect dirt and pests.

[0114] "Means for recording analysis results" refers to a device or function that stores the detection results of dirt and pests obtained by the AI ​​model in a database along with a confidence score.

[0115] "Means for generating notification messages" refers to a device or function that creates appropriate notification messages for areas requiring cleaning or areas where pests have been detected, based on the analysis results.

[0116] "Means for sending notification messages" refers to a device or function that sends generated notification messages to the user's terminal in real time.

[0117] "User interface means" refers to a device or application that provides an interface for a user to communicate with a system.

[0118] A "chatbot" refers to an artificial intelligence-based conversational agent that automatically generates answers to questions from users about cleaning methods and tools to be used.

[0119] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[0120] Camera image acquisition and analysis

[0121] server

[0122] The server periodically receives video data from recording devices installed within the home. These devices can be standard webcams. Each camera is located in a different part of the home (e.g., kitchen, living room, entryway) and transmits video in a stream format at regular intervals.

[0123] The received video data is first pre-processed. This pre-processing includes noise reduction (using Gaussian filters, etc.), resolution conversion (converting to HD resolution), and color space conversion (converting from RGB to grayscale). Pre-processing is performed using image processing libraries such as OpenCV.

[0124] Preprocessed video frames are sent to an AI model using a convolutional neural network (CNN). This model is built with deep learning libraries such as TensorFlow and PyTorch. The AI ​​model detects dirt and pests in the video and generates a confidence score. The detection results are recorded in a database along with the confidence score. The recorded data includes location information and timestamps of the detected areas.

[0125] Generating and sending notifications

[0126] server

[0127] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. Based on the area that needs cleaning or where pests have been detected, it generates an appropriate notification message. For example, it might generate a message such as, "The kitchen drain is dirty. It's time to clean it."

[0128] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[0129] terminal

[0130] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[0131] Chatbot support

[0132] User

[0133] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[0134] terminal

[0135] Questions from users are sent to the server in real time.

[0136] server

[0137] The server analyzes the received question using a natural language processing model and generates an appropriate answer. This process uses generative AI models such as GPT-3. The generated answer is then sent back to the user's terminal and displayed in the chatbot interface.

[0138] Specific examples of operation

[0139] For example, suppose a camera installed in the kitchen drain transmits video data at 7:00 AM one day. The server receives this video data and performs noise reduction and grayscale conversion using a Gaussian filter. Then, it analyzes it with an AI model and detects dirt in the drain with a 95% confidence level. The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in its database and determines that cleaning is necessary. It generates a notification message saying, "Your kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server replies with specific steps: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." This allows the user to clean effectively.

[0140] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0141] Step 1:

[0142] server

[0143] Video acquisition

[0144] Input: Video data transmitted in stream format from a home recording device.

[0145] Output: Video data received by the server

[0146] Example of operation: A camera installed in the kitchen sends video data to a server every 5 minutes. The server receives this data.

[0147] Step 2:

[0148] server

[0149] Preprocessing of video data

[0150] Input: Received video data

[0151] Output: Pre-processed video data (data after noise reduction, resolution conversion, and color space conversion)

[0152] Specific example of operation: The server removes noise from the received video data using a Gaussian filter, converts the resolution to HD, and converts it from RGB to grayscale.

[0153] Step 3:

[0154] server

[0155] AI-based video analysis

[0156] Input: Preprocessed video data

[0157] Output: Detection results for dirt and pests and their confidence scores.

[0158] Specific example of operation: The server inputs pre-processed video into an AI model (for example, a model using CNN) and detects dirt in the kitchen drain with a 95% confidence level.

[0159] Step 4:

[0160] server

[0161] Record of analysis results

[0162] Input: AI model detection results and confidence score

[0163] Output: Recorded detection results (including location information and timestamp)

[0164] Example of operation: The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in the database.

[0165] Step 5:

[0166] server

[0167] Generating notification messages

[0168] Input: Recorded detection results

[0169] Output: Notification message (reminds for cleaning or other actions)

[0170] Example of operation: Based on the recorded data, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it."

[0171] Step 6:

[0172] server

[0173] Sending notification messages

[0174] Input: Generated notification message

[0175] Output: Notification message sent to the user's terminal

[0176] Example of operation: The server sends the generated notification message as a push notification to the user's smartphone.

[0177] Step 7:

[0178] terminal

[0179] User notification display

[0180] Input: Notification message sent from the server

[0181] Output: Notification message displayed to the user

[0182] Example of operation: A notification appears on the user's smartphone saying, "The kitchen drain is dirty. It's time to clean it."

[0183] Step 8:

[0184] User

[0185] Questions for the chatbot

[0186] Input: User questions (e.g., "How do I clean the drain?")

[0187] Output: Question sent to the server

[0188] Example of operation: The user uses a smartphone app to input the question, "How do I clean a drain?"

[0189] Step 9:

[0190] server

[0191] Question analysis and answer generation

[0192] Input: Question from a user

[0193] Output: Generated answer

[0194] Example of operation: The server analyzes the received question using a generative AI model such as GPT-3 and generates the answer, "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[0195] Step 10:

[0196] server

[0197] Submit your response

[0198] Input: Generated answer

[0199] Output: Response sent to the user's terminal

[0200] Example of operation: The generated answer is sent to the user's smartphone.

[0201] Step 11:

[0202] terminal

[0203] Displaying responses to users

[0204] Input: Response sent from the server

[0205] Output: Answer displayed to the user

[0206] Example of operation: The user's smartphone displays the following message: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Clean with disinfectant."

[0207] (Application Example 1)

[0208] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0209] Traditional home cleaning systems were useful for detecting dirt and pests, but they lacked security features to detect suspicious movements or behavior. Therefore, there is a need to effectively address home security. Furthermore, they lacked a means to immediately suggest safety measures to users. Thus, there is a demand for systems that improve not only home cleaning but also overall security.

[0210] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0211] In this invention, the server includes video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, suspicious behavior detection means, and safety measure suggestion means. This makes it possible to detect not only dirt and pests in the home but also suspicious behavior and provide the user with immediate notifications and safety measures.

[0212] "Video acquisition means" refers to a function for periodically receiving video data from a camera device installed in the home.

[0213] "Video analysis means" refers to a function that preprocesses received video data and uses an AI model to analyze dirt, pests, suspicious behavior, etc.

[0214] The "analysis result recording means" is a function for recording the analysis results of detected dirt, pests, and suspicious behavior in a database.

[0215] The "notification generation means" is a function that automatically generates notification messages for the user based on the analysis results.

[0216] A "notification transmission method" is a function for sending generated notification messages to the user's terminal in real time.

[0217] A "user interface means" is a function that provides an interface for a user to interact with the system.

[0218] A "chatbot" is a function that provides an automated response feature to answer questions from users.

[0219] "Suspicious behavior detection means" refers to a function that complements video analysis means and detects suspicious movements and actions.

[0220] The "safety measures suggestion mechanism" is a function that suggests appropriate safety measures to the user when suspicious behavior is detected.

[0221] This invention is a system for efficiently and effectively cleaning and securing a home. This system operates through the cooperation of three entities: a server, a terminal, and a user, and utilizes in-home camera equipment to detect dirt, pests, and suspicious activity.

[0222] Camera image acquisition and analysis

[0223] server

[0224] Acquiring camera footage:

[0225] The server periodically receives video data from camera devices installed within the home. This data is received in stream format using RTSP (Real Time Streaming Protocol). By receiving video data from cameras installed in different locations within the home, the entire house is covered.

[0226] Pre-processing:

[0227] The received video data is first preprocessed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion using the OpenCV library.

[0228] AI analysis:

[0229] The pre-processed video frames are analyzed using deep learning algorithms such as convolutional neural networks (CNNs) based on TensorFlow or PyTorch. This analysis detects dirt, pests, and suspicious behavior.

[0230] Record of results:

[0231] Detection results are recorded in a database (PostgreSQL or MongoDB) along with a confidence score. The data also includes location information and timestamps of the detected areas.

[0232] Generating and sending notifications

[0233] server

[0234] Notification generation:

[0235] The latest detection results are periodically checked from the database, and if the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention. The same determination is made if suspicious behavior is detected.

[0236] Send notification:

[0237] The generated notification message is sent to the user's device. Real-time push notifications are sent using Firebase Cloud Messaging (FCM).

[0238] User terminal

[0239] Receive and display:

[0240] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to know in real time which areas need cleaning and what suspicious activity is happening in the home.

[0241] Chatbot support

[0242] User

[0243] Enter your question:

[0244] Users can use their devices to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?"

[0245] server

[0246] Response generation:

[0247] The system analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, follow these steps: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0248] Send response:

[0249] The generated response is sent back to the user's device and displayed in the chatbot interface.

[0250] Specific example

[0251] For example, if a camera installed in the kitchen detects dirt, the server generates a notification saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. Furthermore, if suspicious activity is detected in the kitchen, a notification saying, "There is suspicious activity in the kitchen. Please check," is also sent. If the user asks the chatbot, "What should I do if there is a suspicious person in the kitchen?", they will receive the answer, "If possible, move to a safe place and call the police. We also recommend activating your security system." This allows the user to take a quick and appropriate action.

[0252] Example of a prompt

[0253] What should I do if a suspicious person is detected by a home security camera?

[0254] Thus, this system integrates and manages household cleaning and security, providing users with immediately useful information and solutions.

[0255] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0256] Step 1: Receiving and pre-processing camera footage

[0257] The server periodically receives video data in stream format using RTSP from a camera device installed in the home. The input is the video stream from the camera, and the output is pre-processed video data that has undergone noise reduction, resolution conversion, and color space conversion.

[0258] Step 2: Detection using AI analysis

[0259] The server analyzes pre-processed video data using a convolutional neural network (CNN) based on TensorFlow or PyTorch. Here, it detects dirt, pests, and suspicious behavior. The input is pre-processed video data, and the output is the detection results for dirt, pests, and suspicious behavior.

[0260] Step 3: Recording the detection results

[0261] The server records the detection results in a database (PostgreSQL or MongoDB). This step also records the location information and timestamps of the detected areas. The input is the detection result from the AI ​​analysis, and the output is the detection result recorded in the database.

[0262] Step 4: Generate Notifications

[0263] The server periodically checks the database for the latest detection results and generates an appropriate notification message if the confidence score exceeds a certain threshold. A similar procedure is used to generate a notification message if suspicious activity is detected. The input is the detection results from the database, and the output is the generated notification message.

[0264] Step 5: Sending Notifications

[0265] The server sends the generated notification message to the user's device in real time using Firebase Cloud Messaging (FCM). The input is the generated notification message, and the output is the notification sent to the user's device.

[0266] Step 6: Receiving and displaying notifications

[0267] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to see in real time where cleaning is needed and any suspicious activity in the home. The input is the notification from the server, and the output is the notification displayed on the terminal.

[0268] Step 7: Enter your question for the chatbot.

[0269] The user uses a terminal to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?" The input is the user's question, and the output is the question data sent to the server.

[0270] Step 8: Question analysis and answer generation

[0271] The server analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." The input is the user's question, and the output is the generated answer.

[0272] Step 9: Submit and view your response

[0273] The server sends the generated response back to the user's terminal and displays it on the chatbot interface. The input is the generated response, and the output is the response displayed on the user's terminal.

[0274] In this way, each step works in conjunction to create a system that comprehensively manages household cleaning and security, providing users with quick and appropriate information and solutions.

[0275] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0276] The present invention is a cleaning assistance system in a home incorporating an emotion engine for recognizing the user's emotion, which enhances the efficiency of cleaning and provides more personalized support to the user. This system operates with the cooperation of three entities: a server, a terminal, and the user.

[0277] Acquisition and analysis of camera images

[0278] Server

[0279] The server periodically receives video data from a plurality of cameras installed in the home. The video data is received in a stream format and can be processed in real time.

[0280] Preprocess the received video data. The preprocessing includes noise removal, resolution conversion, and color space conversion.

[0281] Input the preprocessed video frames into an AI model to detect dirt and pests. The AI model uses a deep learning algorithm (e.g., convolutional neural network).

[0282] Record the detection results and confidence scores in the database. The record also includes a timestamp and the position information of the camera.

[0283] Generation and transmission of notifications

[0284] Server

[0285] Periodically check the latest detection results from the database. If the confidence score exceeds a certain threshold, it is determined that the location requires cleaning or attention.

[0286] Generate an appropriate notification message and send it to the user terminal. For example, create a notification with the content "The drain in the kitchen is dirty. It's time to clean it."

[0287] Terminal

[0288] The user terminal receives the notification from the server and displays it to the user in real time. This allows the user to respond immediately.

[0289] Support by chatbot

[0290] User

[0291] The user uses the terminal to input questions to the system about cleaning methods and appropriate tools. For example, input a question like "How do I clean the drain?"

[0292] Terminal

[0293] The user's question is sent to the server in real time.

[0294] Server

[0295] Analyze the question using a natural language processing model and generate an appropriate answer. For example, generate specific cleaning procedures like "To clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with a disinfectant."

[0296] Send the generated answer to the user terminal and display it on the chatbot interface.

[0297] Integration of an emotion engine

[0298] Server

[0299] The server incorporates an emotion engine and analyzes the user's voice data and video data. For example, it uses voice data from smartphones or smart speakers, or camera video data to determine the user's emotional state in real time.

[0300] Based on the emotional state determined by the emotion engine (e.g., stress, joy, surprise), the content and expression of the notification message are adjusted. For example, when the user is in a stressed state, the notification message is made more friendly and includes words of encouragement.

[0301] Specific examples

[0302] For example, the server detects an emotion such as "fatigue" from the tone of the user's voice or expression. In this case, it generates content including a kind suggestion like "You seem tired today. Shall we do the drain cleaning a little later?"

[0303] When the user asks a question like "How do I clean the drain?", if the emotion engine detects "interest", it provides specific steps with an encouraging message like "Cleaning is easy! First, let's remove the drain cover."

[0304] In this way, according to the present invention, personalized support according to the user's emotional state is provided, so the cleaning efficiency is improved and the user experience becomes more comfortable.

[0305] The following explains the processing flow.

[0306] Acquisition and analysis of camera video

[0307] Step 1: Acquisition of video

[0308] Server

[0309] The server periodically receives video data from multiple cameras installed within the home. Each camera is located in a designated area (e.g., kitchen, living room, bathroom) and transmits the video data as a real-time stream.

[0310] Step 2: Video Preprocessing

[0311] server

[0312] The received video data is preprocessed. This preprocessing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[0313] Step 3: Detection of dirt and pests

[0314] server

[0315] Pre-processed video data is input into an AI model. The AI ​​model uses a trained convolutional neural network (CNN) to detect dirt and pests in the video. The detection results are output along with a confidence score.

[0316] Step 4: Saving the detection results

[0317] server

[0318] The detection results and confidence score are recorded in a database. The record includes timestamps of the video frames, camera location information, and information identifying the detected object.

[0319] Generating and sending notifications

[0320] Step 1: Check the latest detection results.

[0321] server

[0322] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[0323] Step 2: Generate notification message

[0324] server

[0325] Based on the location where dirt or pests are detected, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." can be created.

[0326] Step 3: Sending a notification message

[0327] server

[0328] The generated notification message is sent to the user's device. The notification is configured to be sent in real time.

[0329] Step 4: Displaying notification messages

[0330] terminal

[0331] The user's terminal receives a notification from the server and displays it to the user immediately. This allows the user to respond instantly.

[0332] Chatbot support

[0333] Step 1: Enter your question

[0334] User

[0335] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[0336] Step 2: Submit your question

[0337] terminal

[0338] The entered questions are sent to the server in real time.

[0339] Step 3: Analyzing the Question

[0340] server

[0341] The received question is analyzed using a natural language processing model. The intent of the question is identified, and an appropriate answer is generated.

[0342] Step 4: Generating the answer

[0343] server

[0344] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0345] Step 5: Submit your response

[0346] server

[0347] The generated response is sent to the user's device.

[0348] Step 6: Display the answer

[0349] terminal

[0350] The received response is displayed in the chatbot interface. The user can then use this information to perform specific cleaning steps.

[0351] Embedding an emotion engine

[0352] Step 1: Obtain user sentiment data

[0353] server

[0354] The server acquires the user's voice and video data. This includes data from smartphones, smart speakers, or cameras.

[0355] Step 2: Sentiment Analysis

[0356] server

[0357] The emotion engine analyzes acquired audio and video data to determine the user's emotional state in real time. Emotional states range widely, including stress, joy, and surprise.

[0358] Step 3: Utilizing emotional data

[0359] server

[0360] The emotion engine determines the user's emotional state and adjusts the content and wording of notification messages and responses accordingly. For example, if a user is stressed, the notification message might be made more friendly and include words of encouragement.

[0361] Specific example

[0362] server

[0363] The system detects "fatigue" from the user's tone of voice and facial expression. In this case, it generates a gentle notification message such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[0364] User

[0365] If a user asks "How do I clean the drain?" and the emotion engine detects "interest," the server will provide specific instructions along with an encouraging message such as "It's easy to clean! First, let's remove the drain cover."

[0366] This processing flow allows the present invention to provide personalized support tailored to the user's emotional state, thereby increasing cleaning efficiency and improving the user experience.

[0367] (Example 2)

[0368] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0369] Traditional home cleaning systems have limited automation and have failed to completely alleviate the user's burden. Furthermore, existing systems did not consider the user's emotional state and only offered a single notification method, resulting in a suboptimal user experience. This could potentially increase user stress.

[0370] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0371] In this invention, the server includes video acquisition means, video preprocessing means, AI analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, and emotion analysis means. This enables everything from automatic detection of dirt and pests to personalized notifications and responses tailored to the user's emotional state.

[0372] "Video acquisition means" refers to a means of receiving video data in real time from a video device installed in a home.

[0373] "Video preprocessing means" refers to means for removing noise from received video data, converting the resolution, and converting the color space.

[0374] "AI analysis means" refers to a method that analyzes pre-processed video data and uses a generative AI model to detect specific objects such as dirt or pests.

[0375] The "analysis result recording means" is a means for recording the detection results and confidence scores obtained by the AI ​​analysis means in a database.

[0376] A "notification generation means" is a means for generating notification messages for the user based on the analysis results.

[0377] "Notification transmission means" refers to a means for sending the generated notification message to the user's terminal.

[0378] "User interface means" refers to means that provide an interface for users to access a system and receive notifications and responses.

[0379] A "chatbot" is a means of automatically generating appropriate answers to user questions and providing those answers.

[0380] "Emotion analysis means" refers to a method for analyzing a user's voice data and video data to determine the user's emotional state in real time.

[0381] This invention is a home cleaning assistant system that incorporates an emotion engine that recognizes the user's emotions. This system operates through the cooperation of three entities: a server, a terminal, and the user.

[0382] The server receives video data in real time from multiple video devices (e.g., cameras) installed in the home. The video data is acquired in stream format, and then noise reduction, resolution conversion, and color space conversion are performed by video preprocessing. This utilizes technologies such as Gaussian filters and the OpenCV library.

[0383] The pre-processed video data is input into an AI analysis system to detect dirt and pests. Here, a convolutional neural network (CNN)-based generative AI model is used to identify dirt and pests. Implementation using the TensorFlow library is preferable.

[0384] Detection results and confidence scores are recorded in a database by an analysis result recording device. The record includes timestamps and location information of the video device, and utilizes a MySQL database. The server periodically checks the database, and if the confidence score of the latest detection result exceeds a certain threshold, an appropriate notification message is created by a notification generation device.

[0385] The notification generation mechanism creates a notification with content such as, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's device via the notification sending mechanism. In this case, Firebase Cloud Messaging (FCM) is suitable. The device receives this notification in real time and displays it to the user.

[0386] Users can access the system through a terminal and input questions about cleaning methods and appropriate tools. Questions such as "How do I clean the drain?" are sent to the server in real time via the user interface. The server analyzes these questions using a natural language processing model (e.g., BERT) and generates appropriate answers. For example, it might provide specific cleaning instructions such as, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0387] The system incorporates emotion analysis capabilities, analyzing the user's voice and video data to determine their emotional state in real time. It uses the Google Cloud Speech-to-Text API for voice analysis and the Emotion API for emotion determination. The system adjusts the content and wording of notification messages according to the user's emotional state. For example, if the user is fatigued, it might generate a message including a helpful suggestion such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[0388] As a concrete example, consider a scenario where the server generates a notification message saying, "You seem tired today. Shall we postpone cleaning the drain for a while?" An example of the prompt text is as follows:

[0389] Example of a prompt:

[0390] "We are informing you of the current situation: The camera has detected dirt in the kitchen drain. However, the user is fatigued, so please include a gentle suggestion in the notification message."

[0391] Thus, the cleaning assistant system of the present invention provides personalized support according to the user's emotional state, improving cleaning efficiency and making the user experience more comfortable.

[0392] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0393] Step 1: Acquire camera footage

[0394] server

[0395] The server receives video data in real time from multiple cameras installed within the home. The cameras transmit data in a 30 frames per second (FPS) stream format. The received video data is encoded in H.264 format. The input is video data from each camera, and the output is video data in stream format.

[0396] Specific operation: Video data transmitted from each camera is received in real time by a dedicated receiving module.

[0397] Step 2: Preprocessing of video data

[0398] server

[0399] The server preprocesses the received video data by denoising, converting the resolution, and converting the color space. A Gaussian filter is used for denoising, and the resolution conversion involves downsampling from 1080p to 720p. Color space conversion is performed from RGB to grayscale. The input is video data in stream format, and the output is preprocessed video data.

[0400] Specific operation: Using the OpenCV library, a series of preprocessing algorithms are applied to the received video data.

[0401] Step 3: AI-powered detection of dirt and pests

[0402] server

[0403] The server inputs pre-processed video data into a convolutional neural network (CNN)-based generative AI model to detect dirt and pests. During this process, features are extracted from each frame and input into the AI ​​model. The input is pre-processed video data, and the output is the detection result and confidence score.

[0404] Specific operation: Using the TensorFlow library, a pre-trained AI model is applied to detect the presence or absence of dirt or pests within 0.1 seconds for each frame.

[0405] Step 4: Record the detection results

[0406] server

[0407] The server records detection results and confidence scores from the AI ​​model in a database. Recorded items include timestamps and camera location information. The input is the detection results and confidence scores, and the output is the recorded data stored in the database.

[0408] Specific operation: Connect to the MySQL database and insert each detection result into a dedicated table.

[0409] Step 5: Generate Notifications

[0410] server

[0411] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. It then generates an appropriate notification message. The input is the detection results from the database, and the output is the notification message.

[0412] Specific action: Generate a notification with a message like, "The kitchen drain is dirty. It's time to clean it."

[0413] Step 6: Send the notification

[0414] server

[0415] The server sends the generated notification message to the user's device. Firebase Cloud Messaging (FCM) is used for push notifications. The input is the notification message, and the output is the notification sent to the user's device.

[0416] Specific operation: Use the FCM API to send notifications to the user's smartphone or tablet.

[0417] Step 7: Responding to user inquiries

[0418] User

[0419] Users access the system through a terminal and input questions about cleaning methods and appropriate tools. The input is the user's question, and the output is the question data.

[0420] terminal

[0421] The terminal sends user questions to the server in real time. The input is the user's question, and the output is the question data sent to the server.

[0422] server

[0423] The server analyzes the user's question using a natural language processing model (e.g., BERT) and generates an appropriate answer. The input is the question data, and the output is the answer message.

[0424] Specific operation: The Transformers library is used for natural language processing, and an algorithm is applied to generate appropriate answers to questions.

[0425] Step 8: Adjusting Sentiment Analysis and Notifications

[0426] server

[0427] The server uses emotion analysis tools to analyze the user's voice and video data and determine the user's emotional state in real time. Based on the emotional state, it adjusts the content and wording of notification messages. The input is the user's voice and video data, and the output is the adjusted notification message.

[0428] Specific operation: The Google Cloud Speech-to-Text API is used for speech analysis, and the Emotion API is used for emotion detection, dynamically changing the wording of notification messages.

[0429] In this way, the entire system works together to provide personalized support to the user, improving cleaning efficiency and making the user experience more comfortable.

[0430] (Application Example 2)

[0431] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0432] In existing brick-and-mortar stores, providing personalized service tailored to customers' emotional states is difficult, often resulting in a uniform service quality and consequently lower customer satisfaction. Furthermore, there is no system in place to recognize customers' emotions in real time from their facial expressions and voices and respond appropriately based on that information. As a result, the quality of the customer experience remains stagnant.

[0433] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means, personalized notification means, and video acquisition means. This makes it possible to recognize the customer's emotional state in real time and send a personalized notification message to store staff accordingly.

[0434] "Emotion recognition means" refers to a function that analyzes audio and video data to determine the user's emotional state in real time.

[0435] A "personalized notification method" is a function that adjusts the content and expression of notification messages based on the user's emotional state as determined by the emotion recognition method.

[0436] "Video acquisition means" refers to a camera device and a function that periodically receives video data in a stream format.

[0437] The "video analysis means" refers to a function that preprocesses received video data and uses an AI model to detect dirt and pests.

[0438] The "analysis result recording means" is a function that records the results of video analysis and stores them in a database that includes timestamps and location information.

[0439] The "notification generation means" is a function that generates an appropriate notification message based on the analysis results.

[0440] "Notification transmission means" refers to a function that sends the generated notification message to the user interface means.

[0441] A "user interface means" is an interface for exchanging information between a user and a system.

[0442] A "chatbot" is a function that receives questions from users, generates appropriate answers, and sends responses.

[0443] The present invention is a system that recognizes customer emotions in a physical store and provides personalized responses based on those emotions. This system includes emotion recognition means, personalized notification means, video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, and chatbot means.

[0444] Hardware configuration

[0445] Smart glasses: Store staff wear them, and a camera device collects video data of customers.

[0446] Server: Processes video and audio data and generates notification messages based on the analysis results.

[0447] Display interface: Integrated into smart glasses, it displays notification messages.

[0448] Software Configuration

[0449] OpenCV: Used for preprocessing and analyzing video data.

[0450] DeepFace: An algorithm for facial recognition and emotion analysis.

[0451] Natural language processing model: Analyzes user questions and generates appropriate answers.

[0452] Database: Used to record analysis results.

[0453] Data processing and data calculation

[0454] 1. Video acquisition method: The camera device of the smart glasses collects video data in real time and transmits it to the server.

[0455] 2. Video analysis means: The server preprocesses the received video data (noise reduction, resolution conversion, etc.) and uses DeepFace to analyze the customer's emotions.

[0456] 3. Means for recording analysis results: The results of face recognition and emotion analysis are recorded in a database along with timestamps and location information.

[0457] 4. Emotion Recognition Method: Determine the customer's emotional state (e.g., anger, joy, surprise, etc.) in real time.

[0458] 5. Personalized notification means: Based on the customer's emotions determined by the emotion recognition means, an appropriate notification message is generated.

[0459] 6. Notification generation means: Creates a personalized notification message and displays it on the smart glasses' display via a notification sending means.

[0460] Adding specific examples

[0461] For example, if the server detects a customer's anger from the video data it receives, the personalized notification system will generate a message such as, "This customer is irritated. Please respond calmly," and display it on the smart glasses' screen. This allows store staff to adjust their customer service to match the customer's emotional state.

[0462] Examples of prompts to input into a generative AI model:

[0463] Prompt: Video frame showing the customer's face

[0464] Output: Analyzes customer emotions in real time and generates emotion-based response notification messages.

[0465] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0466] Step 1:

[0467] Acquisition of video data

[0468] A camera device built into the smart glasses collects video data of customers in the store in real time. The video data is transmitted from the smart glasses to the server in stream format. In this step, the input is images of customers in the store, and the output is the video data distributed to the server.

[0469] Step 2:

[0470] Preprocessing of video data

[0471] The server performs preprocessing on the received video data, including noise reduction, resolution conversion, and color space conversion. This process is carried out using OpenCV. The input is video data in stream format, and the output is preprocessed video frames.

[0472] Step 3:

[0473] emotion recognition

[0474] The server inputs pre-processed video frames into DeepFace for face recognition and emotion analysis. The input is pre-processed video frames, and the output is the customer's emotion (e.g., anger, joy, surprise) and its confidence score.

[0475] Step 4:

[0476] Record of results

[0477] The analysis results are stored in a database. The records include analyzed emotions, timestamps, and camera location information. The input is the emotion recognition result and its associated information, and the output is a database entry.

[0478] Step 5:

[0479] Generating notification messages

[0480] Based on the customer's emotions as determined by the emotion recognition system, an appropriate notification message is generated. The input is data from emotion recognition, and the output is a notification message. For example, if the customer is showing anger, the message "This customer is irritated. Please respond calmly." will be generated.

[0481] Step 6:

[0482] Sending notification messages

[0483] The generated notification message is sent to the smart glasses' display via the notification sending device. The input is the notification message, and the output is the text displayed on the smart glasses' display.

[0484] Step 7:

[0485] Customer support

[0486] Upon receiving the notification, store staff will respond to the customer according to the message displayed on the smart glasses' screen. The input is the notification message displayed on the smart glasses, and the output is personalized customer service.

[0487] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0488] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0489] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0490] [Second Embodiment]

[0491] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0492] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0493] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0494] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0495] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0496] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0497] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0498] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0499] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0500] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0501] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0502] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0503] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[0504] Camera image acquisition and analysis

[0505] server

[0506] The server periodically receives video data from camera devices installed within the home. Each camera is located in a different place within the home and transmits video in a stream format at regular intervals.

[0507] The received video data is first pre-processed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion.

[0508] The pre-processed video frames are sent to an AI model to detect dirt and pests. The AI ​​model uses deep learning algorithms such as pre-trained convolutional neural networks (CNNs).

[0509] Detection results are recorded in the database along with a confidence score. The data also includes location information and timestamps of the detected area.

[0510] Generating and sending notifications

[0511] server

[0512] The system periodically checks the latest detection results from the database, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention.

[0513] Based on the areas that need cleaning or where pests have been detected, the system generates appropriate notification messages. For example, it might generate a message like, "The kitchen drain is dirty. It's time to clean it."

[0514] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[0515] terminal

[0516] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[0517] Chatbot support

[0518] User

[0519] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[0520] terminal

[0521] Questions from users are sent to the server in real time.

[0522] server

[0523] The server analyzes the received question using a natural language processing model and generates an appropriate answer. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[0524] The generated response is sent back to the user's device and displayed in the chatbot interface.

[0525] Specific example

[0526] For example, suppose a camera installed in the kitchen drain sends video data at 7:00 AM one day. The server receives this video data and analyzes it using an AI model. As a result of the analysis, if dirt is detected in the drain, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server will respond with specific instructions. This allows the user to clean effectively.

[0527] Thus, the present invention makes it possible to clean the home efficiently and effectively, and to maintain a clean living environment on a daily basis.

[0528] The following describes the processing flow.

[0529] Camera image acquisition and analysis

[0530] Step 1: Acquire video

[0531] server

[0532] The server periodically receives video data from each camera installed within the home. Since the video data is transmitted in a stream format, real-time analysis is possible.

[0533] Step 2: Pretreatment

[0534] server

[0535] The received video data is first pre-processed. This pre-processing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[0536] Step 3: Video Analysis

[0537] server

[0538] Pre-processed video frames are input into the AI ​​model. The AI ​​model uses a pre-trained convolutional neural network (CNN) to detect dirt and pests in the images. The detection results are output along with a confidence score.

[0539] Step 4: Saving the analysis results

[0540] server

[0541] The detection results and confidence scores are recorded in a database. The record also includes timestamps of the video frames and camera location information. This stores information that will be useful for later reference.

[0542] Generating and sending notifications

[0543] Step 1: Extract the necessary notifications

[0544] server

[0545] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[0546] Step 2: Generate notification message

[0547] server

[0548] Based on the extracted information, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." is generated.

[0549] Step 3: Sending a notification message

[0550] server

[0551] The generated notification message is sent to the user's device. The notification is sent in real time so that the user can understand it immediately.

[0552] Step 4: Displaying notification messages

[0553] terminal

[0554] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to respond quickly.

[0555] Chatbot support

[0556] Step 1: Enter your question

[0557] User

[0558] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[0559] Step 2: Submit your question

[0560] terminal

[0561] The entered questions are sent to the server in real time.

[0562] Step 3: Analyzing the Question

[0563] server

[0564] The received question is analyzed using a natural language processing model. This analysis helps understand the user's intent and generates an appropriate answer.

[0565] Step 4: Generating the answer

[0566] server

[0567] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0568] Step 5: Submit your response

[0569] server

[0570] The generated response is sent to the user's device.

[0571] Step 6: Display the answer

[0572] terminal

[0573] The received response is displayed in the chatbot interface. The user can then use this displayed information to perform specific cleaning steps.

[0574] This series of processing steps allows users to clean their homes efficiently and effectively. Furthermore, this process helps maintain cleanliness in the household while reducing the user's burden.

[0575] (Example 1)

[0576] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0577] Household cleaning is time-consuming and laborious, and early detection of dirt and pests is particularly difficult. Furthermore, a lack of information on cleaning methods and tools often prevents proper treatment. This contributes to the problem of difficulty in maintaining cleanliness in the home.

[0578] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0579] In this invention, the server includes means for acquiring video, means for pre-processing video data, means for analyzing using an AI model, means for recording analysis results, means for generating notification messages, means for sending notification messages, user interface means, and chatbot means. This makes it possible to detect and notify about dirt and pests in the home at an early stage and provide information to the user quickly.

[0580] "Video acquisition means" refers to a device that periodically receives video data from a recording device installed in a home.

[0581] "Means for pre-processing video data" refers to a device or function that performs processing such as noise reduction, resolution conversion, and color space conversion on received video data.

[0582] "Methods of analysis using AI models" refers to devices or functions that analyze pre-processed video data using artificial intelligence technologies such as convolutional neural networks (CNNs) to detect dirt and pests.

[0583] "Means for recording analysis results" refers to a device or function that stores the detection results of dirt and pests obtained by the AI ​​model in a database along with a confidence score.

[0584] "Means for generating notification messages" refers to a device or function that creates appropriate notification messages for areas requiring cleaning or areas where pests have been detected, based on the analysis results.

[0585] "Means for sending notification messages" refers to a device or function that sends generated notification messages to the user's terminal in real time.

[0586] "User interface means" refers to a device or application that provides an interface for a user to communicate with a system.

[0587] A "chatbot" refers to an artificial intelligence-based conversational agent that automatically generates answers to questions from users about cleaning methods and tools to be used.

[0588] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[0589] Camera image acquisition and analysis

[0590] server

[0591] The server periodically receives video data from recording devices installed within the home. These devices can be standard webcams. Each camera is located in a different part of the home (e.g., kitchen, living room, entryway) and transmits video in a stream format at regular intervals.

[0592] The received video data is first pre-processed. This pre-processing includes noise reduction (using Gaussian filters, etc.), resolution conversion (converting to HD resolution), and color space conversion (converting from RGB to grayscale). Pre-processing is performed using image processing libraries such as OpenCV.

[0593] Preprocessed video frames are sent to an AI model using a convolutional neural network (CNN). This model is built with deep learning libraries such as TensorFlow and PyTorch. The AI ​​model detects dirt and pests in the video and generates a confidence score. The detection results are recorded in a database along with the confidence score. The recorded data includes location information and timestamps of the detected areas.

[0594] Generating and sending notifications

[0595] server

[0596] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. Based on the area that needs cleaning or where pests have been detected, it generates an appropriate notification message. For example, it might generate a message such as, "The kitchen drain is dirty. It's time to clean it."

[0597] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[0598] terminal

[0599] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[0600] Chatbot support

[0601] User

[0602] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[0603] terminal

[0604] Questions from users are sent to the server in real time.

[0605] server

[0606] The server analyzes the received question using a natural language processing model and generates an appropriate answer. This process uses generative AI models such as GPT-3. The generated answer is then sent back to the user's terminal and displayed in the chatbot interface.

[0607] Specific examples of operation

[0608] For example, suppose a camera installed in the kitchen drain transmits video data at 7:00 AM one day. The server receives this video data and performs noise reduction and grayscale conversion using a Gaussian filter. Then, it analyzes it with an AI model and detects dirt in the drain with a 95% confidence level. The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in its database and determines that cleaning is necessary. It generates a notification message saying, "Your kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server replies with specific steps: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." This allows the user to clean effectively.

[0609] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0610] Step 1:

[0611] server

[0612] Video acquisition

[0613] Input: Video data transmitted in stream format from a home recording device.

[0614] Output: Video data received by the server

[0615] Example of operation: A camera installed in the kitchen sends video data to a server every 5 minutes. The server receives this data.

[0616] Step 2:

[0617] server

[0618] Preprocessing of video data

[0619] Input: Received video data

[0620] Output: Pre-processed video data (data after noise reduction, resolution conversion, and color space conversion)

[0621] Specific example of operation: The server removes noise from the received video data using a Gaussian filter, converts the resolution to HD, and converts it from RGB to grayscale.

[0622] Step 3:

[0623] server

[0624] AI-based video analysis

[0625] Input: Preprocessed video data

[0626] Output: Detection results for dirt and pests and their confidence scores.

[0627] Specific example of operation: The server inputs pre-processed video into an AI model (for example, a model using CNN) and detects dirt in the kitchen drain with a 95% confidence level.

[0628] Step 4:

[0629] server

[0630] Record of analysis results

[0631] Input: AI model detection results and confidence score

[0632] Output: Recorded detection results (including location information and timestamp)

[0633] Example of operation: The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in the database.

[0634] Step 5:

[0635] server

[0636] Generating notification messages

[0637] Input: Recorded detection results

[0638] Output: Notification message (reminds for cleaning or other actions)

[0639] Example of operation: Based on the recorded data, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it."

[0640] Step 6:

[0641] server

[0642] Sending notification messages

[0643] Input: Generated notification message

[0644] Output: Notification message sent to the user's terminal

[0645] Example of operation: The server sends the generated notification message as a push notification to the user's smartphone.

[0646] Step 7:

[0647] terminal

[0648] User notification display

[0649] Input: Notification message sent from the server

[0650] Output: Notification message displayed to the user

[0651] Example of operation: A notification appears on the user's smartphone saying, "The kitchen drain is dirty. It's time to clean it."

[0652] Step 8:

[0653] User

[0654] Questions for the chatbot

[0655] Input: User questions (e.g., "How do I clean the drain?")

[0656] Output: Question sent to the server

[0657] Example of operation: The user uses a smartphone app to input the question, "How do I clean a drain?"

[0658] Step 9:

[0659] server

[0660] Question analysis and answer generation

[0661] Input: Question from a user

[0662] Output: Generated answer

[0663] Example of operation: The server analyzes the received question using a generative AI model such as GPT-3 and generates the answer, "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[0664] Step 10:

[0665] server

[0666] Submit your response

[0667] Input: Generated answer

[0668] Output: Response sent to the user's terminal

[0669] Example of operation: The generated answer is sent to the user's smartphone.

[0670] Step 11:

[0671] terminal

[0672] Displaying responses to users

[0673] Input: Response sent from the server

[0674] Output: Answer displayed to the user

[0675] Example of operation: The user's smartphone displays the following message: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Clean with disinfectant."

[0676] (Application Example 1)

[0677] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0678] Traditional home cleaning systems were useful for detecting dirt and pests, but they lacked security features to detect suspicious movements or behavior. Therefore, there is a need to effectively address home security. Furthermore, they lacked a means to immediately suggest safety measures to users. Thus, there is a demand for systems that improve not only home cleaning but also overall security.

[0679] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0680] In this invention, the server includes video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, suspicious behavior detection means, and safety measure suggestion means. This makes it possible to detect not only dirt and pests in the home but also suspicious behavior and provide the user with immediate notifications and safety measures.

[0681] "Video acquisition means" refers to a function for periodically receiving video data from a camera device installed in the home.

[0682] "Video analysis means" refers to a function that preprocesses received video data and uses an AI model to analyze dirt, pests, suspicious behavior, etc.

[0683] The "analysis result recording means" is a function for recording the analysis results of detected dirt, pests, and suspicious behavior in a database.

[0684] The "notification generation means" is a function that automatically generates notification messages for the user based on the analysis results.

[0685] A "notification transmission method" is a function for sending generated notification messages to the user's terminal in real time.

[0686] A "user interface means" is a function that provides an interface for a user to interact with the system.

[0687] A "chatbot" is a function that provides an automated response feature to answer questions from users.

[0688] "Suspicious behavior detection means" refers to a function that complements video analysis means and detects suspicious movements and actions.

[0689] The "safety measures suggestion mechanism" is a function that suggests appropriate safety measures to the user when suspicious behavior is detected.

[0690] This invention is a system for efficiently and effectively cleaning and securing a home. This system operates through the cooperation of three entities: a server, a terminal, and a user, and utilizes in-home camera equipment to detect dirt, pests, and suspicious activity.

[0691] Camera image acquisition and analysis

[0692] server

[0693] Acquiring camera footage:

[0694] The server periodically receives video data from camera devices installed within the home. This data is received in stream format using RTSP (Real Time Streaming Protocol). By receiving video data from cameras installed in different locations within the home, the entire house is covered.

[0695] Pre-processing:

[0696] The received video data is first preprocessed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion using the OpenCV library.

[0697] AI analysis:

[0698] The pre-processed video frames are analyzed using deep learning algorithms such as convolutional neural networks (CNNs) based on TensorFlow or PyTorch. This analysis detects dirt, pests, and suspicious behavior.

[0699] Record of results:

[0700] Detection results are recorded in a database (PostgreSQL or MongoDB) along with a confidence score. The data also includes location information and timestamps of the detected areas.

[0701] Generating and sending notifications

[0702] server

[0703] Notification generation:

[0704] The latest detection results are periodically checked from the database, and if the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention. The same determination is made if suspicious behavior is detected.

[0705] Send notification:

[0706] The generated notification message is sent to the user's device. Real-time push notifications are sent using Firebase Cloud Messaging (FCM).

[0707] User terminal

[0708] Receive and display:

[0709] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to know in real time which areas need cleaning and what suspicious activity is happening in the home.

[0710] Chatbot support

[0711] User

[0712] Enter your question:

[0713] Users can use their devices to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?"

[0714] server

[0715] Response generation:

[0716] The system analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, follow these steps: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0717] Send response:

[0718] The generated response is sent back to the user's device and displayed in the chatbot interface.

[0719] Specific example

[0720] For example, if a camera installed in the kitchen detects dirt, the server generates a notification saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. Furthermore, if suspicious activity is detected in the kitchen, a notification saying, "There is suspicious activity in the kitchen. Please check," is also sent. If the user asks the chatbot, "What should I do if there is a suspicious person in the kitchen?", they will receive the answer, "If possible, move to a safe place and call the police. We also recommend activating your security system." This allows the user to take a quick and appropriate action.

[0721] Example of a prompt

[0722] What should I do if a suspicious person is detected by a home security camera?

[0723] Thus, this system integrates and manages household cleaning and security, providing users with immediately useful information and solutions.

[0724] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0725] Step 1: Receiving and pre-processing camera footage

[0726] The server periodically receives video data in stream format using RTSP from a camera device installed in the home. The input is the video stream from the camera, and the output is pre-processed video data that has undergone noise reduction, resolution conversion, and color space conversion.

[0727] Step 2: Detection using AI analysis

[0728] The server analyzes pre-processed video data using a convolutional neural network (CNN) based on TensorFlow or PyTorch. Here, it detects dirt, pests, and suspicious behavior. The input is pre-processed video data, and the output is the detection results for dirt, pests, and suspicious behavior.

[0729] Step 3: Recording the detection results

[0730] The server records the detection results in a database (PostgreSQL or MongoDB). This step also records the location information and timestamps of the detected areas. The input is the detection result from the AI ​​analysis, and the output is the detection result recorded in the database.

[0731] Step 4: Generate Notifications

[0732] The server periodically checks the database for the latest detection results and generates an appropriate notification message if the confidence score exceeds a certain threshold. A similar procedure is used to generate a notification message if suspicious activity is detected. The input is the detection results from the database, and the output is the generated notification message.

[0733] Step 5: Sending Notifications

[0734] The server sends the generated notification message to the user's device in real time using Firebase Cloud Messaging (FCM). The input is the generated notification message, and the output is the notification sent to the user's device.

[0735] Step 6: Receiving and displaying notifications

[0736] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to see in real time where cleaning is needed and any suspicious activity in the home. The input is the notification from the server, and the output is the notification displayed on the terminal.

[0737] Step 7: Enter your question for the chatbot.

[0738] The user uses a terminal to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?" The input is the user's question, and the output is the question data sent to the server.

[0739] Step 8: Question analysis and answer generation

[0740] The server analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." The input is the user's question, and the output is the generated answer.

[0741] Step 9: Submit and view your response

[0742] The server sends the generated response back to the user's terminal and displays it on the chatbot interface. The input is the generated response, and the output is the response displayed on the user's terminal.

[0743] In this way, each step works in conjunction to create a system that comprehensively manages household cleaning and security, providing users with quick and appropriate information and solutions.

[0744] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0745] This invention is a home cleaning assistant system that incorporates an emotion engine to recognize user emotions, thereby improving cleaning efficiency and providing more personalized support to the user. This system operates through the cooperation of three entities: a server, a terminal, and the user.

[0746] Camera image acquisition and analysis

[0747] server

[0748] The server periodically receives video data from multiple cameras installed within the home. The video data is received in stream format, allowing for real-time processing.

[0749] The received video data is preprocessed. Preprocessing includes noise reduction, resolution conversion, and color space conversion.

[0750] The pre-processed video frames are input into an AI model to detect dirt and pests. The AI ​​model uses a deep learning algorithm (e.g., a convolutional neural network).

[0751] The detection results and confidence scores are recorded in a database. The record also includes timestamps and camera location information.

[0752] Generating and sending notifications

[0753] server

[0754] The system periodically checks the latest detection results from the database, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention.

[0755] Generate an appropriate notification message and send it to the user's device. For example, create a notification that says, "The kitchen drain is dirty. It's time to clean it."

[0756] terminal

[0757] The user's terminal receives notifications from the server and displays them to the user in real time. This allows the user to respond immediately.

[0758] Chatbot support

[0759] User

[0760] The user uses a terminal to input questions into the system about cleaning methods and appropriate tools. For example, they might input a question like, "How do I clean the drain?"

[0761] terminal

[0762] Questions from users are sent to the server in real time.

[0763] server

[0764] The system analyzes questions using a natural language processing model and generates appropriate answers. For example, it can generate specific cleaning instructions such as, "To clean the drain, follow these steps: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0765] The generated response is sent to the user's device and displayed in the chatbot interface.

[0766] Embedding an emotion engine

[0767] server

[0768] The server incorporates an emotion engine that analyzes the user's voice and video data. For example, it uses voice data from a smartphone or smart speaker, or camera video data, to determine the user's emotional state in real time.

[0769] The content and wording of notification messages are adjusted based on the emotional state determined by the emotion engine (e.g., stress, joy, surprise). For example, if the user is stressed, the notification message will be made more friendly and include words of encouragement.

[0770] Specific example

[0771] For example, the server can detect emotions such as "fatigue" from the user's tone of voice and facial expressions. In this case, it generates a notification message that includes a helpful suggestion such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[0772] If a user asks, "How do I clean the drain?", and the emotion engine detects "interest," it will provide specific instructions along with an encouraging message such as, "It's easy to clean! First, let's remove the drain cover."

[0773] Thus, the present invention provides personalized support tailored to the user's emotional state, improving cleaning efficiency and making the user experience more comfortable.

[0774] The following describes the processing flow.

[0775] Camera image acquisition and analysis

[0776] Step 1: Acquire video

[0777] server

[0778] The server periodically receives video data from multiple cameras installed within the home. Each camera is located in a designated area (e.g., kitchen, living room, bathroom) and transmits the video data as a real-time stream.

[0779] Step 2: Video Preprocessing

[0780] server

[0781] The received video data is preprocessed. This preprocessing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[0782] Step 3: Detection of dirt and pests

[0783] server

[0784] Pre-processed video data is input into an AI model. The AI ​​model uses a trained convolutional neural network (CNN) to detect dirt and pests in the video. The detection results are output along with a confidence score.

[0785] Step 4: Saving the detection results

[0786] server

[0787] The detection results and confidence score are recorded in a database. The record includes timestamps of the video frames, camera location information, and information identifying the detected object.

[0788] Generating and sending notifications

[0789] Step 1: Check the latest detection results.

[0790] server

[0791] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[0792] Step 2: Generate notification message

[0793] server

[0794] Based on the location where dirt or pests are detected, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." can be created.

[0795] Step 3: Sending a notification message

[0796] server

[0797] The generated notification message is sent to the user's device. The notification is configured to be sent in real time.

[0798] Step 4: Displaying notification messages

[0799] terminal

[0800] The user's terminal receives a notification from the server and displays it to the user immediately. This allows the user to respond instantly.

[0801] Chatbot support

[0802] Step 1: Enter your question

[0803] User

[0804] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[0805] Step 2: Submit your question

[0806] terminal

[0807] The entered questions are sent to the server in real time.

[0808] Step 3: Analyzing the Question

[0809] server

[0810] The received question is analyzed using a natural language processing model. The intent of the question is identified, and an appropriate answer is generated.

[0811] Step 4: Generating the answer

[0812] server

[0813] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0814] Step 5: Submit your response

[0815] server

[0816] The generated response is sent to the user's device.

[0817] Step 6: Display the answer

[0818] terminal

[0819] The received response is displayed in the chatbot interface. The user can then use this information to perform specific cleaning steps.

[0820] Embedding an emotion engine

[0821] Step 1: Obtain user sentiment data

[0822] server

[0823] The server acquires the user's voice and video data. This includes data from smartphones, smart speakers, or cameras.

[0824] Step 2: Sentiment Analysis

[0825] server

[0826] The emotion engine analyzes acquired audio and video data to determine the user's emotional state in real time. Emotional states range widely, including stress, joy, and surprise.

[0827] Step 3: Utilizing emotional data

[0828] server

[0829] The emotion engine determines the user's emotional state and adjusts the content and wording of notification messages and responses accordingly. For example, if a user is stressed, the notification message might be made more friendly and include words of encouragement.

[0830] Specific example

[0831] server

[0832] The system detects "fatigue" from the user's tone of voice and facial expression. In this case, it generates a gentle notification message such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[0833] User

[0834] If a user asks "How do I clean the drain?" and the emotion engine detects "interest," the server will provide specific instructions along with an encouraging message such as "It's easy to clean! First, let's remove the drain cover."

[0835] This processing flow allows the present invention to provide personalized support tailored to the user's emotional state, thereby increasing cleaning efficiency and improving the user experience.

[0836] (Example 2)

[0837] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0838] Traditional home cleaning systems have limited automation and have failed to completely alleviate the user's burden. Furthermore, existing systems did not consider the user's emotional state and only offered a single notification method, resulting in a suboptimal user experience. This could potentially increase user stress.

[0839] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0840] In this invention, the server includes video acquisition means, video preprocessing means, AI analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, and emotion analysis means. This enables everything from automatic detection of dirt and pests to personalized notifications and responses tailored to the user's emotional state.

[0841] "Video acquisition means" refers to a means of receiving video data in real time from a video device installed in a home.

[0842] "Video preprocessing means" refers to means for removing noise from received video data, converting the resolution, and converting the color space.

[0843] "AI analysis means" refers to a method that analyzes pre-processed video data and uses a generative AI model to detect specific objects such as dirt or pests.

[0844] The "analysis result recording means" is a means for recording the detection results and confidence scores obtained by the AI ​​analysis means in a database.

[0845] A "notification generation means" is a means for generating notification messages for the user based on the analysis results.

[0846] "Notification transmission means" refers to a means for sending the generated notification message to the user's terminal.

[0847] "User interface means" refers to means that provide an interface for users to access a system and receive notifications and responses.

[0848] A "chatbot" is a means of automatically generating appropriate answers to user questions and providing those answers.

[0849] "Emotion analysis means" refers to a method for analyzing a user's voice data and video data to determine the user's emotional state in real time.

[0850] This invention is a home cleaning assistant system that incorporates an emotion engine that recognizes the user's emotions. This system operates through the cooperation of three entities: a server, a terminal, and the user.

[0851] The server receives video data in real time from multiple video devices (e.g., cameras) installed in the home. The video data is acquired in stream format, and then noise reduction, resolution conversion, and color space conversion are performed by video preprocessing. This utilizes technologies such as Gaussian filters and the OpenCV library.

[0852] The pre-processed video data is input into an AI analysis system to detect dirt and pests. Here, a convolutional neural network (CNN)-based generative AI model is used to identify dirt and pests. Implementation using the TensorFlow library is preferable.

[0853] Detection results and confidence scores are recorded in a database by an analysis result recording device. The record includes timestamps and location information of the video device, and utilizes a MySQL database. The server periodically checks the database, and if the confidence score of the latest detection result exceeds a certain threshold, an appropriate notification message is created by a notification generation device.

[0854] The notification generation mechanism creates a notification with content such as, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's device via the notification sending mechanism. In this case, Firebase Cloud Messaging (FCM) is suitable. The device receives this notification in real time and displays it to the user.

[0855] Users can access the system through a terminal and input questions about cleaning methods and appropriate tools. Questions such as "How do I clean the drain?" are sent to the server in real time via the user interface. The server analyzes these questions using a natural language processing model (e.g., BERT) and generates appropriate answers. For example, it might provide specific cleaning instructions such as, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[0856] The system incorporates emotion analysis capabilities, analyzing the user's voice and video data to determine their emotional state in real time. It uses the Google Cloud Speech-to-Text API for voice analysis and the Emotion API for emotion determination. The system adjusts the content and wording of notification messages according to the user's emotional state. For example, if the user is fatigued, it might generate a message including a helpful suggestion such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[0857] As a concrete example, consider a scenario where the server generates a notification message saying, "You seem tired today. Shall we postpone cleaning the drain for a while?" An example of the prompt text is as follows:

[0858] Example of a prompt:

[0859] "We are informing you of the current situation: The camera has detected dirt in the kitchen drain. However, the user is fatigued, so please include a gentle suggestion in the notification message."

[0860] Thus, the cleaning assistant system of the present invention provides personalized support according to the user's emotional state, improving cleaning efficiency and making the user experience more comfortable.

[0861] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0862] Step 1: Acquire camera footage

[0863] server

[0864] The server receives video data in real time from multiple cameras installed within the home. The cameras transmit data in a 30 frames per second (FPS) stream format. The received video data is encoded in H.264 format. The input is video data from each camera, and the output is video data in stream format.

[0865] Specific operation: Video data transmitted from each camera is received in real time by a dedicated receiving module.

[0866] Step 2: Preprocessing of video data

[0867] server

[0868] The server preprocesses the received video data by denoising, converting the resolution, and converting the color space. A Gaussian filter is used for denoising, and the resolution conversion involves downsampling from 1080p to 720p. Color space conversion is performed from RGB to grayscale. The input is video data in stream format, and the output is preprocessed video data.

[0869] Specific operation: Using the OpenCV library, a series of preprocessing algorithms are applied to the received video data.

[0870] Step 3: AI-powered detection of dirt and pests

[0871] server

[0872] The server inputs pre-processed video data into a convolutional neural network (CNN)-based generative AI model to detect dirt and pests. During this process, features are extracted from each frame and input into the AI ​​model. The input is pre-processed video data, and the output is the detection result and confidence score.

[0873] Specific operation: Using the TensorFlow library, a pre-trained AI model is applied to detect the presence or absence of dirt or pests within 0.1 seconds for each frame.

[0874] Step 4: Record the detection results

[0875] server

[0876] The server records detection results and confidence scores from the AI ​​model in a database. Recorded items include timestamps and camera location information. The input is the detection results and confidence scores, and the output is the recorded data stored in the database.

[0877] Specific operation: Connect to the MySQL database and insert each detection result into a dedicated table.

[0878] Step 5: Generate Notifications

[0879] server

[0880] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. It then generates an appropriate notification message. The input is the detection results from the database, and the output is the notification message.

[0881] Specific action: Generate a notification with a message like, "The kitchen drain is dirty. It's time to clean it."

[0882] Step 6: Send the notification

[0883] server

[0884] The server sends the generated notification message to the user's device. Firebase Cloud Messaging (FCM) is used for push notifications. The input is the notification message, and the output is the notification sent to the user's device.

[0885] Specific operation: Use the FCM API to send notifications to the user's smartphone or tablet.

[0886] Step 7: Responding to user inquiries

[0887] User

[0888] Users access the system through a terminal and input questions about cleaning methods and appropriate tools. The input is the user's question, and the output is the question data.

[0889] terminal

[0890] The terminal sends user questions to the server in real time. The input is the user's question, and the output is the question data sent to the server.

[0891] server

[0892] The server analyzes the user's question using a natural language processing model (e.g., BERT) and generates an appropriate answer. The input is the question data, and the output is the answer message.

[0893] Specific operation: The Transformers library is used for natural language processing, and an algorithm is applied to generate appropriate answers to questions.

[0894] Step 8: Adjusting Sentiment Analysis and Notifications

[0895] server

[0896] The server uses emotion analysis tools to analyze the user's voice and video data and determine the user's emotional state in real time. Based on the emotional state, it adjusts the content and wording of notification messages. The input is the user's voice and video data, and the output is the adjusted notification message.

[0897] Specific operation: The Google Cloud Speech-to-Text API is used for speech analysis, and the Emotion API is used for emotion detection, dynamically changing the wording of notification messages.

[0898] In this way, the entire system works together to provide personalized support to the user, improving cleaning efficiency and making the user experience more comfortable.

[0899] (Application Example 2)

[0900] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0901] In existing brick-and-mortar stores, providing personalized service tailored to customers' emotional states is difficult, often resulting in a uniform service quality and consequently lower customer satisfaction. Furthermore, there is no system in place to recognize customers' emotions in real time from their facial expressions and voices and respond appropriately based on that information. As a result, the quality of the customer experience remains stagnant.

[0902] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means, personalized notification means, and video acquisition means. This makes it possible to recognize the customer's emotional state in real time and send a personalized notification message to store staff accordingly.

[0903] "Emotion recognition means" refers to a function that analyzes audio and video data to determine the user's emotional state in real time.

[0904] A "personalized notification method" is a function that adjusts the content and expression of notification messages based on the user's emotional state as determined by the emotion recognition method.

[0905] "Video acquisition means" refers to a camera device and a function that periodically receives video data in a stream format.

[0906] The "video analysis means" refers to a function that preprocesses received video data and uses an AI model to detect dirt and pests.

[0907] The "analysis result recording means" is a function that records the results of video analysis and stores them in a database that includes timestamps and location information.

[0908] The "notification generation means" is a function that generates an appropriate notification message based on the analysis results.

[0909] "Notification transmission means" refers to a function that sends the generated notification message to the user interface means.

[0910] A "user interface means" is an interface for exchanging information between a user and a system.

[0911] A "chatbot" is a function that receives questions from users, generates appropriate answers, and sends responses.

[0912] The present invention is a system that recognizes customer emotions in a physical store and provides personalized responses based on those emotions. This system includes emotion recognition means, personalized notification means, video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, and chatbot means.

[0913] Hardware configuration

[0914] Smart glasses: Store staff wear them, and a camera device collects video data of customers.

[0915] Server: Processes video and audio data and generates notification messages based on the analysis results.

[0916] Display interface: Integrated into smart glasses, it displays notification messages.

[0917] Software Configuration

[0918] OpenCV: Used for preprocessing and analyzing video data.

[0919] DeepFace: An algorithm for facial recognition and emotion analysis.

[0920] Natural language processing model: Analyzes user questions and generates appropriate answers.

[0921] Database: Used to record analysis results.

[0922] Data processing and data calculation

[0923] 1. Video acquisition method: The camera device of the smart glasses collects video data in real time and transmits it to the server.

[0924] 2. Video analysis means: The server preprocesses the received video data (noise reduction, resolution conversion, etc.) and uses DeepFace to analyze the customer's emotions.

[0925] 3. Means for recording analysis results: The results of face recognition and emotion analysis are recorded in a database along with timestamps and location information.

[0926] 4. Emotion Recognition Method: Determine the customer's emotional state (e.g., anger, joy, surprise, etc.) in real time.

[0927] 5. Personalized notification means: Based on the customer's emotions determined by the emotion recognition means, an appropriate notification message is generated.

[0928] 6. Notification generation means: Creates a personalized notification message and displays it on the smart glasses' display via a notification sending means.

[0929] Adding specific examples

[0930] For example, if the server detects a customer's anger from the video data it receives, the personalized notification system will generate a message such as, "This customer is irritated. Please respond calmly," and display it on the smart glasses' screen. This allows store staff to adjust their customer service to match the customer's emotional state.

[0931] Examples of prompts to input into a generative AI model:

[0932] Prompt: Video frame showing the customer's face

[0933] Output: Analyzes customer emotions in real time and generates emotion-based response notification messages.

[0934] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0935] Step 1:

[0936] Acquisition of video data

[0937] A camera device built into the smart glasses collects video data of customers in the store in real time. The video data is transmitted from the smart glasses to the server in stream format. In this step, the input is images of customers in the store, and the output is the video data distributed to the server.

[0938] Step 2:

[0939] Preprocessing of video data

[0940] The server performs preprocessing on the received video data, including noise reduction, resolution conversion, and color space conversion. This process is carried out using OpenCV. The input is video data in stream format, and the output is preprocessed video frames.

[0941] Step 3:

[0942] emotion recognition

[0943] The server inputs pre-processed video frames into DeepFace for face recognition and emotion analysis. The input is pre-processed video frames, and the output is the customer's emotion (e.g., anger, joy, surprise) and its confidence score.

[0944] Step 4:

[0945] Record of results

[0946] The analysis results are stored in a database. The records include analyzed emotions, timestamps, and camera location information. The input is the emotion recognition result and its associated information, and the output is a database entry.

[0947] Step 5:

[0948] Generating notification messages

[0949] Based on the customer's emotions as determined by the emotion recognition system, an appropriate notification message is generated. The input is data from emotion recognition, and the output is a notification message. For example, if the customer is showing anger, the message "This customer is irritated. Please respond calmly." will be generated.

[0950] Step 6:

[0951] Sending notification messages

[0952] The generated notification message is sent to the smart glasses' display via the notification sending device. The input is the notification message, and the output is the text displayed on the smart glasses' display.

[0953] Step 7:

[0954] Customer support

[0955] Upon receiving the notification, store staff will respond to the customer according to the message displayed on the smart glasses' screen. The input is the notification message displayed on the smart glasses, and the output is personalized customer service.

[0956] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0957] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0958] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0959] [Third Embodiment]

[0960] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0961] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0962] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0963] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0964] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0965] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0966] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0967] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0968] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0969] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0970] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0971] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0972] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[0973] Camera image acquisition and analysis

[0974] server

[0975] The server periodically receives video data from camera devices installed within the home. Each camera is located in a different place within the home and transmits video in a stream format at regular intervals.

[0976] The received video data is first pre-processed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion.

[0977] The pre-processed video frames are sent to an AI model to detect dirt and pests. The AI ​​model uses deep learning algorithms such as pre-trained convolutional neural networks (CNNs).

[0978] Detection results are recorded in the database along with a confidence score. The data also includes location information and timestamps of the detected area.

[0979] Generating and sending notifications

[0980] server

[0981] The system periodically checks the latest detection results from the database, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention.

[0982] Based on the areas that need cleaning or where pests have been detected, the system generates appropriate notification messages. For example, it might generate a message like, "The kitchen drain is dirty. It's time to clean it."

[0983] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[0984] terminal

[0985] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[0986] Chatbot support

[0987] User

[0988] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[0989] terminal

[0990] Questions from users are sent to the server in real time.

[0991] server

[0992] The server analyzes the received question using a natural language processing model and generates an appropriate answer. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[0993] The generated response is sent back to the user's device and displayed in the chatbot interface.

[0994] Specific example

[0995] For example, suppose a camera installed in the kitchen drain sends video data at 7:00 AM one day. The server receives this video data and analyzes it using an AI model. As a result of the analysis, if dirt is detected in the drain, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server will respond with specific instructions. This allows the user to clean effectively.

[0996] Thus, the present invention makes it possible to clean the home efficiently and effectively, and to maintain a clean living environment on a daily basis.

[0997] The following describes the processing flow.

[0998] Camera image acquisition and analysis

[0999] Step 1: Acquire video

[1000] server

[1001] The server periodically receives video data from each camera installed within the home. Since the video data is transmitted in a stream format, real-time analysis is possible.

[1002] Step 2: Pretreatment

[1003] server

[1004] The received video data is first pre-processed. This pre-processing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[1005] Step 3: Video Analysis

[1006] server

[1007] Pre-processed video frames are input into the AI ​​model. The AI ​​model uses a pre-trained convolutional neural network (CNN) to detect dirt and pests in the images. The detection results are output along with a confidence score.

[1008] Step 4: Saving the analysis results

[1009] server

[1010] The detection results and confidence scores are recorded in a database. The record also includes timestamps of the video frames and camera location information. This stores information that will be useful for later reference.

[1011] Generating and sending notifications

[1012] Step 1: Extract the necessary notifications

[1013] server

[1014] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[1015] Step 2: Generate notification message

[1016] server

[1017] Based on the extracted information, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." is generated.

[1018] Step 3: Sending a notification message

[1019] server

[1020] The generated notification message is sent to the user's device. The notification is sent in real time so that the user can understand it immediately.

[1021] Step 4: Displaying notification messages

[1022] terminal

[1023] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to respond quickly.

[1024] Chatbot support

[1025] Step 1: Enter your question

[1026] User

[1027] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[1028] Step 2: Submit your question

[1029] terminal

[1030] The entered questions are sent to the server in real time.

[1031] Step 3: Analyzing the Question

[1032] server

[1033] The received question is analyzed using a natural language processing model. This analysis helps understand the user's intent and generates an appropriate answer.

[1034] Step 4: Generating the answer

[1035] server

[1036] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1037] Step 5: Submit your response

[1038] server

[1039] The generated response is sent to the user's device.

[1040] Step 6: Display the answer

[1041] terminal

[1042] The received response is displayed in the chatbot interface. The user can then use this displayed information to perform specific cleaning steps.

[1043] This series of processing steps allows users to clean their homes efficiently and effectively. Furthermore, this process helps maintain cleanliness in the household while reducing the user's burden.

[1044] (Example 1)

[1045] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1046] Household cleaning is time-consuming and laborious, and early detection of dirt and pests is particularly difficult. Furthermore, a lack of information on cleaning methods and tools often prevents proper treatment. This contributes to the problem of difficulty in maintaining cleanliness in the home.

[1047] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1048] In this invention, the server includes means for acquiring video, means for pre-processing video data, means for analyzing using an AI model, means for recording analysis results, means for generating notification messages, means for sending notification messages, user interface means, and chatbot means. This makes it possible to detect and notify about dirt and pests in the home at an early stage and provide information to the user quickly.

[1049] "Video acquisition means" refers to a device that periodically receives video data from a recording device installed in a home.

[1050] "Means for pre-processing video data" refers to a device or function that performs processing such as noise reduction, resolution conversion, and color space conversion on received video data.

[1051] "Methods of analysis using AI models" refers to devices or functions that analyze pre-processed video data using artificial intelligence technologies such as convolutional neural networks (CNNs) to detect dirt and pests.

[1052] "Means for recording analysis results" refers to a device or function that stores the detection results of dirt and pests obtained by the AI ​​model in a database along with a confidence score.

[1053] "Means for generating notification messages" refers to a device or function that creates appropriate notification messages for areas requiring cleaning or areas where pests have been detected, based on the analysis results.

[1054] "Means for sending notification messages" refers to a device or function that sends generated notification messages to the user's terminal in real time.

[1055] "User interface means" refers to a device or application that provides an interface for a user to communicate with a system.

[1056] A "chatbot" refers to an artificial intelligence-based conversational agent that automatically generates answers to questions from users about cleaning methods and tools to be used.

[1057] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[1058] Camera image acquisition and analysis

[1059] server

[1060] The server periodically receives video data from recording devices installed within the home. These devices can be standard webcams. Each camera is located in a different part of the home (e.g., kitchen, living room, entryway) and transmits video in a stream format at regular intervals.

[1061] The received video data is first pre-processed. This pre-processing includes noise reduction (using Gaussian filters, etc.), resolution conversion (converting to HD resolution), and color space conversion (converting from RGB to grayscale). Pre-processing is performed using image processing libraries such as OpenCV.

[1062] Preprocessed video frames are sent to an AI model using a convolutional neural network (CNN). This model is built with deep learning libraries such as TensorFlow and PyTorch. The AI ​​model detects dirt and pests in the video and generates a confidence score. The detection results are recorded in a database along with the confidence score. The recorded data includes location information and timestamps of the detected areas.

[1063] Generating and sending notifications

[1064] server

[1065] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. Based on the area that needs cleaning or where pests have been detected, it generates an appropriate notification message. For example, it might generate a message such as, "The kitchen drain is dirty. It's time to clean it."

[1066] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[1067] terminal

[1068] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[1069] Chatbot support

[1070] User

[1071] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[1072] terminal

[1073] Questions from users are sent to the server in real time.

[1074] server

[1075] The server analyzes the received question using a natural language processing model and generates an appropriate answer. This process uses generative AI models such as GPT-3. The generated answer is then sent back to the user's terminal and displayed in the chatbot interface.

[1076] Specific examples of operation

[1077] For example, suppose a camera installed in the kitchen drain transmits video data at 7:00 AM one day. The server receives this video data and performs noise reduction and grayscale conversion using a Gaussian filter. Then, it analyzes it with an AI model and detects dirt in the drain with a 95% confidence level. The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in its database and determines that cleaning is necessary. It generates a notification message saying, "Your kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server replies with specific steps: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." This allows the user to clean effectively.

[1078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1079] Step 1:

[1080] server

[1081] Video acquisition

[1082] Input: Video data transmitted in stream format from a home recording device.

[1083] Output: Video data received by the server

[1084] Example of operation: A camera installed in the kitchen sends video data to a server every 5 minutes. The server receives this data.

[1085] Step 2:

[1086] server

[1087] Preprocessing of video data

[1088] Input: Received video data

[1089] Output: Pre-processed video data (data after noise reduction, resolution conversion, and color space conversion)

[1090] Specific example of operation: The server removes noise from the received video data using a Gaussian filter, converts the resolution to HD, and converts it from RGB to grayscale.

[1091] Step 3:

[1092] server

[1093] AI-based video analysis

[1094] Input: Preprocessed video data

[1095] Output: Detection results for dirt and pests and their confidence scores.

[1096] Specific example of operation: The server inputs pre-processed video into an AI model (for example, a model using CNN) and detects dirt in the kitchen drain with a 95% confidence level.

[1097] Step 4:

[1098] server

[1099] Record of analysis results

[1100] Input: AI model detection results and confidence score

[1101] Output: Recorded detection results (including location information and timestamp)

[1102] Example of operation: The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in the database.

[1103] Step 5:

[1104] server

[1105] Generating notification messages

[1106] Input: Recorded detection results

[1107] Output: Notification message (reminds for cleaning or other actions)

[1108] Example of operation: Based on the recorded data, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it."

[1109] Step 6:

[1110] server

[1111] Sending notification messages

[1112] Input: Generated notification message

[1113] Output: Notification message sent to the user's terminal

[1114] Example of operation: The server sends the generated notification message as a push notification to the user's smartphone.

[1115] Step 7:

[1116] terminal

[1117] User notification display

[1118] Input: Notification message sent from the server

[1119] Output: Notification message displayed to the user

[1120] Example of operation: A notification appears on the user's smartphone saying, "The kitchen drain is dirty. It's time to clean it."

[1121] Step 8:

[1122] User

[1123] Questions for the chatbot

[1124] Input: User questions (e.g., "How do I clean the drain?")

[1125] Output: Question sent to the server

[1126] Example of operation: The user uses a smartphone app to input the question, "How do I clean a drain?"

[1127] Step 9:

[1128] server

[1129] Question analysis and answer generation

[1130] Input: Question from a user

[1131] Output: Generated answer

[1132] Example of operation: The server analyzes the received question using a generative AI model such as GPT-3 and generates the answer, "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[1133] Step 10:

[1134] server

[1135] Submit your response

[1136] Input: Generated answer

[1137] Output: Response sent to the user's terminal

[1138] Example of operation: The generated answer is sent to the user's smartphone.

[1139] Step 11:

[1140] terminal

[1141] Displaying responses to users

[1142] Input: Response sent from the server

[1143] Output: Answer displayed to the user

[1144] Example of operation: The user's smartphone displays the following message: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Clean with disinfectant."

[1145] (Application Example 1)

[1146] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1147] Traditional home cleaning systems were useful for detecting dirt and pests, but they lacked security features to detect suspicious movements or behavior. Therefore, there is a need to effectively address home security. Furthermore, they lacked a means to immediately suggest safety measures to users. Thus, there is a demand for systems that improve not only home cleaning but also overall security.

[1148] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1149] In this invention, the server includes video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, suspicious behavior detection means, and safety measure suggestion means. This makes it possible to detect not only dirt and pests in the home but also suspicious behavior and provide the user with immediate notifications and safety measures.

[1150] "Video acquisition means" refers to a function for periodically receiving video data from a camera device installed in the home.

[1151] "Video analysis means" refers to a function that preprocesses received video data and uses an AI model to analyze dirt, pests, suspicious behavior, etc.

[1152] The "analysis result recording means" is a function for recording the analysis results of detected dirt, pests, and suspicious behavior in a database.

[1153] The "notification generation means" is a function that automatically generates notification messages for the user based on the analysis results.

[1154] A "notification transmission method" is a function for sending generated notification messages to the user's terminal in real time.

[1155] A "user interface means" is a function that provides an interface for a user to interact with the system.

[1156] A "chatbot" is a function that provides an automated response feature to answer questions from users.

[1157] "Suspicious behavior detection means" refers to a function that complements video analysis means and detects suspicious movements and actions.

[1158] The "safety measures suggestion mechanism" is a function that suggests appropriate safety measures to the user when suspicious behavior is detected.

[1159] This invention is a system for efficiently and effectively cleaning and securing a home. This system operates through the cooperation of three entities: a server, a terminal, and a user, and utilizes in-home camera equipment to detect dirt, pests, and suspicious activity.

[1160] Camera image acquisition and analysis

[1161] server

[1162] Acquiring camera footage:

[1163] The server periodically receives video data from camera devices installed within the home. This data is received in stream format using RTSP (Real Time Streaming Protocol). By receiving video data from cameras installed in different locations within the home, the entire house is covered.

[1164] Pre-processing:

[1165] The received video data is first preprocessed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion using the OpenCV library.

[1166] AI analysis:

[1167] The pre-processed video frames are analyzed using deep learning algorithms such as convolutional neural networks (CNNs) based on TensorFlow or PyTorch. This analysis detects dirt, pests, and suspicious behavior.

[1168] Record of results:

[1169] Detection results are recorded in a database (PostgreSQL or MongoDB) along with a confidence score. The data also includes location information and timestamps of the detected areas.

[1170] Generating and sending notifications

[1171] server

[1172] Notification generation:

[1173] The latest detection results are periodically checked from the database, and if the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention. The same determination is made if suspicious behavior is detected.

[1174] Send notification:

[1175] The generated notification message is sent to the user's device. Real-time push notifications are sent using Firebase Cloud Messaging (FCM).

[1176] User terminal

[1177] Receive and display:

[1178] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to know in real time which areas need cleaning and what suspicious activity is happening in the home.

[1179] Chatbot support

[1180] User

[1181] Enter your question:

[1182] Users can use their devices to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?"

[1183] server

[1184] Response generation:

[1185] The system analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, follow these steps: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1186] Send response:

[1187] The generated response is sent back to the user's device and displayed in the chatbot interface.

[1188] Specific example

[1189] For example, if a camera installed in the kitchen detects dirt, the server generates a notification saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. Furthermore, if suspicious activity is detected in the kitchen, a notification saying, "There is suspicious activity in the kitchen. Please check," is also sent. If the user asks the chatbot, "What should I do if there is a suspicious person in the kitchen?", they will receive the answer, "If possible, move to a safe place and call the police. We also recommend activating your security system." This allows the user to take a quick and appropriate action.

[1190] Example of a prompt

[1191] What should I do if a suspicious person is detected by a home security camera?

[1192] Thus, this system integrates and manages household cleaning and security, providing users with immediately useful information and solutions.

[1193] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1194] Step 1: Receiving and pre-processing camera footage

[1195] The server periodically receives video data in stream format using RTSP from a camera device installed in the home. The input is the video stream from the camera, and the output is pre-processed video data that has undergone noise reduction, resolution conversion, and color space conversion.

[1196] Step 2: Detection using AI analysis

[1197] The server analyzes pre-processed video data using a convolutional neural network (CNN) based on TensorFlow or PyTorch. Here, it detects dirt, pests, and suspicious behavior. The input is pre-processed video data, and the output is the detection results for dirt, pests, and suspicious behavior.

[1198] Step 3: Recording the detection results

[1199] The server records the detection results in a database (PostgreSQL or MongoDB). This step also records the location information and timestamps of the detected areas. The input is the detection result from the AI ​​analysis, and the output is the detection result recorded in the database.

[1200] Step 4: Generate Notifications

[1201] The server periodically checks the database for the latest detection results and generates an appropriate notification message if the confidence score exceeds a certain threshold. A similar procedure is used to generate a notification message if suspicious activity is detected. The input is the detection results from the database, and the output is the generated notification message.

[1202] Step 5: Sending Notifications

[1203] The server sends the generated notification message to the user's device in real time using Firebase Cloud Messaging (FCM). The input is the generated notification message, and the output is the notification sent to the user's device.

[1204] Step 6: Receiving and displaying notifications

[1205] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to see in real time where cleaning is needed and any suspicious activity in the home. The input is the notification from the server, and the output is the notification displayed on the terminal.

[1206] Step 7: Enter your question for the chatbot.

[1207] The user uses a terminal to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?" The input is the user's question, and the output is the question data sent to the server.

[1208] Step 8: Question analysis and answer generation

[1209] The server analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." The input is the user's question, and the output is the generated answer.

[1210] Step 9: Submit and view your response

[1211] The server sends the generated response back to the user's terminal and displays it on the chatbot interface. The input is the generated response, and the output is the response displayed on the user's terminal.

[1212] In this way, each step works in conjunction to create a system that comprehensively manages household cleaning and security, providing users with quick and appropriate information and solutions.

[1213] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1214] This invention is a home cleaning assistant system that incorporates an emotion engine to recognize user emotions, thereby improving cleaning efficiency and providing more personalized support to the user. This system operates through the cooperation of three entities: a server, a terminal, and the user.

[1215] Camera image acquisition and analysis

[1216] server

[1217] The server periodically receives video data from multiple cameras installed within the home. The video data is received in stream format, allowing for real-time processing.

[1218] The received video data is preprocessed. Preprocessing includes noise reduction, resolution conversion, and color space conversion.

[1219] The pre-processed video frames are input into an AI model to detect dirt and pests. The AI ​​model uses a deep learning algorithm (e.g., a convolutional neural network).

[1220] The detection results and confidence scores are recorded in a database. The record also includes timestamps and camera location information.

[1221] Generating and sending notifications

[1222] server

[1223] The system periodically checks the latest detection results from the database, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention.

[1224] Generate an appropriate notification message and send it to the user's device. For example, create a notification that says, "The kitchen drain is dirty. It's time to clean it."

[1225] terminal

[1226] The user's terminal receives notifications from the server and displays them to the user in real time. This allows the user to respond immediately.

[1227] Chatbot support

[1228] User

[1229] The user uses a terminal to input questions into the system about cleaning methods and appropriate tools. For example, they might input a question like, "How do I clean the drain?"

[1230] terminal

[1231] Questions from users are sent to the server in real time.

[1232] server

[1233] The system analyzes questions using a natural language processing model and generates appropriate answers. For example, it can generate specific cleaning instructions such as, "To clean the drain, follow these steps: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1234] The generated response is sent to the user's device and displayed in the chatbot interface.

[1235] Embedding an emotion engine

[1236] server

[1237] The server incorporates an emotion engine that analyzes the user's voice and video data. For example, it uses voice data from a smartphone or smart speaker, or camera video data, to determine the user's emotional state in real time.

[1238] The content and wording of notification messages are adjusted based on the emotional state determined by the emotion engine (e.g., stress, joy, surprise). For example, if the user is stressed, the notification message will be made more friendly and include words of encouragement.

[1239] Specific example

[1240] For example, the server can detect emotions such as "fatigue" from the user's tone of voice and facial expressions. In this case, it generates a notification message that includes a helpful suggestion such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[1241] If a user asks, "How do I clean the drain?", and the emotion engine detects "interest," it will provide specific instructions along with an encouraging message such as, "It's easy to clean! First, let's remove the drain cover."

[1242] Thus, the present invention provides personalized support tailored to the user's emotional state, improving cleaning efficiency and making the user experience more comfortable.

[1243] The following describes the processing flow.

[1244] Camera image acquisition and analysis

[1245] Step 1: Acquire video

[1246] server

[1247] The server periodically receives video data from multiple cameras installed within the home. Each camera is located in a designated area (e.g., kitchen, living room, bathroom) and transmits the video data as a real-time stream.

[1248] Step 2: Video Preprocessing

[1249] server

[1250] The received video data is preprocessed. This preprocessing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[1251] Step 3: Detection of dirt and pests

[1252] server

[1253] Pre-processed video data is input into an AI model. The AI ​​model uses a trained convolutional neural network (CNN) to detect dirt and pests in the video. The detection results are output along with a confidence score.

[1254] Step 4: Saving the detection results

[1255] server

[1256] The detection results and confidence score are recorded in a database. The record includes timestamps of the video frames, camera location information, and information identifying the detected object.

[1257] Generating and sending notifications

[1258] Step 1: Check the latest detection results.

[1259] server

[1260] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[1261] Step 2: Generate notification message

[1262] server

[1263] Based on the location where dirt or pests are detected, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." can be created.

[1264] Step 3: Sending a notification message

[1265] server

[1266] The generated notification message is sent to the user's device. The notification is configured to be sent in real time.

[1267] Step 4: Displaying notification messages

[1268] terminal

[1269] The user's terminal receives a notification from the server and displays it to the user immediately. This allows the user to respond instantly.

[1270] Chatbot support

[1271] Step 1: Enter your question

[1272] User

[1273] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[1274] Step 2: Submit your question

[1275] terminal

[1276] The entered questions are sent to the server in real time.

[1277] Step 3: Analyzing the Question

[1278] server

[1279] The received question is analyzed using a natural language processing model. The intent of the question is identified, and an appropriate answer is generated.

[1280] Step 4: Generating the answer

[1281] server

[1282] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1283] Step 5: Submit your response

[1284] server

[1285] The generated response is sent to the user's device.

[1286] Step 6: Display the answer

[1287] terminal

[1288] The received response is displayed in the chatbot interface. The user can then use this information to perform specific cleaning steps.

[1289] Embedding an emotion engine

[1290] Step 1: Obtain user sentiment data

[1291] server

[1292] The server acquires the user's voice and video data. This includes data from smartphones, smart speakers, or cameras.

[1293] Step 2: Sentiment Analysis

[1294] server

[1295] The emotion engine analyzes acquired audio and video data to determine the user's emotional state in real time. Emotional states range widely, including stress, joy, and surprise.

[1296] Step 3: Utilizing emotional data

[1297] server

[1298] The emotion engine determines the user's emotional state and adjusts the content and wording of notification messages and responses accordingly. For example, if a user is stressed, the notification message might be made more friendly and include words of encouragement.

[1299] Specific example

[1300] server

[1301] The system detects "fatigue" from the user's tone of voice and facial expression. In this case, it generates a gentle notification message such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[1302] User

[1303] If a user asks "How do I clean the drain?" and the emotion engine detects "interest," the server will provide specific instructions along with an encouraging message such as "It's easy to clean! First, let's remove the drain cover."

[1304] This processing flow allows the present invention to provide personalized support tailored to the user's emotional state, thereby increasing cleaning efficiency and improving the user experience.

[1305] (Example 2)

[1306] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1307] Traditional home cleaning systems have limited automation and have failed to completely alleviate the user's burden. Furthermore, existing systems did not consider the user's emotional state and only offered a single notification method, resulting in a suboptimal user experience. This could potentially increase user stress.

[1308] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1309] In this invention, the server includes video acquisition means, video preprocessing means, AI analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, and emotion analysis means. This enables everything from automatic detection of dirt and pests to personalized notifications and responses tailored to the user's emotional state.

[1310] "Video acquisition means" refers to a means of receiving video data in real time from a video device installed in a home.

[1311] "Video preprocessing means" refers to means for removing noise from received video data, converting the resolution, and converting the color space.

[1312] "AI analysis means" refers to a method that analyzes pre-processed video data and uses a generative AI model to detect specific objects such as dirt or pests.

[1313] The "analysis result recording means" is a means for recording the detection results and confidence scores obtained by the AI ​​analysis means in a database.

[1314] A "notification generation means" is a means for generating notification messages for the user based on the analysis results.

[1315] "Notification transmission means" refers to a means for sending the generated notification message to the user's terminal.

[1316] "User interface means" refers to means that provide an interface for users to access a system and receive notifications and responses.

[1317] A "chatbot" is a means of automatically generating appropriate answers to user questions and providing those answers.

[1318] "Emotion analysis means" refers to a method for analyzing a user's voice data and video data to determine the user's emotional state in real time.

[1319] This invention is a home cleaning assistant system that incorporates an emotion engine that recognizes the user's emotions. This system operates through the cooperation of three entities: a server, a terminal, and the user.

[1320] The server receives video data in real time from multiple video devices (e.g., cameras) installed in the home. The video data is acquired in stream format, and then noise reduction, resolution conversion, and color space conversion are performed by video preprocessing. This utilizes technologies such as Gaussian filters and the OpenCV library.

[1321] The pre-processed video data is input into an AI analysis system to detect dirt and pests. Here, a convolutional neural network (CNN)-based generative AI model is used to identify dirt and pests. Implementation using the TensorFlow library is preferable.

[1322] Detection results and confidence scores are recorded in a database by an analysis result recording device. The record includes timestamps and location information of the video device, and utilizes a MySQL database. The server periodically checks the database, and if the confidence score of the latest detection result exceeds a certain threshold, an appropriate notification message is created by a notification generation device.

[1323] The notification generation mechanism creates a notification with content such as, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's device via the notification sending mechanism. In this case, Firebase Cloud Messaging (FCM) is suitable. The device receives this notification in real time and displays it to the user.

[1324] Users can access the system through a terminal and input questions about cleaning methods and appropriate tools. Questions such as "How do I clean the drain?" are sent to the server in real time via the user interface. The server analyzes these questions using a natural language processing model (e.g., BERT) and generates appropriate answers. For example, it might provide specific cleaning instructions such as, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1325] The system incorporates emotion analysis capabilities, analyzing the user's voice and video data to determine their emotional state in real time. It uses the Google Cloud Speech-to-Text API for voice analysis and the Emotion API for emotion determination. The system adjusts the content and wording of notification messages according to the user's emotional state. For example, if the user is fatigued, it might generate a message including a helpful suggestion such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[1326] As a concrete example, consider a scenario where the server generates a notification message saying, "You seem tired today. Shall we postpone cleaning the drain for a while?" An example of the prompt text is as follows:

[1327] Example of a prompt:

[1328] "We are informing you of the current situation: The camera has detected dirt in the kitchen drain. However, the user is fatigued, so please include a gentle suggestion in the notification message."

[1329] Thus, the cleaning assistant system of the present invention provides personalized support according to the user's emotional state, improving cleaning efficiency and making the user experience more comfortable.

[1330] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1331] Step 1: Acquire camera footage

[1332] server

[1333] The server receives video data in real time from multiple cameras installed within the home. The cameras transmit data in a 30 frames per second (FPS) stream format. The received video data is encoded in H.264 format. The input is video data from each camera, and the output is video data in stream format.

[1334] Specific operation: Video data transmitted from each camera is received in real time by a dedicated receiving module.

[1335] Step 2: Preprocessing of video data

[1336] server

[1337] The server preprocesses the received video data by denoising, converting the resolution, and converting the color space. A Gaussian filter is used for denoising, and the resolution conversion involves downsampling from 1080p to 720p. Color space conversion is performed from RGB to grayscale. The input is video data in stream format, and the output is preprocessed video data.

[1338] Specific operation: Using the OpenCV library, a series of preprocessing algorithms are applied to the received video data.

[1339] Step 3: AI-powered detection of dirt and pests

[1340] server

[1341] The server inputs pre-processed video data into a convolutional neural network (CNN)-based generative AI model to detect dirt and pests. During this process, features are extracted from each frame and input into the AI ​​model. The input is pre-processed video data, and the output is the detection result and confidence score.

[1342] Specific operation: Using the TensorFlow library, a pre-trained AI model is applied to detect the presence or absence of dirt or pests within 0.1 seconds for each frame.

[1343] Step 4: Record the detection results

[1344] server

[1345] The server records detection results and confidence scores from the AI ​​model in a database. Recorded items include timestamps and camera location information. The input is the detection results and confidence scores, and the output is the recorded data stored in the database.

[1346] Specific operation: Connect to the MySQL database and insert each detection result into a dedicated table.

[1347] Step 5: Generate Notifications

[1348] server

[1349] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. It then generates an appropriate notification message. The input is the detection results from the database, and the output is the notification message.

[1350] Specific action: Generate a notification with a message like, "The kitchen drain is dirty. It's time to clean it."

[1351] Step 6: Send the notification

[1352] server

[1353] The server sends the generated notification message to the user's device. Firebase Cloud Messaging (FCM) is used for push notifications. The input is the notification message, and the output is the notification sent to the user's device.

[1354] Specific operation: Use the FCM API to send notifications to the user's smartphone or tablet.

[1355] Step 7: Responding to user inquiries

[1356] User

[1357] Users access the system through a terminal and input questions about cleaning methods and appropriate tools. The input is the user's question, and the output is the question data.

[1358] terminal

[1359] The terminal sends user questions to the server in real time. The input is the user's question, and the output is the question data sent to the server.

[1360] server

[1361] The server analyzes the user's question using a natural language processing model (e.g., BERT) and generates an appropriate answer. The input is the question data, and the output is the answer message.

[1362] Specific operation: The Transformers library is used for natural language processing, and an algorithm is applied to generate appropriate answers to questions.

[1363] Step 8: Adjusting Sentiment Analysis and Notifications

[1364] server

[1365] The server uses emotion analysis tools to analyze the user's voice and video data and determine the user's emotional state in real time. Based on the emotional state, it adjusts the content and wording of notification messages. The input is the user's voice and video data, and the output is the adjusted notification message.

[1366] Specific operation: The Google Cloud Speech-to-Text API is used for speech analysis, and the Emotion API is used for emotion detection, dynamically changing the wording of notification messages.

[1367] In this way, the entire system works together to provide personalized support to the user, improving cleaning efficiency and making the user experience more comfortable.

[1368] (Application Example 2)

[1369] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1370] In existing brick-and-mortar stores, providing personalized service tailored to customers' emotional states is difficult, often resulting in a uniform service quality and consequently lower customer satisfaction. Furthermore, there is no system in place to recognize customers' emotions in real time from their facial expressions and voices and respond appropriately based on that information. As a result, the quality of the customer experience remains stagnant.

[1371] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means, personalized notification means, and video acquisition means. This makes it possible to recognize the customer's emotional state in real time and send a personalized notification message to store staff accordingly.

[1372] "Emotion recognition means" refers to a function that analyzes audio and video data to determine the user's emotional state in real time.

[1373] A "personalized notification method" is a function that adjusts the content and expression of notification messages based on the user's emotional state as determined by the emotion recognition method.

[1374] "Video acquisition means" refers to a camera device and a function that periodically receives video data in a stream format.

[1375] The "video analysis means" refers to a function that preprocesses received video data and uses an AI model to detect dirt and pests.

[1376] The "analysis result recording means" is a function that records the results of video analysis and stores them in a database that includes timestamps and location information.

[1377] The "notification generation means" is a function that generates an appropriate notification message based on the analysis results.

[1378] "Notification transmission means" refers to a function that sends the generated notification message to the user interface means.

[1379] A "user interface means" is an interface for exchanging information between a user and a system.

[1380] A "chatbot" is a function that receives questions from users, generates appropriate answers, and sends responses.

[1381] The present invention is a system that recognizes customer emotions in a physical store and provides personalized responses based on those emotions. This system includes emotion recognition means, personalized notification means, video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, and chatbot means.

[1382] Hardware configuration

[1383] Smart glasses: Store staff wear them, and a camera device collects video data of customers.

[1384] Server: Processes video and audio data and generates notification messages based on the analysis results.

[1385] Display interface: Integrated into smart glasses, it displays notification messages.

[1386] Software Configuration

[1387] OpenCV: Used for preprocessing and analyzing video data.

[1388] DeepFace: An algorithm for facial recognition and emotion analysis.

[1389] Natural language processing model: Analyzes user questions and generates appropriate answers.

[1390] Database: Used to record analysis results.

[1391] Data processing and data calculation

[1392] 1. Video acquisition method: The camera device of the smart glasses collects video data in real time and transmits it to the server.

[1393] 2. Video analysis means: The server preprocesses the received video data (noise reduction, resolution conversion, etc.) and uses DeepFace to analyze the customer's emotions.

[1394] 3. Means for recording analysis results: The results of face recognition and emotion analysis are recorded in a database along with timestamps and location information.

[1395] 4. Emotion Recognition Method: Determine the customer's emotional state (e.g., anger, joy, surprise, etc.) in real time.

[1396] 5. Personalized notification means: Based on the customer's emotions determined by the emotion recognition means, an appropriate notification message is generated.

[1397] 6. Notification generation means: Creates a personalized notification message and displays it on the smart glasses' display via a notification sending means.

[1398] Adding specific examples

[1399] For example, if the server detects a customer's anger from the video data it receives, the personalized notification system will generate a message such as, "This customer is irritated. Please respond calmly," and display it on the smart glasses' screen. This allows store staff to adjust their customer service to match the customer's emotional state.

[1400] Examples of prompts to input into a generative AI model:

[1401] Prompt: Video frame showing the customer's face

[1402] Output: Analyzes customer emotions in real time and generates emotion-based response notification messages.

[1403] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1404] Step 1:

[1405] Acquisition of video data

[1406] A camera device built into the smart glasses collects video data of customers in the store in real time. The video data is transmitted from the smart glasses to the server in stream format. In this step, the input is images of customers in the store, and the output is the video data distributed to the server.

[1407] Step 2:

[1408] Preprocessing of video data

[1409] The server performs preprocessing on the received video data, including noise reduction, resolution conversion, and color space conversion. This process is carried out using OpenCV. The input is video data in stream format, and the output is preprocessed video frames.

[1410] Step 3:

[1411] emotion recognition

[1412] The server inputs pre-processed video frames into DeepFace for face recognition and emotion analysis. The input is pre-processed video frames, and the output is the customer's emotion (e.g., anger, joy, surprise) and its confidence score.

[1413] Step 4:

[1414] Record of results

[1415] The analysis results are stored in a database. The records include analyzed emotions, timestamps, and camera location information. The input is the emotion recognition result and its associated information, and the output is a database entry.

[1416] Step 5:

[1417] Generating notification messages

[1418] Based on the customer's emotions as determined by the emotion recognition system, an appropriate notification message is generated. The input is data from emotion recognition, and the output is a notification message. For example, if the customer is showing anger, the message "This customer is irritated. Please respond calmly." will be generated.

[1419] Step 6:

[1420] Sending notification messages

[1421] The generated notification message is sent to the smart glasses' display via the notification sending device. The input is the notification message, and the output is the text displayed on the smart glasses' display.

[1422] Step 7:

[1423] Customer support

[1424] Upon receiving the notification, store staff will respond to the customer according to the message displayed on the smart glasses' screen. The input is the notification message displayed on the smart glasses, and the output is personalized customer service.

[1425] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1426] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1427] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1428] [Fourth Embodiment]

[1429] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1430] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1431] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1432] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1433] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1434] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1435] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1436] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1437] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1438] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1439] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1440] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1441] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1442] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[1443] Camera image acquisition and analysis

[1444] server

[1445] The server periodically receives video data from camera devices installed within the home. Each camera is located in a different place within the home and transmits video in a stream format at regular intervals.

[1446] The received video data is first pre-processed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion.

[1447] The pre-processed video frames are sent to an AI model to detect dirt and pests. The AI ​​model uses deep learning algorithms such as pre-trained convolutional neural networks (CNNs).

[1448] Detection results are recorded in the database along with a confidence score. The data also includes location information and timestamps of the detected area.

[1449] Generating and sending notifications

[1450] server

[1451] The system periodically checks the latest detection results from the database, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention.

[1452] Based on the areas that need cleaning or where pests have been detected, the system generates appropriate notification messages. For example, it might generate a message like, "The kitchen drain is dirty. It's time to clean it."

[1453] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[1454] terminal

[1455] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[1456] Chatbot support

[1457] User

[1458] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[1459] terminal

[1460] Questions from users are sent to the server in real time.

[1461] server

[1462] The server analyzes the received question using a natural language processing model and generates an appropriate answer. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[1463] The generated response is sent back to the user's device and displayed in the chatbot interface.

[1464] Specific example

[1465] For example, suppose a camera installed in the kitchen drain sends video data at 7:00 AM one day. The server receives this video data and analyzes it using an AI model. As a result of the analysis, if dirt is detected in the drain, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server will respond with specific instructions. This allows the user to clean effectively.

[1466] Thus, the present invention makes it possible to clean the home efficiently and effectively, and to maintain a clean living environment on a daily basis.

[1467] The following describes the processing flow.

[1468] Camera image acquisition and analysis

[1469] Step 1: Acquire video

[1470] server

[1471] The server periodically receives video data from each camera installed within the home. Since the video data is transmitted in a stream format, real-time analysis is possible.

[1472] Step 2: Pretreatment

[1473] server

[1474] The received video data is first pre-processed. This pre-processing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[1475] Step 3: Video Analysis

[1476] server

[1477] Pre-processed video frames are input into the AI ​​model. The AI ​​model uses a pre-trained convolutional neural network (CNN) to detect dirt and pests in the images. The detection results are output along with a confidence score.

[1478] Step 4: Saving the analysis results

[1479] server

[1480] The detection results and confidence scores are recorded in a database. The record also includes timestamps of the video frames and camera location information. This stores information that will be useful for later reference.

[1481] Generating and sending notifications

[1482] Step 1: Extract the necessary notifications

[1483] server

[1484] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[1485] Step 2: Generate notification message

[1486] server

[1487] Based on the extracted information, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." is generated.

[1488] Step 3: Sending a notification message

[1489] server

[1490] The generated notification message is sent to the user's device. The notification is sent in real time so that the user can understand it immediately.

[1491] Step 4: Displaying notification messages

[1492] terminal

[1493] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to respond quickly.

[1494] Chatbot support

[1495] Step 1: Enter your question

[1496] User

[1497] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[1498] Step 2: Submit your question

[1499] terminal

[1500] The entered questions are sent to the server in real time.

[1501] Step 3: Analyzing the Question

[1502] server

[1503] The received question is analyzed using a natural language processing model. This analysis helps understand the user's intent and generates an appropriate answer.

[1504] Step 4: Generating the answer

[1505] server

[1506] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1507] Step 5: Submit your response

[1508] server

[1509] The generated response is sent to the user's device.

[1510] Step 6: Display the answer

[1511] terminal

[1512] The received response is displayed in the chatbot interface. The user can then use this displayed information to perform specific cleaning steps.

[1513] This series of processing steps allows users to clean their homes efficiently and effectively. Furthermore, this process helps maintain cleanliness in the household while reducing the user's burden.

[1514] (Example 1)

[1515] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1516] Household cleaning is time-consuming and laborious, and early detection of dirt and pests is particularly difficult. Furthermore, a lack of information on cleaning methods and tools often prevents proper treatment. This contributes to the problem of difficulty in maintaining cleanliness in the home.

[1517] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1518] In this invention, the server includes means for acquiring video, means for pre-processing video data, means for analyzing using an AI model, means for recording analysis results, means for generating notification messages, means for sending notification messages, user interface means, and chatbot means. This makes it possible to detect and notify about dirt and pests in the home at an early stage and provide information to the user quickly.

[1519] "Video acquisition means" refers to a device that periodically receives video data from a recording device installed in a home.

[1520] "Means for pre-processing video data" refers to a device or function that performs processing such as noise reduction, resolution conversion, and color space conversion on received video data.

[1521] "Methods of analysis using AI models" refers to devices or functions that analyze pre-processed video data using artificial intelligence technologies such as convolutional neural networks (CNNs) to detect dirt and pests.

[1522] "Means for recording analysis results" refers to a device or function that stores the detection results of dirt and pests obtained by the AI ​​model in a database along with a confidence score.

[1523] "Means for generating notification messages" refers to a device or function that creates appropriate notification messages for areas requiring cleaning or areas where pests have been detected, based on the analysis results.

[1524] "Means for sending notification messages" refers to a device or function that sends generated notification messages to the user's terminal in real time.

[1525] "User interface means" refers to a device or application that provides an interface for a user to communicate with a system.

[1526] A "chatbot" refers to an artificial intelligence-based conversational agent that automatically generates answers to questions from users about cleaning methods and tools to be used.

[1527] This invention is a system for efficiently and effectively cleaning within the home. This system operates through the cooperation of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[1528] Camera image acquisition and analysis

[1529] server

[1530] The server periodically receives video data from recording devices installed within the home. These devices can be standard webcams. Each camera is located in a different part of the home (e.g., kitchen, living room, entryway) and transmits video in a stream format at regular intervals.

[1531] The received video data is first pre-processed. This pre-processing includes noise reduction (using Gaussian filters, etc.), resolution conversion (converting to HD resolution), and color space conversion (converting from RGB to grayscale). Pre-processing is performed using image processing libraries such as OpenCV.

[1532] Preprocessed video frames are sent to an AI model using a convolutional neural network (CNN). This model is built with deep learning libraries such as TensorFlow and PyTorch. The AI ​​model detects dirt and pests in the video and generates a confidence score. The detection results are recorded in a database along with the confidence score. The recorded data includes location information and timestamps of the detected areas.

[1533] Generating and sending notifications

[1534] server

[1535] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. Based on the area that needs cleaning or where pests have been detected, it generates an appropriate notification message. For example, it might generate a message such as, "The kitchen drain is dirty. It's time to clean it."

[1536] The generated notification message is sent to the user's device. The notification is sent in real time and displayed as a push notification or an in-app message.

[1537] terminal

[1538] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to understand the need for cleaning in real time.

[1539] Chatbot support

[1540] User

[1541] Users can use their devices to ask the system questions about cleaning methods, tips, and tools to use. For example, they might type, "How do I clean the drain?"

[1542] terminal

[1543] Questions from users are sent to the server in real time.

[1544] server

[1545] The server analyzes the received question using a natural language processing model and generates an appropriate answer. This process uses generative AI models such as GPT-3. The generated answer is then sent back to the user's terminal and displayed in the chatbot interface.

[1546] Specific examples of operation

[1547] For example, suppose a camera installed in the kitchen drain transmits video data at 7:00 AM one day. The server receives this video data and performs noise reduction and grayscale conversion using a Gaussian filter. Then, it analyzes it with an AI model and detects dirt in the drain with a 95% confidence level. The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in its database and determines that cleaning is necessary. It generates a notification message saying, "Your kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. The user sees this notification and starts cleaning the drain. Also, if the user asks the chatbot, "How do I clean the drain?", the server replies with specific steps: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." This allows the user to clean effectively.

[1548] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1549] Step 1:

[1550] server

[1551] Video acquisition

[1552] Input: Video data transmitted in stream format from a home recording device.

[1553] Output: Video data received by the server

[1554] Example of operation: A camera installed in the kitchen sends video data to a server every 5 minutes. The server receives this data.

[1555] Step 2:

[1556] server

[1557] Preprocessing of video data

[1558] Input: Received video data

[1559] Output: Pre-processed video data (data after noise reduction, resolution conversion, and color space conversion)

[1560] Specific example of operation: The server removes noise from the received video data using a Gaussian filter, converts the resolution to HD, and converts it from RGB to grayscale.

[1561] Step 3:

[1562] server

[1563] AI-based video analysis

[1564] Input: Preprocessed video data

[1565] Output: Detection results for dirt and pests and their confidence scores.

[1566] Specific example of operation: The server inputs pre-processed video into an AI model (for example, a model using CNN) and detects dirt in the kitchen drain with a 95% confidence level.

[1567] Step 4:

[1568] server

[1569] Record of analysis results

[1570] Input: AI model detection results and confidence score

[1571] Output: Recorded detection results (including location information and timestamp)

[1572] Example of operation: The server records the data "Kitchen drain, October 1, 2023, 7:00 AM, 95% confidence level" in the database.

[1573] Step 5:

[1574] server

[1575] Generating notification messages

[1576] Input: Recorded detection results

[1577] Output: Notification message (reminds for cleaning or other actions)

[1578] Example of operation: Based on the recorded data, the server generates a notification message saying, "The kitchen drain is dirty. It's time to clean it."

[1579] Step 6:

[1580] server

[1581] Sending notification messages

[1582] Input: Generated notification message

[1583] Output: Notification message sent to the user's terminal

[1584] Example of operation: The server sends the generated notification message as a push notification to the user's smartphone.

[1585] Step 7:

[1586] terminal

[1587] User notification display

[1588] Input: Notification message sent from the server

[1589] Output: Notification message displayed to the user

[1590] Example of operation: A notification appears on the user's smartphone saying, "The kitchen drain is dirty. It's time to clean it."

[1591] Step 8:

[1592] User

[1593] Questions for the chatbot

[1594] Input: User questions (e.g., "How do I clean the drain?")

[1595] Output: Question sent to the server

[1596] Example of operation: The user uses a smartphone app to input the question, "How do I clean a drain?"

[1597] Step 9:

[1598] server

[1599] Question analysis and answer generation

[1600] Input: Question from a user

[1601] Output: Generated answer

[1602] Example of operation: The server analyzes the received question using a generative AI model such as GPT-3 and generates the answer, "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant."

[1603] Step 10:

[1604] server

[1605] Submit your response

[1606] Input: Generated answer

[1607] Output: Response sent to the user's terminal

[1608] Example of operation: The generated answer is sent to the user's smartphone.

[1609] Step 11:

[1610] terminal

[1611] Displaying responses to users

[1612] Input: Response sent from the server

[1613] Output: Answer displayed to the user

[1614] Example of operation: The user's smartphone displays the following message: "Here's how to clean the drain: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Clean with disinfectant."

[1615] (Application Example 1)

[1616] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1617] Traditional home cleaning systems were useful for detecting dirt and pests, but they lacked security features to detect suspicious movements or behavior. Therefore, there is a need to effectively address home security. Furthermore, they lacked a means to immediately suggest safety measures to users. Thus, there is a demand for systems that improve not only home cleaning but also overall security.

[1618] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1619] In this invention, the server includes video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, suspicious behavior detection means, and safety measure suggestion means. This makes it possible to detect not only dirt and pests in the home but also suspicious behavior and provide the user with immediate notifications and safety measures.

[1620] "Video acquisition means" refers to a function for periodically receiving video data from a camera device installed in the home.

[1621] "Video analysis means" refers to a function that preprocesses received video data and uses an AI model to analyze dirt, pests, suspicious behavior, etc.

[1622] The "analysis result recording means" is a function for recording the analysis results of detected dirt, pests, and suspicious behavior in a database.

[1623] The "notification generation means" is a function that automatically generates notification messages for the user based on the analysis results.

[1624] A "notification transmission method" is a function for sending generated notification messages to the user's terminal in real time.

[1625] A "user interface means" is a function that provides an interface for a user to interact with the system.

[1626] A "chatbot" is a function that provides an automated response feature to answer questions from users.

[1627] "Suspicious behavior detection means" refers to a function that complements video analysis means and detects suspicious movements and actions.

[1628] The "safety measures suggestion mechanism" is a function that suggests appropriate safety measures to the user when suspicious behavior is detected.

[1629] This invention is a system for efficiently and effectively cleaning and securing a home. This system operates through the cooperation of three entities: a server, a terminal, and a user, and utilizes in-home camera equipment to detect dirt, pests, and suspicious activity.

[1630] Camera image acquisition and analysis

[1631] server

[1632] Acquiring camera footage:

[1633] The server periodically receives video data from camera devices installed within the home. This data is received in stream format using RTSP (Real Time Streaming Protocol). By receiving video data from cameras installed in different locations within the home, the entire house is covered.

[1634] Pre-processing:

[1635] The received video data is first preprocessed. This includes standard image processing such as noise reduction, resolution conversion, and color space conversion using the OpenCV library.

[1636] AI analysis:

[1637] The pre-processed video frames are analyzed using deep learning algorithms such as convolutional neural networks (CNNs) based on TensorFlow or PyTorch. This analysis detects dirt, pests, and suspicious behavior.

[1638] Record of results:

[1639] Detection results are recorded in a database (PostgreSQL or MongoDB) along with a confidence score. The data also includes location information and timestamps of the detected areas.

[1640] Generating and sending notifications

[1641] server

[1642] Notification generation:

[1643] The latest detection results are periodically checked from the database, and if the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention. The same determination is made if suspicious behavior is detected.

[1644] Send notification:

[1645] The generated notification message is sent to the user's device. Real-time push notifications are sent using Firebase Cloud Messaging (FCM).

[1646] User terminal

[1647] Receive and display:

[1648] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to know in real time which areas need cleaning and what suspicious activity is happening in the home.

[1649] Chatbot support

[1650] User

[1651] Enter your question:

[1652] Users can use their devices to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?"

[1653] server

[1654] Response generation:

[1655] The system analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, follow these steps: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1656] Send response:

[1657] The generated response is sent back to the user's device and displayed in the chatbot interface.

[1658] Specific example

[1659] For example, if a camera installed in the kitchen detects dirt, the server generates a notification saying, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's smartphone. Furthermore, if suspicious activity is detected in the kitchen, a notification saying, "There is suspicious activity in the kitchen. Please check," is also sent. If the user asks the chatbot, "What should I do if there is a suspicious person in the kitchen?", they will receive the answer, "If possible, move to a safe place and call the police. We also recommend activating your security system." This allows the user to take a quick and appropriate action.

[1660] Example of a prompt

[1661] What should I do if a suspicious person is detected by a home security camera?

[1662] Thus, this system integrates and manages household cleaning and security, providing users with immediately useful information and solutions.

[1663] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1664] Step 1: Receiving and pre-processing camera footage

[1665] The server periodically receives video data in stream format using RTSP from a camera device installed in the home. The input is the video stream from the camera, and the output is pre-processed video data that has undergone noise reduction, resolution conversion, and color space conversion.

[1666] Step 2: Detection using AI analysis

[1667] The server analyzes pre-processed video data using a convolutional neural network (CNN) based on TensorFlow or PyTorch. Here, it detects dirt, pests, and suspicious behavior. The input is pre-processed video data, and the output is the detection results for dirt, pests, and suspicious behavior.

[1668] Step 3: Recording the detection results

[1669] The server records the detection results in a database (PostgreSQL or MongoDB). This step also records the location information and timestamps of the detected areas. The input is the detection result from the AI ​​analysis, and the output is the detection result recorded in the database.

[1670] Step 4: Generate Notifications

[1671] The server periodically checks the database for the latest detection results and generates an appropriate notification message if the confidence score exceeds a certain threshold. A similar procedure is used to generate a notification message if suspicious activity is detected. The input is the detection results from the database, and the output is the generated notification message.

[1672] Step 5: Sending Notifications

[1673] The server sends the generated notification message to the user's device in real time using Firebase Cloud Messaging (FCM). The input is the generated notification message, and the output is the notification sent to the user's device.

[1674] Step 6: Receiving and displaying notifications

[1675] The user's terminal receives notifications from the server and displays them to the user immediately. This allows the user to see in real time where cleaning is needed and any suspicious activity in the home. The input is the notification from the server, and the output is the notification displayed on the terminal.

[1676] Step 7: Enter your question for the chatbot.

[1677] The user uses a terminal to ask the system questions about cleaning methods and safety measures. For example, they might type, "How do I clean the drain?" The input is the user's question, and the output is the question data sent to the server.

[1678] Step 8: Question analysis and answer generation

[1679] The server analyzes user questions using natural language processing models (such as GPT-3 or BERT) and generates appropriate answers. For example, it might generate an answer like, "To clean a drain, do the following: 1. Remove the drain cover. 2. Remove the dirt with a brush. 3. Wash with disinfectant." The input is the user's question, and the output is the generated answer.

[1680] Step 9: Submit and view your response

[1681] The server sends the generated response back to the user's terminal and displays it on the chatbot interface. The input is the generated response, and the output is the response displayed on the user's terminal.

[1682] In this way, each step works in conjunction to create a system that comprehensively manages household cleaning and security, providing users with quick and appropriate information and solutions.

[1683] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1684] This invention is a home cleaning assistant system that incorporates an emotion engine to recognize user emotions, thereby improving cleaning efficiency and providing more personalized support to the user. This system operates through the cooperation of three entities: a server, a terminal, and the user.

[1685] Camera image acquisition and analysis

[1686] server

[1687] The server periodically receives video data from multiple cameras installed within the home. The video data is received in stream format, allowing for real-time processing.

[1688] The received video data is preprocessed. Preprocessing includes noise reduction, resolution conversion, and color space conversion.

[1689] The pre-processed video frames are input into an AI model to detect dirt and pests. The AI ​​model uses a deep learning algorithm (e.g., a convolutional neural network).

[1690] The detection results and confidence scores are recorded in a database. The record also includes timestamps and camera location information.

[1691] Generating and sending notifications

[1692] server

[1693] The system periodically checks the latest detection results from the database, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention.

[1694] Generate an appropriate notification message and send it to the user's device. For example, create a notification that says, "The kitchen drain is dirty. It's time to clean it."

[1695] terminal

[1696] The user's terminal receives notifications from the server and displays them to the user in real time. This allows the user to respond immediately.

[1697] Chatbot support

[1698] User

[1699] The user uses a terminal to input questions into the system about cleaning methods and appropriate tools. For example, they might input a question like, "How do I clean the drain?"

[1700] terminal

[1701] Questions from users are sent to the server in real time.

[1702] server

[1703] The system analyzes questions using a natural language processing model and generates appropriate answers. For example, it can generate specific cleaning instructions such as, "To clean the drain, follow these steps: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1704] The generated response is sent to the user's device and displayed in the chatbot interface.

[1705] Embedding an emotion engine

[1706] server

[1707] The server incorporates an emotion engine that analyzes the user's voice and video data. For example, it uses voice data from a smartphone or smart speaker, or camera video data, to determine the user's emotional state in real time.

[1708] The content and wording of notification messages are adjusted based on the emotional state determined by the emotion engine (e.g., stress, joy, surprise). For example, if the user is stressed, the notification message will be made more friendly and include words of encouragement.

[1709] Specific example

[1710] For example, the server can detect emotions such as "fatigue" from the user's tone of voice and facial expressions. In this case, it generates a notification message that includes a helpful suggestion such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[1711] If a user asks, "How do I clean the drain?", and the emotion engine detects "interest," it will provide specific instructions along with an encouraging message such as, "It's easy to clean! First, let's remove the drain cover."

[1712] Thus, the present invention provides personalized support tailored to the user's emotional state, improving cleaning efficiency and making the user experience more comfortable.

[1713] The following describes the processing flow.

[1714] Camera image acquisition and analysis

[1715] Step 1: Acquire video

[1716] server

[1717] The server periodically receives video data from multiple cameras installed within the home. Each camera is located in a designated area (e.g., kitchen, living room, bathroom) and transmits the video data as a real-time stream.

[1718] Step 2: Video Preprocessing

[1719] server

[1720] The received video data is preprocessed. This preprocessing includes noise reduction (e.g., Gaussian filtering), resolution conversion (e.g., conversion from high resolution to low resolution), and color space conversion (e.g., RGB to grayscale).

[1721] Step 3: Detection of dirt and pests

[1722] server

[1723] Pre-processed video data is input into an AI model. The AI ​​model uses a trained convolutional neural network (CNN) to detect dirt and pests in the video. The detection results are output along with a confidence score.

[1724] Step 4: Saving the detection results

[1725] server

[1726] The detection results and confidence score are recorded in a database. The record includes timestamps of the video frames, camera location information, and information identifying the detected object.

[1727] Generating and sending notifications

[1728] Step 1: Check the latest detection results.

[1729] server

[1730] The latest detection results are regularly checked from the database. If the confidence score exceeds a certain threshold, it is determined that the area requires cleaning or attention.

[1731] Step 2: Generate notification message

[1732] server

[1733] Based on the location where dirt or pests are detected, an appropriate notification message is generated. For example, a message such as "The kitchen drain is dirty. It's time to clean it." can be created.

[1734] Step 3: Sending a notification message

[1735] server

[1736] The generated notification message is sent to the user's device. The notification is configured to be sent in real time.

[1737] Step 4: Displaying notification messages

[1738] terminal

[1739] The user's terminal receives a notification from the server and displays it to the user immediately. This allows the user to respond instantly.

[1740] Chatbot support

[1741] Step 1: Enter your question

[1742] User

[1743] Users use the chatbot interface on their device to enter questions about cleaning methods and appropriate tools. For example, they might type, "How do I clean the drain?"

[1744] Step 2: Submit your question

[1745] terminal

[1746] The entered questions are sent to the server in real time.

[1747] Step 3: Analyzing the Question

[1748] server

[1749] The received question is analyzed using a natural language processing model. The intent of the question is identified, and an appropriate answer is generated.

[1750] Step 4: Generating the answer

[1751] server

[1752] Generate specific cleaning methods and procedures in response to a question. For example, it might generate an answer like, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1753] Step 5: Submit your response

[1754] server

[1755] The generated response is sent to the user's device.

[1756] Step 6: Display the answer

[1757] terminal

[1758] The received response is displayed in the chatbot interface. The user can then use this information to perform specific cleaning steps.

[1759] Embedding an emotion engine

[1760] Step 1: Obtain user sentiment data

[1761] server

[1762] The server acquires the user's voice and video data. This includes data from smartphones, smart speakers, or cameras.

[1763] Step 2: Sentiment Analysis

[1764] server

[1765] The emotion engine analyzes acquired audio and video data to determine the user's emotional state in real time. Emotional states range widely, including stress, joy, and surprise.

[1766] Step 3: Utilizing emotional data

[1767] server

[1768] The emotion engine determines the user's emotional state and adjusts the content and wording of notification messages and responses accordingly. For example, if a user is stressed, the notification message might be made more friendly and include words of encouragement.

[1769] Specific example

[1770] server

[1771] The system detects "fatigue" from the user's tone of voice and facial expression. In this case, it generates a gentle notification message such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[1772] User

[1773] If a user asks "How do I clean the drain?" and the emotion engine detects "interest," the server will provide specific instructions along with an encouraging message such as "It's easy to clean! First, let's remove the drain cover."

[1774] This processing flow allows the present invention to provide personalized support tailored to the user's emotional state, thereby increasing cleaning efficiency and improving the user experience.

[1775] (Example 2)

[1776] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1777] Traditional home cleaning systems have limited automation and have failed to completely alleviate the user's burden. Furthermore, existing systems did not consider the user's emotional state and only offered a single notification method, resulting in a suboptimal user experience. This could potentially increase user stress.

[1778] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1779] In this invention, the server includes video acquisition means, video preprocessing means, AI analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, chatbot means, and emotion analysis means. This enables everything from automatic detection of dirt and pests to personalized notifications and responses tailored to the user's emotional state.

[1780] "Video acquisition means" refers to a means of receiving video data in real time from a video device installed in a home.

[1781] "Video preprocessing means" refers to means for removing noise from received video data, converting the resolution, and converting the color space.

[1782] "AI analysis means" refers to a method that analyzes pre-processed video data and uses a generative AI model to detect specific objects such as dirt or pests.

[1783] The "analysis result recording means" is a means for recording the detection results and confidence scores obtained by the AI ​​analysis means in a database.

[1784] A "notification generation means" is a means for generating notification messages for the user based on the analysis results.

[1785] "Notification transmission means" refers to a means for sending the generated notification message to the user's terminal.

[1786] "User interface means" refers to means that provide an interface for users to access a system and receive notifications and responses.

[1787] A "chatbot" is a means of automatically generating appropriate answers to user questions and providing those answers.

[1788] "Emotion analysis means" refers to a method for analyzing a user's voice data and video data to determine the user's emotional state in real time.

[1789] This invention is a home cleaning assistant system that incorporates an emotion engine that recognizes the user's emotions. This system operates through the cooperation of three entities: a server, a terminal, and the user.

[1790] The server receives video data in real time from multiple video devices (e.g., cameras) installed in the home. The video data is acquired in stream format, and then noise reduction, resolution conversion, and color space conversion are performed by video preprocessing. This utilizes technologies such as Gaussian filters and the OpenCV library.

[1791] The pre-processed video data is input into an AI analysis system to detect dirt and pests. Here, a convolutional neural network (CNN)-based generative AI model is used to identify dirt and pests. Implementation using the TensorFlow library is preferable.

[1792] Detection results and confidence scores are recorded in a database by an analysis result recording device. The record includes timestamps and location information of the video device, and utilizes a MySQL database. The server periodically checks the database, and if the confidence score of the latest detection result exceeds a certain threshold, an appropriate notification message is created by a notification generation device.

[1793] The notification generation mechanism creates a notification with content such as, "The kitchen drain is dirty. It's time to clean it," and sends it to the user's device via the notification sending mechanism. In this case, Firebase Cloud Messaging (FCM) is suitable. The device receives this notification in real time and displays it to the user.

[1794] Users can access the system through a terminal and input questions about cleaning methods and appropriate tools. Questions such as "How do I clean the drain?" are sent to the server in real time via the user interface. The server analyzes these questions using a natural language processing model (e.g., BERT) and generates appropriate answers. For example, it might provide specific cleaning instructions such as, "To clean the drain, do the following: 1. Remove the drain cover. 2. Remove dirt with a brush. 3. Wash with disinfectant."

[1795] The system incorporates emotion analysis capabilities, analyzing the user's voice and video data to determine their emotional state in real time. It uses the Google Cloud Speech-to-Text API for voice analysis and the Emotion API for emotion determination. The system adjusts the content and wording of notification messages according to the user's emotional state. For example, if the user is fatigued, it might generate a message including a helpful suggestion such as, "You seem tired today. Shall we postpone cleaning the drain for a while?"

[1796] As a concrete example, consider a scenario where the server generates a notification message saying, "You seem tired today. Shall we postpone cleaning the drain for a while?" An example of the prompt text is as follows:

[1797] Example of a prompt:

[1798] "We are informing you of the current situation: The camera has detected dirt in the kitchen drain. However, the user is fatigued, so please include a gentle suggestion in the notification message."

[1799] Thus, the cleaning assistant system of the present invention provides personalized support according to the user's emotional state, improving cleaning efficiency and making the user experience more comfortable.

[1800] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1801] Step 1: Acquire camera footage

[1802] server

[1803] The server receives video data in real time from multiple cameras installed within the home. The cameras transmit data in a 30 frames per second (FPS) stream format. The received video data is encoded in H.264 format. The input is video data from each camera, and the output is video data in stream format.

[1804] Specific operation: Video data transmitted from each camera is received in real time by a dedicated receiving module.

[1805] Step 2: Preprocessing of video data

[1806] server

[1807] The server preprocesses the received video data by denoising, converting the resolution, and converting the color space. A Gaussian filter is used for denoising, and the resolution conversion involves downsampling from 1080p to 720p. Color space conversion is performed from RGB to grayscale. The input is video data in stream format, and the output is preprocessed video data.

[1808] Specific operation: Using the OpenCV library, a series of preprocessing algorithms are applied to the received video data.

[1809] Step 3: AI-powered detection of dirt and pests

[1810] server

[1811] The server inputs pre-processed video data into a convolutional neural network (CNN)-based generative AI model to detect dirt and pests. During this process, features are extracted from each frame and input into the AI ​​model. The input is pre-processed video data, and the output is the detection result and confidence score.

[1812] Specific operation: Using the TensorFlow library, a pre-trained AI model is applied to detect the presence or absence of dirt or pests within 0.1 seconds for each frame.

[1813] Step 4: Record the detection results

[1814] server

[1815] The server records detection results and confidence scores from the AI ​​model in a database. Recorded items include timestamps and camera location information. The input is the detection results and confidence scores, and the output is the recorded data stored in the database.

[1816] Specific operation: Connect to the MySQL database and insert each detection result into a dedicated table.

[1817] Step 5: Generate Notifications

[1818] server

[1819] The server periodically checks the database for the latest detection results, and if the confidence score exceeds a certain threshold, it determines that the area requires cleaning or attention. It then generates an appropriate notification message. The input is the detection results from the database, and the output is the notification message.

[1820] Specific action: Generate a notification with a message like, "The kitchen drain is dirty. It's time to clean it."

[1821] Step 6: Send the notification

[1822] server

[1823] The server sends the generated notification message to the user's device. Firebase Cloud Messaging (FCM) is used for push notifications. The input is the notification message, and the output is the notification sent to the user's device.

[1824] Specific operation: Use the FCM API to send notifications to the user's smartphone or tablet.

[1825] Step 7: Responding to user inquiries

[1826] User

[1827] Users access the system through a terminal and input questions about cleaning methods and appropriate tools. The input is the user's question, and the output is the question data.

[1828] terminal

[1829] The terminal sends user questions to the server in real time. The input is the user's question, and the output is the question data sent to the server.

[1830] server

[1831] The server analyzes the user's question using a natural language processing model (e.g., BERT) and generates an appropriate answer. The input is the question data, and the output is the answer message.

[1832] Specific operation: The Transformers library is used for natural language processing, and an algorithm is applied to generate appropriate answers to questions.

[1833] Step 8: Adjusting Sentiment Analysis and Notifications

[1834] server

[1835] The server uses emotion analysis tools to analyze the user's voice and video data and determine the user's emotional state in real time. Based on the emotional state, it adjusts the content and wording of notification messages. The input is the user's voice and video data, and the output is the adjusted notification message.

[1836] Specific operation: The Google Cloud Speech-to-Text API is used for speech analysis, and the Emotion API is used for emotion detection, dynamically changing the wording of notification messages.

[1837] In this way, the entire system works together to provide personalized support to the user, improving cleaning efficiency and making the user experience more comfortable.

[1838] (Application Example 2)

[1839] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1840] In existing brick-and-mortar stores, providing personalized service tailored to customers' emotional states is difficult, often resulting in a uniform service quality and consequently lower customer satisfaction. Furthermore, there is no system in place to recognize customers' emotions in real time from their facial expressions and voices and respond appropriately based on that information. As a result, the quality of the customer experience remains stagnant.

[1841] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means, personalized notification means, and video acquisition means. This makes it possible to recognize the customer's emotional state in real time and send a personalized notification message to store staff accordingly.

[1842] "Emotion recognition means" refers to a function that analyzes audio and video data to determine the user's emotional state in real time.

[1843] A "personalized notification method" is a function that adjusts the content and expression of notification messages based on the user's emotional state as determined by the emotion recognition method.

[1844] "Video acquisition means" refers to a camera device and a function that periodically receives video data in a stream format.

[1845] The "video analysis means" refers to a function that preprocesses received video data and uses an AI model to detect dirt and pests.

[1846] The "analysis result recording means" is a function that records the results of video analysis and stores them in a database that includes timestamps and location information.

[1847] The "notification generation means" is a function that generates an appropriate notification message based on the analysis results.

[1848] "Notification transmission means" refers to a function that sends the generated notification message to the user interface means.

[1849] A "user interface means" is an interface for exchanging information between a user and a system.

[1850] A "chatbot" is a function that receives questions from users, generates appropriate answers, and sends responses.

[1851] The present invention is a system that recognizes customer emotions in a physical store and provides personalized responses based on those emotions. This system includes emotion recognition means, personalized notification means, video acquisition means, video analysis means, analysis result recording means, notification generation means, notification transmission means, user interface means, and chatbot means.

[1852] Hardware configuration

[1853] Smart glasses: Store staff wear them, and a camera device collects video data of customers.

[1854] Server: Processes video and audio data and generates notification messages based on the analysis results.

[1855] Display interface: Integrated into smart glasses, it displays notification messages.

[1856] Software Configuration

[1857] OpenCV: Used for preprocessing and analyzing video data.

[1858] DeepFace: An algorithm for facial recognition and emotion analysis.

[1859] Natural language processing model: Analyzes user questions and generates appropriate answers.

[1860] Database: Used to record analysis results.

[1861] Data processing and data calculation

[1862] 1. Video acquisition method: The camera device of the smart glasses collects video data in real time and transmits it to the server.

[1863] 2. Video analysis means: The server preprocesses the received video data (noise reduction, resolution conversion, etc.) and uses DeepFace to analyze the customer's emotions.

[1864] 3. Means for recording analysis results: The results of face recognition and emotion analysis are recorded in a database along with timestamps and location information.

[1865] 4. Emotion Recognition Method: Determine the customer's emotional state (e.g., anger, joy, surprise, etc.) in real time.

[1866] 5. Personalized notification means: Based on the customer's emotions determined by the emotion recognition means, an appropriate notification message is generated.

[1867] 6. Notification generation means: Creates a personalized notification message and displays it on the smart glasses' display via a notification sending means.

[1868] Adding specific examples

[1869] For example, if the server detects a customer's anger from the video data it receives, the personalized notification system will generate a message such as, "This customer is irritated. Please respond calmly," and display it on the smart glasses' screen. This allows store staff to adjust their customer service to match the customer's emotional state.

[1870] Examples of prompts to input into a generative AI model:

[1871] Prompt: Video frame showing the customer's face

[1872] Output: Analyzes customer emotions in real time and generates emotion-based response notification messages.

[1873] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1874] Step 1:

[1875] Acquisition of video data

[1876] A camera device built into the smart glasses collects video data of customers in the store in real time. The video data is transmitted from the smart glasses to the server in stream format. In this step, the input is images of customers in the store, and the output is the video data distributed to the server.

[1877] Step 2:

[1878] Preprocessing of video data

[1879] The server performs preprocessing on the received video data, including noise reduction, resolution conversion, and color space conversion. This process is carried out using OpenCV. The input is video data in stream format, and the output is preprocessed video frames.

[1880] Step 3:

[1881] emotion recognition

[1882] The server inputs pre-processed video frames into DeepFace for face recognition and emotion analysis. The input is pre-processed video frames, and the output is the customer's emotion (e.g., anger, joy, surprise) and its confidence score.

[1883] Step 4:

[1884] Record of results

[1885] The analysis results are stored in a database. The records include analyzed emotions, timestamps, and camera location information. The input is the emotion recognition result and its associated information, and the output is a database entry.

[1886] Step 5:

[1887] Generating notification messages

[1888] Based on the customer's emotions as determined by the emotion recognition system, an appropriate notification message is generated. The input is data from emotion recognition, and the output is a notification message. For example, if the customer is showing anger, the message "This customer is irritated. Please respond calmly." will be generated.

[1889] Step 6:

[1890] Sending notification messages

[1891] The generated notification message is sent to the smart glasses' display via the notification sending device. The input is the notification message, and the output is the text displayed on the smart glasses' display.

[1892] Step 7:

[1893] Customer support

[1894] Upon receiving the notification, store staff will respond to the customer according to the message displayed on the smart glasses' screen. The input is the notification message displayed on the smart glasses, and the output is personalized customer service.

[1895] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1896] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1897] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1898] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1899] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1900] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1901] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1902] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1903] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1904] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1905] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1906] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1907] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1908] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1909] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1910] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1911] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1912] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1913] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1914] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1915] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1916] The following is further disclosed regarding the embodiments described above.

[1917] (Claim 1)

[1918] Means of acquiring video,

[1919] Video analysis means,

[1920] means for recording analysis results,

[1921] Notification generation means,

[1922] Notification sending method,

[1923] User interface means,

[1924] Chatbot methods and

[1925] A system that includes this.

[1926] (Claim 2)

[1927] The system according to claim 1, wherein the video acquisition means includes a camera device that periodically receives video data, and the video data is received in stream format.

[1928] (Claim 3)

[1929] The system according to claim 1, characterized in that the video analysis means preprocesses the received video data and uses an AI model for detecting dirt and pests.

[1930] (Claim 4)

[1931] The system according to claim 1, wherein the analysis result recording means records the detection result in a database along with a confidence score, and includes a timestamp and camera location information.

[1932] (Claim 5)

[1933] The notification generation means is characterized by extracting areas that require cleaning or attention when the confidence score exceeds a certain threshold, and generating an appropriate notification message, as described in claim 1.

[1934] (Claim 6)

[1935] The system according to claim 1, characterized in that the notification sending means sends the generated notification message to the user's terminal and displays it in real time as a push notification or in-app message.

[1936] (Claim 7)

[1937] The system according to claim 1, characterized in that the chatbot means receives a question from a user, analyzes it using a natural language processing model, generates a specific answer, and displays it via a user interface means.

[1938] "Example 1"

[1939] (Claim 1)

[1940] Means of acquiring video,

[1941] A means of pre-processing video data,

[1942] Methods of analysis using AI models,

[1943] A means of recording the analysis results,

[1944] A means for generating a notification message,

[1945] Means for sending notification messages,

[1946] User interface means,

[1947] Chatbot methods and

[1948] A system that includes this.

[1949] (Claim 2)

[1950] The system according to claim 1, wherein the video acquisition means includes a shooting device that periodically receives video data, and the video data is received in stream format.

[1951] (Claim 3)

[1952] The system according to claim 1 is characterized in that the means for preprocessing video data includes processes for noise reduction, resolution conversion, and color space conversion, and an AI model is used to detect dirt and pests.

[1953] "Application Example 1"

[1954] (Claim 1)

[1955] Means of acquiring video,

[1956] Video analysis means,

[1957] means for recording analysis results,

[1958] Notification generation means,

[1959] Notification sending method,

[1960] User interface means,

[1961] Chatbot methods and

[1962] Suspicious behavior detection means,

[1963] Means of proposing safety measures,

[1964] A system that includes this.

[1965] (Claim 2)

[1966] The system according to claim 1, wherein the video acquisition means includes a camera device that periodically receives video data, and the video data is received in stream format.

[1967] (Claim 3)

[1968] The system according to claim 1, characterized in that the video analysis means preprocesses the received video data and uses an AI model to detect dirt, pests, and suspicious behavior.

[1969] "Example 2 of combining an emotion engine"

[1970] (Claim 1)

[1971] Means of acquiring video,

[1972] Image preprocessing means,

[1973] AI analysis method,

[1974] means for recording analysis results,

[1975] Notification generation means,

[1976] Notification sending method,

[1977] User interface means,

[1978] Chatbot methods and

[1979] Emotion analysis methods,

[1980] A system that includes this.

[1981] (Claim 2)

[1982] The system according to claim 1, wherein the video acquisition means includes a video device that periodically receives video data, and the video data is received in stream format.

[1983] (Claim 3)

[1984] The system according to claim 1, characterized in that the video preprocessing means processes the received video data by noise reduction, resolution conversion, and color space conversion.

[1985] (Claim 4)

[1986] The system according to claim 1, characterized in that the AI ​​analysis means analyzes received video data and uses a generative AI model for detecting dirt and pests.

[1987] (Claim 5)

[1988] The system according to claim 1, characterized in that the notification generation means generates a notification message for the user based on the analysis results, and the emotion analysis means adjusts the content and expression of the notification message according to the user's emotional state.

[1989] (Claim 6)

[1990] The system according to claim 1, characterized in that the chatbot means generates an appropriate answer based on the user's question and provides the answer through the user interface means.

[1991] "Application example 2 when combining with an emotional engine"

[1992] (Claim 1)

[1993] Means of acquiring video,

[1994] Video analysis means,

[1995] means for recording analysis results,

[1996] Notification generation means,

[1997] Notification sending method,

[1998] User interface means,

[1999] Chatbot methods and

[2000] Means of recognizing emotions,

[2001] Personalized notification methods,

[2002] A system that includes this.

[2003] (Claim 2)

[2004] The system according to claim 1, wherein the video acquisition means includes a camera device that periodically receives video data, and the video data is received in stream format.

[2005] (Claim 3)

[2006] The system according to claim 1, characterized in that the video analysis means preprocesses the received video data and uses an AI model for detecting dirt and pests.

[2007] (Claim 4)

[2008] The system according to claim 1, characterized in that the emotion recognition means analyzes audio data and video data to determine the user's emotional state in real time.

[2009] (Claim 5)

[2010] The system according to claim 1, characterized in that the personalized notification means adjusts the content and expression of the notification message based on the user's emotional state determined by the emotion recognition means. [Explanation of symbols]

[2011] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

[Claim 1] Includes a camera device that periodically receives video data, and video acquisition means that receives the video data in stream format, A video analysis method that preprocesses received video data and uses an AI model to detect dirt and pests, The detection results are recorded in a database along with a confidence score, and the analysis results recording means includes a timestamp and camera location information. A notification generation means that extracts areas requiring cleaning or attention when the confidence score exceeds a certain threshold and generates an appropriate notification message, A notification sending means that sends the generated notification message to the user's terminal, A user interface means for displaying in real time as a push notification or in-app message, A chatbot that receives questions from users, analyzes them using a natural language processing model, generates specific answers, and displays them via a user interface. A system that includes this.

Images