system

Abstract

We provide the system. [Solution] An image capture means for detecting the discarded waste, An image transmission means for sending image data of the above-mentioned discarded waste to a cloud server, A generating artificial intelligence means for classifying waste by analyzing the above image data, A matching means for comparing the above analysis results with a waste identification database of a domestic municipality, A feedback mechanism that provides feedback to the user based on the above matching results, An alert mechanism that issues an alert when it detects an error in the feedback provided by the above-mentioned feedback mechanism, A system equipped with an interactive concierge service that provides real-time guidance to users.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern aging societies, elderly people with impaired cognitive functions often misclassify garbage, which frequently causes problems in waste disposal in local communities. There is a need for a mechanism to prevent such problems and support elderly people in accurately classifying garbage. In addition, it is necessary for family members and caregivers living separately to remotely monitor the garbage classification of the elderly and be able to intervene as needed.

Means for Solving the Problems

[0005] This invention detects waste deposited in a trash can using an image capture device attached to the trash can and transmits the image data to a cloud server. An artificial intelligence generator on the cloud server analyzes the image data to identify the type of waste. The analysis results are then compared with a waste identification database of local municipalities to provide the user with accurate waste sorting feedback. Furthermore, if incorrect sorting is detected, an alert device immediately issues an alert, and an interactive concierge device guides the user in real time on the correct sorting method. The system also has a function to measure the accumulated amount of waste and predict the next collection date, which is communicated to the user. Additionally, remote monitoring devices allow family members or caregivers living far away to monitor the elderly person's waste sorting status in real time. These measures can solve the problem of incorrect waste sorting by the elderly and improve the efficiency and accuracy of waste management for society as a whole.

[0006] "Discarded waste" refers to garbage, unwanted items, and recyclable materials that have been placed in trash cans or waste containers.

[0007] "Image capture means" refers to a device or system that uses a camera, sensor, or the like to take images of the waste that has been submitted.

[0008] "Image transmission means" refers to a device or system that transmits captured image data to a cloud server or other devices on the network.

[0009] A "cloud server" refers to a remote server that stores, processes, and manages data via the internet.

[0010] "Generative artificial intelligence means" refers to an artificial intelligence system that uses technologies such as machine learning and deep learning to analyze image data and perform specific tasks.

[0011] The "Domestic Municipal Waste Identification Database" refers to a database that compiles the waste sorting rules established by each municipality in Japan.

[0012] "Verification means" refers to a device or system that compares the analysis results of a generating artificial intelligence means with data from a domestic municipal waste identification database.

[0013] "Feedback means" refers to a device or system that provides information to the user based on the matching results.

[0014] An "alert device" refers to a device or system that issues a warning when incorrect waste sorting is detected.

[0015] "Interactive concierge service" refers to a device or system that provides guidance and support through interactive dialogue with the user.

[0016] "Accumulation amount measuring means" refers to a device or system for measuring the amount of waste in a trash can.

[0017] "Prediction means" refers to a device or system that predicts the next collection date based on data from the accumulation amount measurement means.

[0018] "Notification means" refers to a device or system that notifies the user of prediction results or other important information.

[0019] "Remote monitoring means" refers to a device or system that remotely monitors the status of waste input and other data. [Brief explanation of the drawing]

[0020] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4]It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0021] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0024] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0025] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0026] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0028] [First Embodiment]

[0029] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0030] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0033] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0036] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0040] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0041] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. The details of the system's program processing and specific examples are described below.

[0042] System Configuration

[0043] 1. Device (trash can camera and sensors)

[0044] It is equipped with a sensor that detects when trash is dropped in and a camera that captures images of the dropped-in trash.

[0045] It has the function to send captured image data to a cloud server.

[0046] 2. Cloud Server

[0047] The system receives image data and uses artificial intelligence (AI) to identify the type of waste.

[0048] The analysis results are compared with the waste identification database of local governments in Japan, and feedback is provided to users.

[0049] If an incorrect sorting is detected, an alert will be issued and an interactive concierge will be activated.

[0050] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[0051] 3. Users (using a dedicated application)

[0052] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[0053] It is possible to monitor the status of garbage disposal remotely.

[0054] Program processing

[0055] This program will be executed in the following steps.

[0056] 1. Detecting and capturing images of trash being dropped off.

[0057] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[0058] The device sends the captured image data to the cloud server.

[0059] 2. Image Analysis

[0060] The server receives image data and passes it to the artificial intelligence model for generating data.

[0061] An AI model analyzes the image and identifies the type of waste.

[0062] 3. Matching and Feedback

[0063] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0064] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[0065] 4. Alerts and interactive concierge

[0066] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[0067] 5. Prediction of accumulated amount and next collection date

[0068] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[0069] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[0070] 6. Remote Monitoring

[0071] Users can monitor the status of their trash collection in real time through a dedicated application.

[0072] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[0073] Specific example

[0074] 1. In the case of correct sorting

[0075] The user throws the plastic bottle into the trash can.

[0076] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[0077] The server analyzes the image and identifies it as plastic.

[0078] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[0079] 2. In the case of incorrect sorting

[0080] A user threw a metal can into a trash can, but the cloud server analysis and comparison results detected an error (for example, classifying it as combustible waste).

[0081] An alert system issues an alert, and an interactive concierge guides the user on the correct way to sort metal cans for disposal.

[0082] This invention will improve the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, it will enable family members and caregivers living separately to monitor waste sorting status in real time and provide appropriate support.

[0083] The following describes the processing flow.

[0084] Step 1:

[0085] Terminal: "Detects the dropping off of trash"

[0086] A sensor installed in the trash can detects when trash is placed inside.

[0087] When trash is detected being dropped in, the sensor activates the camera.

[0088] Step 2:

[0089] Device: "Capture image"

[0090] The camera captures an image of the trash that has been dropped off.

[0091] The captured image data is temporarily stored inside the device.

[0092] Step 3:

[0093] Terminal: "Send image data to the cloud server"

[0094] The device sends the captured image data to the cloud server.

[0095] Once the transmission is complete, the device waits for a response from the cloud server.

[0096] Step 4:

[0097] Server: "Receive image data"

[0098] The cloud server receives the image data.

[0099] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[0100] Step 5:

[0101] Server: "Perform image analysis"

[0102] The AI ​​model analyzes the image data.

[0103] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[0104] Step 6:

[0105] Server: "Compare analysis results with the waste identification database."

[0106] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0107] Through verification, we determine whether the waste has been sorted appropriately.

[0108] Step 7:

[0109] Server: "Provide feedback to users"

[0110] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[0111] Feedback is sent to users through a dedicated application.

[0112] Step 8:

[0113] Device: "Receive and display feedback"

[0114] The terminal receives feedback from the server.

[0115] Display the received feedback to the user.

[0116] Step 9:

[0117] Server: "Issues an alert in case of incorrect sorting."

[0118] If incorrect classification is detected, the server will immediately issue an alert.

[0119] The alert will point out the error to the user and provide the correct sorting method.

[0120] Step 10:

[0121] Terminal: "Launch the interactive concierge"

[0122] When an alert is issued, the device activates its interactive concierge function.

[0123] The concierge function provides users with guidance and support in an interactive format.

[0124] Step 11:

[0125] Terminal: "Measure accumulated amount"

[0126] A sensor inside the trash can measures the amount of waste accumulated.

[0127] The measured data is sent to the cloud server.

[0128] Step 12:

[0129] Server: "Predict the next collection date"

[0130] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[0131] The prediction results are notified to the user through a dedicated application.

[0132] Step 13:

[0133] User: "Perform remote monitoring"

[0134] A dedicated application is used to monitor the status of garbage disposal in real time.

[0135] Users can assist elderly individuals with sorting waste as needed.

[0136] The above describes the specific flow of program processing in the system of the present invention.

[0137] (Example 1)

[0138] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0139] Proper waste sorting within the home is difficult for many people, and mistakes are frequent, especially among the elderly and busy households. As a result, the efficiency and accuracy of waste disposal decline, making it difficult for local governments to collect and recycle waste. On the other hand, there are limited ways for family members living far away to support elderly people with waste sorting. There is a need to solve this problem and improve the accuracy of waste sorting.

[0140] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0141] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for sending the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a storage amount measuring means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. This improves the accuracy of waste sorting, streamlines waste disposal, and enables support from remote locations.

[0142] A "sensor means" refers to a device or function used to detect the dropping of trash.

[0143] "Camera means" refers to a photographic device or function used to capture images of the trash that has been dropped off.

[0144] "Image transmission means" refers to a device or function for transmitting captured image data to a cloud server.

[0145] "Generative artificial intelligence means" refers to AI technology and functions used to analyze image data and classify waste.

[0146] "Verification means" refers to a device or function for verifying analysis results against a waste identification database.

[0147] A "feedback mechanism" refers to a device or function that provides feedback to the user based on the analysis results.

[0148] An "alert mechanism" is a device or function that issues an alert when an error in the feedback is detected.

[0149] "Interactive concierge services" refer to conversational systems and functions that provide real-time guidance to users.

[0150] "Accumulation amount measuring means" refers to a device or function for measuring the accumulated amount of waste.

[0151] A "prediction tool" is a function for predicting the next collection date based on the measured amount of accumulated waste.

[0152] "Notification means" refers to devices or functions used to notify the user of the prediction results.

[0153] "Remote monitoring means" refers to devices or functions that allow users to remotely monitor the status of waste disposal.

[0154] This invention provides an IoT system that assists with household waste sorting, and in particular, provides a configuration for achieving accurate waste classification and sorting. The system detects the waste that is dropped off, analyzes its image to identify the type of waste, and notifies the user of the correct sorting method.

[0155] System Configuration

[0156] This system includes the following main components and means:

[0157] 1. Device (trash can sensor and camera)

[0158] Sensors: Installed to detect the dropping of trash. Infrared sensors and ultrasonic sensors are used.

[0159] Camera method: Capture images of the dropped-off trash. A high-resolution camera is used to photograph the dropped-off trash.

[0160] 2. Cloud Server

[0161] Image transmission method: Captured image data is sent to a cloud server. This utilizes a Wi-Fi module and transmits data using the HTTP protocol.

[0162] Generative AI method: Analyzes received image data to identify the type of waste. For this purpose, AI technologies such as the Google Cloud Vision API are used.

[0163] Verification method: The analysis results are compared with a waste identification database. Based on the verification results, the correct waste sorting method is confirmed.

[0164] Feedback method: Notify the user of the feedback results. For example, use push notifications to notify a dedicated application.

[0165] Alerting method: An alert is issued when an error in feedback is detected. The alert is sent to the user via push notification.

[0166] Interactive concierge system: A dialogue system that provides real-time guidance to users. This may utilize services such as Amazon Lex.

[0167] 3. Managing the amount of trash in the trash cans and the collection schedule.

[0168] Accumulation amount measurement method: Measures the amount of accumulated waste in the trash can. The amount of waste is measured using an ultrasonic sensor or similar device.

[0169] Prediction method: Predict the next collection date based on the accumulated amount and the local government's collection schedule. Based on the prediction results, determine the next collection date.

[0170] Notification method: Notify the user of the prediction results. For example, send a notification such as, "The next collection date is in 2 days."

[0171] 4. User Interface

[0172] Remote monitoring method: A dedicated application allows users to monitor the status of garbage disposal in real time. Family members and caregivers can also assist elderly individuals with their garbage sorting.

[0173] Specific example

[0174] In the case of correct sorting

[0175] The user throws the plastic bottle into the trash can.

[0176] The device's sensors detect the mail being dropped in, and the camera captures an image.

[0177] The device sends image data to the cloud server.

[0178] The server analyzes the image and identifies it as plastic.

[0179] If the matching result is correct, the server notifies the user with "Correct classification."

[0180] In case of incorrect sorting

[0181] The user throws the metal can into the trash can.

[0182] The device's sensors detect the mail being dropped in, and the camera captures an image.

[0183] The device sends image data to the cloud server.

[0184] The server analyzes the image to identify incorrect sorting (for example, classifying it as combustible waste).

[0185] The server issues an alert, activates an interactive concierge, and guides the user through the correct sorting method.

[0186] Example of a prompt

[0187] 1. Scenario: Notifying people about the correct sorting of plastic bottles.

[0188] "A plastic bottle has been placed in the trash can. Please ensure proper sorting."

[0189] 2. Scenario: Notification of incorrect sorting of metal cans.

[0190] "A metal can was placed in the trash can, but it was detected as incorrectly sorted. Please provide instructions on the correct sorting method."

[0191] With the above configuration, this system can improve the accuracy of waste sorting and provide users with quick and appropriate support.

[0192] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0193] Detailed explanation of the system's program processing.

[0194] Step 1: Detecting trash and capturing images

[0195] Input: Garbage is placed in the trash can.

[0196] Operation: The device's sensors detect the placement of trash. These sensors use infrared or ultrasonic sensors. When trash is detected, the device's camera automatically captures an image. The image is saved in high-resolution JPEG format and a timestamp is added.

[0197] Output: Captured image data of the trash.

[0198] Step 2: Sending image data to the cloud

[0199] Input: Image data of trash captured on a device.

[0200] Operation: The device sends captured image data to the cloud server. This data is transmitted via the HTTP protocol using the Wi-Fi module. The data is then uploaded to the cloud server via a REST API.

[0201] Output: Image data stored on a cloud server.

[0202] Step 3: Image Analysis

[0203] Input: Image data stored on a cloud server.

[0204] Operation: The server receives image data and passes it to an artificial intelligence model. This AI model uses the Google Cloud Vision API to analyze objects in the image and identify the type of waste. The analysis uses information such as shape, color, and text.

[0205] Output: Type of waste (e.g., plastic, metal, paper, etc.).

[0206] Step 4: Matching and Feedback

[0207] Input: The type of waste analyzed.

[0208] Operation: The server compares the analysis results with the waste identification database of domestic municipalities. A REST API is used to communicate with the database and verify the accuracy of the analysis results.

[0209] If the matching results are accurate, the server uses a dedicated application to send feedback to the user about the correct sorting.

[0210] Output: Feedback notification such as "Correct sorting."

[0211] Step 5: Alerts and Interactive Concierge

[0212] Input: Matching result when an incorrect classification is detected.

[0213] Operation: When the server detects incorrect sorting, it issues an alert. The alert is pushed to a dedicated application. Additionally, an interactive concierge is activated to teach the user the correct sorting method. Amazon Lex is used for the interactive concierge, providing support via voice and text.

[0214] Output: Alert notifications such as "Incorrect sorting detected," and guidance on sorting methods from an interactive concierge.

[0215] Step 6: Measure the amount of trash in the bins and predict the next collection date.

[0216] Input: Data on the amount of waste bin contents.

[0217] Operation: The device's sensors measure the amount of trash accumulated in the trash can. An ultrasonic sensor is used for this, and the data is sent to a cloud server. The server uses this information to compare it with the local government's collection schedule and predict the next collection date.

[0218] Output: Notification of prediction results such as "The next collection date is in 2 days."

[0219] Step 7: Remote Monitoring

[0220] Input: Real-time data on trash disposal and trash can contents.

[0221] Operation: Users monitor the status of their garbage disposal in real time through a dedicated application. The application displays data from a cloud server and allows users to check past disposal history. In addition, family members and caregivers can remotely assist elderly individuals with their garbage sorting.

[0222] Output: Real-time monitoring data display and notification.

[0223] The above outlines the specific steps of the system's program processing and the detailed operation of each step. This system allows users to obtain accurate waste sorting information in real time, improving the efficiency and accuracy of waste disposal.

[0224] (Application Example 1)

[0225] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0226] Sorting waste generated on factory production lines efficiently and accurately is difficult. Traditional manual waste sorting is prone to human error, and further automation and real-time monitoring are needed. In addition, a convenient display system is required so that workers can constantly check the status of waste sorting.

[0227] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0228] In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a display means for displaying the waste deposit status in real time on a display device worn by the user, and a mechanical means for automating waste deposit. This enables accurate sorting and real-time monitoring of waste.

[0229] "Image capture means" refers to a device that takes pictures of the discarded waste that has been submitted and acquires the image data.

[0230] "Image transmission means" refers to a device or function that transmits acquired image data to a cloud server.

[0231] "Generative artificial intelligence means" refers to an artificial intelligence model that analyzes received image data to identify the type of waste.

[0232] "Verification means" refers to a method of comparing and verifying the analyzed data with the waste identification database of local municipalities within Japan.

[0233] A "feedback method" is a means of notifying the user of the sorting results based on the matching results.

[0234] An "alert system" is a means of issuing a warning when an error in classification is detected.

[0235] An "interactive concierge service" is an interactive system that guides users in real time on the correct sorting methods.

[0236] "Display means" refers to a means of displaying the status of waste disposal in real time on a display device worn by the user.

[0237] "Mechanical means" refers to a device or function for automating the disposal of waste.

[0238] A "storage unit measurement means" is a means of measuring the amount of waste using a sensor.

[0239] A "prediction method" is a means of predicting the next collection date based on the accumulated amount and collection schedule.

[0240] "Notification means" refers to a means of notifying users of information such as the predicted collection date.

[0241] A "remote monitoring method" is a means that allows users to remotely monitor the status of waste disposal.

[0242] This invention is a system for accurately sorting and managing waste generated on a factory production line. The system provides multiple functions, including waste classification, accumulation measurement, collection date prediction, and real-time monitoring.

[0243] System Configuration

[0244] 1. Terminal (waste disposal robot and camera)

[0245] It is equipped with a robot that manages the disposal of waste and uses sensors to detect waste.

[0246] When waste is detected, the camera captures an image and sends that image data to a cloud server.

[0247] 2. Cloud Server

[0248] The received image data is passed to a "generating artificial intelligence means" to identify the type of waste.

[0249] The identified waste types are cross-referenced with the "Domestic Municipal Waste Identification Database."

[0250] Based on the matching results, provide feedback to the user.

[0251] If there are errors in the feedback results, an alert will be issued and an interactive concierge will guide you to the correct solution.

[0252] The system measures the amount of waste accumulated, predicts the next collection date, and notifies the user.

[0253] 3. Users (using smart glasses or dedicated applications)

[0254] You can check the status of your waste disposal in real time through smart glasses.

[0255] Receive notifications and alerts from the cloud server through a dedicated application.

[0256] It is possible to monitor the status of waste disposal remotely.

[0257] Program processing

[0258] The program for this system is processed as follows:

[0259] 1. Waste disposal detection and image capture

[0260] When the device's sensor detects the dropping off of waste, the camera automatically captures an image.

[0261] The acquired image data is sent to a cloud server.

[0262] For example, we will utilize OpenCV as the hardware to be used.

[0263] 2. Image Analysis

[0264] The cloud server receives image data and passes it to a "generative AI model" to identify the type of waste.

[0265] The analysis algorithm uses artificial intelligence models such as TensorFlow and PyTorch.

[0266] 3. Matching and Feedback

[0267] The analysis results generated by artificial intelligence are compared with the "garbage identification database of domestic municipalities."

[0268] Based on the matching results, feedback is provided to the user.

[0269] Database matching is performed using database management systems such as SQL.

[0270] 4. Alerts and interactive concierge

[0271] If an error in sorting is detected, the server will issue an alert, and an interactive system will guide the user on the correct sorting method.

[0272] The conversational concierge will utilize conversational services such as Google Dialogflow and AWS Lex.

[0273] 5. Measurement of accumulated amount and prediction of the next collection date

[0274] Sensors measure the amount of waste accumulated and transmit that information to a cloud server.

[0275] The server predicts the next collection date based on the accumulated amount and collection schedule, and notifies the user.

[0276] The notification is carried out via a dedicated application or smart glasses.

[0277] Specific example

[0278] For example, when an operator throws metal waste into a bin in a factory, the sensor detects the throwing, and the camera captures an image. This image is sent to a cloud server and recognized as metal by a generated AI model. Then, this recognition result is compared with the waste classification database of domestic local governments, and the correct classification method is confirmed. The operator is notified through smart glasses that "this is the correct classification".

[0279] Example of prompt text

[0280] "This is an image of the captured waste. Determine the type of waste and select one from the following categories: [combustible waste, non-combustible waste, plastic, metal, bottle, paper]."

[0281] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[0282] Step 1:

[0283] The terminal detects the throwing of waste with a sensor. The sensor detects that the waste has been discarded into the trash can and notifies the terminal of the event. Based on this information, the camera is activated to capture an image. The input is the waste throwing event, and the output is the captured image data.

[0284] Step 2:

[0285] The terminal sends the captured image data to the cloud server. The image data is uploaded to the cloud server through the network, and preparations for further analysis are completed. The input is the captured image data, and the output is the transmission of the image data to the cloud server.

[0286] Step 3:

[0287] The server passes the received image data to the artificial intelligence means for generation. The server inputs the image data into the AI model and performs analysis to identify the type of waste. The input is the image data uploaded to the cloud server, and the output is the analyzed waste type information.

[0288] Step 4:

[0289] The server matches the identified waste type with the waste identification database of domestic local governments. The database matching is implemented using a database management system such as SQL. The input is the waste type information analyzed by AI, and the output is the matching result.

[0290] Step 5:

[0291] The server provides feedback to the user based on the matching result. The feedback can be received by the user through smart glasses or a dedicated application. The input is the matching result, and the output is the feedback information to the user.

[0292] Step 6:

[0293] If there is an error in the feedback result, the server issues an alert, and the interactive concierge guides the user to the correct sorting method. Google Dialogflow or AWS Lex, etc. is used for the concierge. The input is the error information of the feedback result, and the output is the alert to the user and the interactive guidance.

[0294] Step 7:

[0295] The sensor measures the accumulation amount of waste and transmits the information to the cloud server. The server predicts the next collection date based on the accumulation amount and the collection schedule and notifies the user. The input is the measured accumulation amount, and the output is the predicted next collection date.

[0296] Step 8:

[0297] The user checks the real-time waste mailing status through smart glasses or a dedicated application. Based on this information, the user can take appropriate waste disposal actions. The input is the waste mailing status, and the output is the mailing information displayed in real time.

[0298] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion recognition model 59 and perform specific processing using the user's emotions.

[0299] The present invention is a system that supports waste sorting using an IoT device attached to a household trash can, and further has a function of recognizing the user's emotions and adjusting feedback. Hereinafter, the details and specific examples of the program processing of this system will be described.

[0300] System Configuration

[0301] 1. Terminal (Camera and sensors of the trash can)

[0302] The terminal has a sensor for detecting the mailing of waste and a camera for capturing an image of the mailed waste. [[ID=2,22]]

[0303] <It has a function of transmitting the captured image data to a cloud server.

[0304] 2. Cloud Server

[0305] <The cloud server receives the image data and identifies the type of waste using a generated artificial intelligence (AI).

[0306] <It collates the analysis results with the waste identification database of local governments in the country and provides feedback to the user.

[0307] <When incorrect sorting is detected, it issues an alert and activates an interactive concierge.

[0308] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[0309] It features an emotion recognition engine that recognizes user emotions and appropriately adjusts feedback and guidance.

[0310] 3. Users (using a dedicated application)

[0311] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[0312] It is possible to monitor the status of garbage disposal remotely.

[0313] Program processing

[0314] This program will be executed in the following steps.

[0315] 1. Detecting and capturing images of trash being dropped off.

[0316] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[0317] The device sends the captured image data to the cloud server.

[0318] 2. Image Analysis

[0319] The server receives image data and passes it to the artificial intelligence model for generating data.

[0320] An AI model analyzes the image and identifies the type of waste.

[0321] 3. Matching and Feedback

[0322] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0323] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[0324] 4. Alerts and interactive concierge

[0325] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[0326] 5. Emotion Recognition and Adaptive Feedback

[0327] The server's emotion recognition engine detects the user's emotional state.

[0328] If a user is feeling stressed or confused, an interactive concierge will provide gentle and courteous guidance.

[0329] 6. Prediction of accumulated amount and next collection date

[0330] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[0331] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[0332] 7. Remote Monitoring

[0333] Users can monitor the status of their trash collection in real time through a dedicated application.

[0334] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[0335] Specific example

[0336] 1. In the case of correct sorting

[0337] The user throws the plastic bottle into the trash can.

[0338] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[0339] The server analyzes the image and identifies it as plastic.

[0340] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[0341] 2. In the case of incorrect sorting

[0342] A user throws a metal can into a trash can, but the cloud server's analysis and matching results detect an error (for example, classifying it as combustible waste).

[0343] The server uses an emotion recognition engine to check the user's emotional state.

[0344] An alert is issued, and an interactive concierge calmly instructs, "This is metal waste. Please sort it correctly."

[0345] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[0346] The following describes the processing flow.

[0347] Step 1:

[0348] Terminal: "Detects the dropping off of trash"

[0349] A sensor installed in the trash can detects when trash is placed inside.

[0350] When trash is detected being dropped in, the sensor activates the camera.

[0351] Step 2:

[0352] Device: "Capture image"

[0353] The camera captures an image of the trash that has been dropped off.

[0354] The captured image data is temporarily stored inside the device.

[0355] Step 3:

[0356] Terminal: "Send image data to the cloud server"

[0357] The device sends the captured image data to the cloud server.

[0358] Once the transmission is complete, the device waits for a response from the cloud server.

[0359] Step 4:

[0360] Server: "Receive image data"

[0361] The cloud server receives the image data.

[0362] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[0363] Step 5:

[0364] Server: "Perform image analysis"

[0365] The AI ​​model analyzes the image data.

[0366] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[0367] Step 6:

[0368] Server: "Compare analysis results with the waste identification database."

[0369] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0370] Through verification, we determine whether the waste has been sorted appropriately.

[0371] Step 7:

[0372] Server: "Provide feedback to users"

[0373] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[0374] Feedback is sent to users through a dedicated application.

[0375] Step 8:

[0376] Device: "Receive and display feedback"

[0377] The terminal receives feedback from the server.

[0378] Display the received feedback to the user.

[0379] Step 9:

[0380] Server: "Recognize emotions"

[0381] The server's emotion recognition engine detects the user's response to the feedback.

[0382] Evaluate whether users are experiencing stress or confusion.

[0383] Step 10:

[0384] Server: "Perform feedback adjustments"

[0385] Based on evaluations by the emotion recognition engine, the interactive concierge system adjusts the feedback content.

[0386] If the user is feeling stressed, provide detailed instructions in a gentle tone.

[0387] Step 11:

[0388] Server: "Issues an alert in case of incorrect sorting."

[0389] If incorrect classification is detected, the server will immediately issue an alert.

[0390] The alert will point out the error to the user and provide the correct sorting method.

[0391] Step 12:

[0392] Terminal: "Launch the interactive concierge"

[0393] When an alert is issued, the device activates its interactive concierge function.

[0394] The concierge function provides users with guidance and support in an interactive format.

[0395] Step 13:

[0396] Terminal: "Measure accumulated amount"

[0397] A sensor inside the trash can measures the amount of waste accumulated.

[0398] The measured data is sent to the cloud server.

[0399] Step 14:

[0400] Server: "Predict the next collection date"

[0401] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[0402] The prediction results are notified to the user through a dedicated application.

[0403] Step 15:

[0404] User: "Perform remote monitoring"

[0405] A dedicated application is used to monitor the status of garbage disposal in real time.

[0406] Users can assist elderly individuals with sorting waste as needed.

[0407] The above describes the specific flow of program processing in the system of the present invention.

[0408] (Example 2)

[0409] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0410] In modern society, waste sorting is a crucial environmental issue, but many people are confused about sorting methods, leading to frequent missorting. Elderly people and those unfamiliar with sorting are particularly prone to stress and confusion during the process. Furthermore, managing the amount of waste accumulated in households and knowing the next collection date is difficult, making timely disposal challenging. Additionally, family members and caregivers living far away have limited means of monitoring waste sorting status. A system is needed to address these challenges and improve the efficiency and accuracy of waste sorting.

[0411] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0412] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for transmitting the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to identify the type of waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an emotion recognition means for detecting the user's emotional state using an emotion recognition engine, an adaptive feedback means for adjusting guidance based on the emotion recognition means, an interactive concierge means for providing guidance to the user in real time, a storage amount measurement means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. As a result, the accuracy of waste sorting is improved, the efficiency of waste processing is increased, user stress is reduced, and sorting work can be carried out smoothly. Furthermore, it becomes easier to provide support because family members or caregivers in remote locations can monitor the mailing status.

[0413] A "sensor device" is a device used to detect waste placed in a trash can.

[0414] A "camera device" is a photographic device used to capture images of the discarded waste that has been submitted.

[0415] "Image transmission means" refers to a communication device for sending captured image data to a cloud server.

[0416] "Generative artificial intelligence means" refers to artificial intelligence technology used to analyze received image data and identify the type of waste.

[0417] "Verification means" refers to a device or software for verifying analysis results against a waste identification database of a domestic municipality.

[0418] A "feedback mechanism" is a device or system that provides the user with the correct sorting method based on the matching results.

[0419] An "alert means" is a device or system that issues an alert when it detects an error in the feedback provided by the feedback means.

[0420] "Emotion recognition means" refers to a device or software that uses an emotion recognition engine to detect the user's emotional state.

[0421] "Adaptive feedback means" refers to a device or system for adjusting guidance based on the emotional state detected by the emotion recognition means.

[0422] An "interactive concierge system" is a dialogue system that provides users with real-time sorting and guidance.

[0423] A "storage volume measuring device" is a sensor device used to measure the amount of waste accumulated.

[0424] "Prediction means" refers to a device or software for predicting the next collection date based on the accumulated amount.

[0425] "Notification means" refers to a device or system for notifying the user of the prediction results.

[0426] "Remote monitoring means" refers to a device or system that allows a user to remotely monitor the status of waste disposal.

[0427] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. Furthermore, it has a function to recognize the user's emotions and adjust the feedback accordingly. The embodiments of this system are described in detail below.

[0428] System Configuration

[0429] 1. Device (trash can camera and sensors)

[0430] A sensor attached to the trash can detects when trash is placed inside.

[0431] When a mail item is detected, the camera automatically captures an image and sends the image data to a cloud server. This device has a built-in communication module that allows it to send data to the cloud server.

[0432] 2. Cloud Server

[0433] The cloud server receives image data sent from the terminal. The received image data is then passed to a generative AI model on the server for analysis.

[0434] The generating AI model analyzes the images and identifies the type of waste. The analysis results are then compared against a waste identification database used by local municipalities in Japan.

[0435] Based on the matching results, users are provided with feedback on correct waste sorting. This feedback is presented to the user via notification or in real time.

[0436] If incorrect sorting is detected, an alert is issued, and an interactive concierge guides you through the correct sorting method.

[0437] The server also uses an emotion recognition engine to detect the user's emotional state. If the user is feeling stressed or confused, an interactive concierge will provide gentle and helpful guidance.

[0438] Sensors measure the amount of trash accumulated in the trash cans, and this data is sent to a cloud server. The cloud server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[0439] 3. User (using a dedicated application)

[0440] Users can receive feedback and alert notifications, as well as guidance from an interactive concierge, through a dedicated application.

[0441] The application allows for remote monitoring of waste disposal status. This enables family members and caregivers to remotely assist elderly individuals with their waste sorting.

[0442] Specific example

[0443] 1. In the case of correct sorting

[0444] The user throws the plastic bottle into the trash can.

[0445] The device's sensor detects the insertion of the item, and the camera captures an image of the inserted plastic bottle. The captured image data is sent to a cloud server.

[0446] The cloud server passes the image data to a generating AI model, which identifies it as a plastic bottle. A verification method confirms the accuracy of the analysis result.

[0447] The user will be notified via a feedback system that "this is the correct sorting."

[0448] 2. In the case of incorrect sorting

[0449] A user throws a metal can into a trash can, but the analysis results incorrectly classify it as combustible waste.

[0450] The cloud server's emotion recognition engine checks the user's emotional state. If the user notices an error and is feeling stressed, an alert is issued.

[0451] Through an interactive concierge system, you'll be gently guided with instructions like, "This is metal waste. Please classify it correctly."

[0452] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[0453] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0454] Step 1:

[0455] Garbage disposal detection and image capture

[0456] The device uses a sensor to detect when waste is placed in the trash can. When a user places waste in the trash can, the sensor detects this action. At that time, the sensor emits a trigger signal indicating "Trash has been placed in the trash can." Subsequently, the camera that receives this trigger signal automatically captures an image. Captured image data is then generated.

[0457] Input: Garbage disposal action

[0458] Output: Image data of the discarded trash

[0459] Step 2:

[0460] Sending image data to a cloud server

[0461] The device sends the captured image data to the cloud server. The captured image data is uploaded to the cloud server via a communication module. During this process, necessary metadata (such as the time of posting) is also added to the image data.

[0462] Input: Captured image data

[0463] Output: Image data uploaded to the cloud server

[0464] Step 3:

[0465] Image analysis

[0466] The server passes the received image data to a generative AI model. The cloud server inputs the received image data into the AI ​​model for analysis. This generative AI model analyzes the images and identifies the type of waste. This analysis yields classification results such as "plastic bottles" or "metal cans."

[0467] Input: Image data uploaded to a cloud server

[0468] Output: Analysis results (type of waste)

[0469] Step 4:

[0470] Matching and Feedback

[0471] The server compares the analysis results with the waste identification database of local municipalities in Japan. The analyzed data is compared with the database to confirm the correct sorting method. A feedback system notifies the user about the correct sorting method. This notification is provided through a dedicated application.

[0472] Input: Analysis results

[0473] Output: Feedback on waste sorting

[0474] Step 5:

[0475] Alerts and interactive concierge

[0476] The server issues an alert if it detects incorrect sorting. When the matching system finds incorrect sorting, it sends a warning to the user through the notification system. Simultaneously, an interactive concierge is activated to guide the user on the correct sorting method.

[0477] Input: Matching results (error detection)

[0478] Output: Alert notifications and interactive concierge feedback

[0479] Step 6:

[0480] Emotion recognition and adaptive feedback

[0481] The server uses an emotion recognition engine to detect the user's emotional state. It analyzes the user's responses to feedback and alerts, and if stress or confusion is detected, the conversational concierge adjusts the tone and content of the feedback.

[0482] Input: User response data

[0483] Output: Feedback adapted to emotional state

[0484] Step 7:

[0485] Accumulated amount and prediction of the next collection date

[0486] A sensor measures the amount of trash in the bins and sends the data to a server. The server predicts the next collection date based on the amount of trash and the local government's collection schedule. The prediction result is then notified to the user through a dedicated application.

[0487] Input: Accumulated data

[0488] Output: Predicted notification of the next collection date

[0489] Step 8:

[0490] Remote monitoring

[0491] Users can monitor the status of their garbage disposal in real time through a dedicated application. The monitoring function within the application allows users and their families to remotely check the disposal status. This function makes it possible to remotely assist elderly people with their garbage sorting.

[0492] Input: Garbage disposal data

[0493] Output: Real-time mailing status display

[0494] (Application Example 2)

[0495] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0496] Traditional waste sorting methods often failed to properly separate waste, resulting in reduced recycling efficiency and potentially exacerbating social and environmental problems. Furthermore, elderly and disabled individuals frequently encountered difficulties with sorting, and there was a lack of support for proper sorting, particularly in public facilities and retail stores. Additionally, managing waste bin levels and timely collection proved challenging, leading to frequent hygiene issues. A solution was needed to address these problems while reducing the burden on users.

[0497] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing guidance to the user in real time, an emotion recognition means for recognizing the user's emotions and adjusting the feedback, a notification means for visually or audibly guiding the user on how to sort waste, and a monitoring means for monitoring the status of waste disposal with a selected device. This enables users to sort waste accurately and easily, improving the accuracy of the sorting system and reducing the environmental burden in physical stores, etc. Furthermore, it becomes possible to manage the amount of waste accumulated in trash cans and predict the timing of collection, thus resolving hygiene issues.

[0498] "Image capture means for detecting deposited waste" refers to a device that captures deposited waste as an image using a camera or other imaging device.

[0499] "Image transmission means for sending image data to a cloud server" refers to a device or mechanism for transmitting acquired image data to a cloud server via the internet.

[0500] "Generative artificial intelligence means" refers to a means that uses artificial intelligence technology to analyze acquired image data and automatically recognize and classify the type of waste.

[0501] "Verification method" refers to a method of verifying the results classified by artificial intelligence against the waste identification database of local municipalities within Japan.

[0502] A "feedback mechanism" is a device or mechanism that informs the user of the correct disposal method or areas for improvement based on the verification results.

[0503] An "alert device" is a device or mechanism that warns the user or system when an error in the feedback is detected.

[0504] An "interactive concierge service" is an interactive support method that provides users with real-time guidance on things like how to sort their waste.

[0505] "Emotion recognition means" refers to technology that recognizes a user's facial expressions, voice, etc., and analyzes their emotional state.

[0506] "Notification means for guiding users on how to sort waste visually or audibly" refers to means of instructing users on how to sort waste visually or audibly using displays or speakers.

[0507] A "monitoring device" refers to a device or mechanism that continuously monitors the status of waste disposal and the amount of waste accumulated in the trash cans using a selected device.

[0508] "Accumulation amount measuring means" refers to sensors or devices used to measure the amount of waste in a trash can.

[0509] A "prediction tool" is a device or program used to calculate the next collection date based on data on the amount of waste accumulated.

[0510] A "notification means" is a device or mechanism for informing users or administrators of predicted results or important information.

[0511] "Remote monitoring means" refers to technologies and devices that allow users to monitor the status of garbage disposal from a remote location.

[0512] System Configuration

[0513] This invention is a system that detects deposited waste and supports users in proper waste sorting. This system consists of the following main hardware and software components.

[0514] hardware

[0515] 1. Camera (e.g., Logitech C920)

[0516] A device for capturing images of garbage that has been dropped off.

[0517] 2. Pressure sensor (e.g., Phidgets 3132)

[0518] A device for detecting the dropping off of trash.

[0519] 3. Display monitor (Example: LG 24MP88HV-S)

[0520] A device for providing visual feedback to users.

[0521] 4. Smartphone (e.g., iPhone 12)

[0522] A mobile device for running dedicated applications.

[0523] software

[0524] 1. Cloud server (e.g., Amazon Web Services)

[0525] A server that receives and analyzes image data to recognize the type of waste.

[0526] 2. Image analysis models (e.g., TensorFlow)

[0527] An artificial intelligence model for identifying the type of waste using acquired image data.

[0528] 3. Emotion recognition engine (e.g., IBM Watson Emotion Analysis)

[0529] An engine that analyzes user emotions and provides appropriate feedback.

[0530] 4. Database (e.g., PostgreSQL)

[0531] It will be used as a waste identification database and compared with the analysis results on the cloud server.

[0532] 5. Dedicated application (e.g., developed using Swift or Kotlin)

[0533] An application for users to monitor their waste sorting status.

[0534] System operation

[0535] 1. Detecting and capturing images of trash being dropped off.

[0536] When the device's pressure sensor detects the dropping off of trash, the camera automatically captures an image. This image data is then sent to a cloud server.

[0537] 2. Image Analysis

[0538] The cloud server passes the received image data to a TensorFlow model, which analyzes the type of garbage.

[0539] 3. Matching and Feedback

[0540] The analysis results are compared with a PostgreSQL database to provide users with feedback on correct waste sorting. This feedback is displayed visually on a display monitor or smartphone.

[0541] 4. Alerts and interactive concierge

[0542] If incorrect sorting is detected, the system will issue an alert and activate an interactive concierge to provide detailed guidance.

[0543] 5. Emotion Recognition and Adaptive Feedback

[0544] IBM Watson's emotion recognition engine analyzes the user's facial expressions and voice, providing appropriate feedback based on their emotions.

[0545] 6. Prediction of accumulated amount and next collection date

[0546] A pressure sensor periodically measures the amount of trash in the trash can and sends the data to a cloud server. The cloud server predicts the next collection date based on the amount of trash and the collection schedule, and notifies the user.

[0547] 7. Remote Monitoring

[0548] Through a dedicated application, users can remotely monitor the status of their trash disposal in real time.

[0549] Specific example

[0550] 1. In the case of correct sorting

[0551] The user throws the plastic bottle into the recycling bin.

[0552] A sensor detects the mail being dropped in, and a camera takes a picture.

[0553] A cloud server analyzes the image and identifies it as a plastic bottle.

[0554] The display shows "This is recyclable waste, thank you!"

[0555] 2. In the case of incorrect sorting

[0556] The user throws the aluminum can into the recycling bin.

[0557] Analysis of the cloud server identified this as metal waste.

[0558] The display shows, "This is metal waste. Please dispose of it in the correct trash can."

[0559] If the user makes a displeased expression, the emotion recognition engine will detect it and gently say, "Recycling is important, let's try again!"

[0560] Example of a prompt

[0561] The following is an example of prompt statements used when analyzing waste using a generative AI model.

[0562] prompt:

[0563] New image data has been received. Please identify the type of trash shown in this image.

[0564] Image data:

[0565] [Base64 encoded image data]

[0566] Database reference:

[0567] Please use the latest database for classifying waste types.

[0568] This system enables users to sort waste easily and accurately, improving the precision of sorting procedures and reducing environmental impact in physical stores and other locations. Furthermore, it allows for management of waste bin volume and prediction of collection timing, resolving hygiene issues.

[0569] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0570] Step 1:

[0571] The device uses a pressure sensor to detect the dropping of trash. When trash is detected, the camera automatically captures an image and sends the image data to a cloud server. The input is the dropping of trash, and the output is the image data sent to the cloud server.

[0572] Step 2:

[0573] The server passes the received image data to a generating AI model for image analysis. Specifically, it uses a TensorFlow model to identify the type of waste. In this process, the input is image data, and the output is the analysis result indicating the type of waste.

[0574] Step 3:

[0575] The server compares the analysis results with the PostgreSQL database to verify the correct classification. The input is the analysis results and database information, and the output is the comparison result. If the comparison result is correct, feedback is provided to the user based on that information.

[0576] Step 4:

[0577] The feedback mechanism provides users with visual feedback via a display monitor or smartphone. The input is the matching result from a cloud server, and the output is a feedback message to the user. For example, it might display, "This is recyclable waste, thank you!"

[0578] Step 5:

[0579] If incorrect sorting is detected, the server issues an alert, and an interactive concierge begins providing detailed instructions. The input is the error detection result, and the output is the alert notification and the activation of the interactive concierge. The concierge displays on the screen, "This is metal waste. Please dispose of it in the correct trash can."

[0580] Step 6:

[0581] The server uses IBM Watson's emotion recognition engine to analyze the user's facial expressions and voice to determine their emotional state. Input is the user's facial expression data and voice, and output is the emotion analysis result. Appropriate feedback is provided based on the user's emotions.

[0582] Step 7:

[0583] The server periodically collects data from pressure sensors to measure the amount of waste in the trash cans. The input is pressure sensor data, and the output is the amount of waste. Based on this data, the server predicts the next collection date and notifies users and administrators.

[0584] Step 8:

[0585] Users can remotely monitor the status of their trash collection in real time through a dedicated application. Input is data from a cloud server, and output is the collection status displayed on the application. This makes it possible to manage trash even from a distance.

[0586] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0587] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0588] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0589] [Second Embodiment]

[0590] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0591] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0592] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0593] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0594] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0595] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0596] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0597] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0598] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0599] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0600] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0601] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0602] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. The details of the system's program processing and specific examples are described below.

[0603] System Configuration

[0604] 1. Device (trash can camera and sensors)

[0605] It is equipped with a sensor that detects when trash is dropped in and a camera that captures images of the dropped-in trash.

[0606] It has the function to send captured image data to a cloud server.

[0607] 2. Cloud Server

[0608] The system receives image data and uses artificial intelligence (AI) to identify the type of waste.

[0609] The analysis results are compared with the waste identification database of local governments in Japan, and feedback is provided to users.

[0610] If an incorrect sorting is detected, an alert will be issued and an interactive concierge will be activated.

[0611] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[0612] 3. Users (using a dedicated application)

[0613] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[0614] It is possible to monitor the status of garbage disposal remotely.

[0615] Program processing

[0616] This program will be executed in the following steps.

[0617] 1. Detecting and capturing images of trash being dropped off.

[0618] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[0619] The device sends the captured image data to the cloud server.

[0620] 2. Image Analysis

[0621] The server receives image data and passes it to the artificial intelligence model for generating data.

[0622] An AI model analyzes the image and identifies the type of waste.

[0623] 3. Matching and Feedback

[0624] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0625] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[0626] 4. Alerts and interactive concierge

[0627] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[0628] 5. Prediction of accumulated amount and next collection date

[0629] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[0630] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[0631] 6. Remote Monitoring

[0632] Users can monitor the status of their trash collection in real time through a dedicated application.

[0633] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[0634] Specific example

[0635] 1. In the case of correct sorting

[0636] The user throws the plastic bottle into the trash can.

[0637] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[0638] The server analyzes the image and identifies it as plastic.

[0639] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[0640] 2. In the case of incorrect sorting

[0641] A user threw a metal can into a trash can, but the cloud server analysis and comparison results detected an error (for example, classifying it as combustible waste).

[0642] An alert system issues an alert, and an interactive concierge guides the user on the correct way to sort metal cans for disposal.

[0643] This invention will improve the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, it will enable family members and caregivers living separately to monitor waste sorting status in real time and provide appropriate support.

[0644] The following describes the processing flow.

[0645] Step 1:

[0646] Terminal: "Detects the dropping off of trash"

[0647] A sensor installed in the trash can detects when trash is placed inside.

[0648] When trash is detected being dropped in, the sensor activates the camera.

[0649] Step 2:

[0650] Device: "Capture image"

[0651] The camera captures an image of the trash that has been dropped off.

[0652] The captured image data is temporarily stored inside the device.

[0653] Step 3:

[0654] Terminal: "Send image data to the cloud server"

[0655] The device sends the captured image data to the cloud server.

[0656] Once the transmission is complete, the device waits for a response from the cloud server.

[0657] Step 4:

[0658] Server: "Receive image data"

[0659] The cloud server receives the image data.

[0660] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[0661] Step 5:

[0662] Server: "Perform image analysis"

[0663] The AI ​​model analyzes the image data.

[0664] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[0665] Step 6:

[0666] Server: "Compare analysis results with the waste identification database."

[0667] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0668] Through verification, we determine whether the waste has been sorted appropriately.

[0669] Step 7:

[0670] Server: "Provide feedback to users"

[0671] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[0672] Feedback is sent to users through a dedicated application.

[0673] Step 8:

[0674] Device: "Receive and display feedback"

[0675] The terminal receives feedback from the server.

[0676] Display the received feedback to the user.

[0677] Step 9:

[0678] Server: "Issues an alert in case of incorrect sorting."

[0679] If incorrect classification is detected, the server will immediately issue an alert.

[0680] The alert will point out the error to the user and provide the correct sorting method.

[0681] Step 10:

[0682] Terminal: "Launch the interactive concierge"

[0683] When an alert is issued, the device activates its interactive concierge function.

[0684] The concierge function provides users with guidance and support in an interactive format.

[0685] Step 11:

[0686] Terminal: "Measure accumulated amount"

[0687] A sensor inside the trash can measures the amount of waste accumulated.

[0688] The measured data is sent to the cloud server.

[0689] Step 12:

[0690] Server: "Predict the next collection date"

[0691] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[0692] The prediction results are notified to the user through a dedicated application.

[0693] Step 13:

[0694] User: "Perform remote monitoring"

[0695] A dedicated application is used to monitor the status of garbage disposal in real time.

[0696] Users can assist elderly individuals with sorting waste as needed.

[0697] The above describes the specific flow of program processing in the system of the present invention.

[0698] (Example 1)

[0699] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0700] Proper waste sorting within the home is difficult for many people, and mistakes are frequent, especially among the elderly and busy households. As a result, the efficiency and accuracy of waste disposal decline, making it difficult for local governments to collect and recycle waste. On the other hand, there are limited ways for family members living far away to support elderly people with waste sorting. There is a need to solve this problem and improve the accuracy of waste sorting.

[0701] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0702] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for sending the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a storage amount measuring means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. This improves the accuracy of waste sorting, streamlines waste disposal, and enables support from remote locations.

[0703] A "sensor means" refers to a device or function used to detect the dropping of trash.

[0704] "Camera means" refers to a photographic device or function used to capture images of the trash that has been dropped off.

[0705] "Image transmission means" refers to a device or function for transmitting captured image data to a cloud server.

[0706] "Generative artificial intelligence means" refers to AI technology and functions used to analyze image data and classify waste.

[0707] "Verification means" refers to a device or function for verifying analysis results against a waste identification database.

[0708] A "feedback mechanism" refers to a device or function that provides feedback to the user based on the analysis results.

[0709] An "alert mechanism" is a device or function that issues an alert when an error in the feedback is detected.

[0710] "Interactive concierge services" refer to conversational systems and functions that provide real-time guidance to users.

[0711] "Accumulation amount measuring means" refers to a device or function for measuring the accumulated amount of waste.

[0712] A "prediction tool" is a function for predicting the next collection date based on the measured amount of accumulated waste.

[0713] "Notification means" refers to devices or functions used to notify the user of the prediction results.

[0714] "Remote monitoring means" refers to devices or functions that allow users to remotely monitor the status of waste disposal.

[0715] This invention provides an IoT system that assists with household waste sorting, and in particular, provides a configuration for achieving accurate waste classification and sorting. The system detects the waste that is dropped off, analyzes its image to identify the type of waste, and notifies the user of the correct sorting method.

[0716] System Configuration

[0717] This system includes the following main components and means:

[0718] 1. Device (trash can sensor and camera)

[0719] Sensors: Installed to detect the dropping of trash. Infrared sensors and ultrasonic sensors are used.

[0720] Camera method: Capture images of the dropped-off trash. A high-resolution camera is used to photograph the dropped-off trash.

[0721] 2. Cloud Server

[0722] Image transmission method: Captured image data is sent to a cloud server. This utilizes a Wi-Fi module and transmits data using the HTTP protocol.

[0723] Generative AI method: Analyzes received image data to identify the type of waste. For this purpose, AI technologies such as the Google Cloud Vision API are used.

[0724] Verification method: The analysis results are compared with a waste identification database. Based on the verification results, the correct waste sorting method is confirmed.

[0725] Feedback method: Notify the user of the feedback results. For example, use push notifications to notify a dedicated application.

[0726] Alerting method: An alert is issued when an error in feedback is detected. The alert is sent to the user via push notification.

[0727] Interactive concierge system: A dialogue system that provides real-time guidance to users. This may utilize services such as Amazon Lex.

[0728] 3. Managing the amount of trash in the trash cans and the collection schedule.

[0729] Accumulation amount measurement method: Measures the amount of accumulated waste in the trash can. The amount of waste is measured using an ultrasonic sensor or similar device.

[0730] Prediction method: Predict the next collection date based on the accumulated amount and the local government's collection schedule. Based on the prediction results, determine the next collection date.

[0731] Notification method: Notify the user of the prediction results. For example, send a notification such as, "The next collection date is in 2 days."

[0732] 4. User Interface

[0733] Remote monitoring method: A dedicated application allows users to monitor the status of garbage disposal in real time. Family members and caregivers can also assist elderly individuals with their garbage sorting.

[0734] Specific example

[0735] In the case of correct sorting

[0736] The user throws the plastic bottle into the trash can.

[0737] The device's sensors detect the mail being dropped in, and the camera captures an image.

[0738] The device sends image data to the cloud server.

[0739] The server analyzes the image and identifies it as plastic.

[0740] If the matching result is correct, the server notifies the user with "Correct classification."

[0741] In case of incorrect sorting

[0742] The user throws the metal can into the trash can.

[0743] The device's sensors detect the mail being dropped in, and the camera captures an image.

[0744] The device sends image data to the cloud server.

[0745] The server analyzes the image to identify incorrect sorting (for example, classifying it as combustible waste).

[0746] The server issues an alert, activates an interactive concierge, and guides the user through the correct sorting method.

[0747] Example of a prompt

[0748] 1. Scenario: Notifying people about the correct sorting of plastic bottles.

[0749] "A plastic bottle has been placed in the trash can. Please ensure proper sorting."

[0750] 2. Scenario: Notification of incorrect sorting of metal cans.

[0751] "A metal can was placed in the trash can, but it was detected as incorrectly sorted. Please provide instructions on the correct sorting method."

[0752] With the above configuration, this system can improve the accuracy of waste sorting and provide users with quick and appropriate support.

[0753] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0754] Detailed explanation of the system's program processing.

[0755] Step 1: Detecting trash and capturing images

[0756] Input: Garbage is placed in the trash can.

[0757] Operation: The device's sensors detect the placement of trash. These sensors use infrared or ultrasonic sensors. When trash is detected, the device's camera automatically captures an image. The image is saved in high-resolution JPEG format and a timestamp is added.

[0758] Output: Captured image data of the trash.

[0759] Step 2: Sending image data to the cloud

[0760] Input: Image data of trash captured on a device.

[0761] Operation: The device sends captured image data to the cloud server. This data is transmitted via the HTTP protocol using the Wi-Fi module. The data is then uploaded to the cloud server via a REST API.

[0762] Output: Image data stored on a cloud server.

[0763] Step 3: Image Analysis

[0764] Input: Image data stored on a cloud server.

[0765] Operation: The server receives image data and passes it to an artificial intelligence model. This AI model uses the Google Cloud Vision API to analyze objects in the image and identify the type of waste. The analysis uses information such as shape, color, and text.

[0766] Output: Type of waste (e.g., plastic, metal, paper, etc.).

[0767] Step 4: Matching and Feedback

[0768] Input: The type of waste analyzed.

[0769] Operation: The server compares the analysis results with the waste identification database of domestic municipalities. A REST API is used to communicate with the database and verify the accuracy of the analysis results.

[0770] If the matching results are accurate, the server uses a dedicated application to send feedback to the user about the correct sorting.

[0771] Output: Feedback notification such as "Correct sorting."

[0772] Step 5: Alerts and Interactive Concierge

[0773] Input: Matching result when an incorrect classification is detected.

[0774] Operation: When the server detects incorrect sorting, it issues an alert. The alert is pushed to a dedicated application. Additionally, an interactive concierge is activated to teach the user the correct sorting method. Amazon Lex is used for the interactive concierge, providing support via voice and text.

[0775] Output: Alert notifications such as "Incorrect sorting detected," and guidance on sorting methods from an interactive concierge.

[0776] Step 6: Measure the amount of trash in the bins and predict the next collection date.

[0777] Input: Data on the amount of waste bin contents.

[0778] Operation: The device's sensors measure the amount of trash accumulated in the trash can. An ultrasonic sensor is used for this, and the data is sent to a cloud server. The server uses this information to compare it with the local government's collection schedule and predict the next collection date.

[0779] Output: Notification of prediction results such as "The next collection date is in 2 days."

[0780] Step 7: Remote Monitoring

[0781] Input: Real-time data on trash disposal and trash can contents.

[0782] Operation: Users monitor the status of their garbage disposal in real time through a dedicated application. The application displays data from a cloud server and allows users to check past disposal history. In addition, family members and caregivers can remotely assist elderly individuals with their garbage sorting.

[0783] Output: Real-time monitoring data display and notification.

[0784] The above outlines the specific steps of the system's program processing and the detailed operation of each step. This system allows users to obtain accurate waste sorting information in real time, improving the efficiency and accuracy of waste disposal.

[0785] (Application Example 1)

[0786] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0787] Sorting waste generated on factory production lines efficiently and accurately is difficult. Traditional manual waste sorting is prone to human error, and further automation and real-time monitoring are needed. In addition, a convenient display system is required so that workers can constantly check the status of waste sorting.

[0788] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0789] In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a display means for displaying the waste deposit status in real time on a display device worn by the user, and a mechanical means for automating waste deposit. This enables accurate sorting and real-time monitoring of waste.

[0790] "Image capture means" refers to a device that takes pictures of the discarded waste that has been submitted and acquires the image data.

[0791] "Image transmission means" refers to a device or function that transmits acquired image data to a cloud server.

[0792] "Generative artificial intelligence means" refers to an artificial intelligence model that analyzes received image data to identify the type of waste.

[0793] "Verification means" refers to a method of comparing and verifying the analyzed data with the waste identification database of local municipalities within Japan.

[0794] A "feedback method" is a means of notifying the user of the sorting results based on the matching results.

[0795] An "alert system" is a means of issuing a warning when an error in classification is detected.

[0796] An "interactive concierge service" is an interactive system that guides users in real time on the correct sorting methods.

[0797] "Display means" refers to a means of displaying the status of waste disposal in real time on a display device worn by the user.

[0798] "Mechanical means" refers to a device or function for automating the disposal of waste.

[0799] A "storage unit measurement means" is a means of measuring the amount of waste using a sensor.

[0800] A "prediction method" is a means of predicting the next collection date based on the accumulated amount and collection schedule.

[0801] "Notification means" refers to a means of notifying users of information such as the predicted collection date.

[0802] A "remote monitoring method" is a means that allows users to remotely monitor the status of waste disposal.

[0803] This invention is a system for accurately sorting and managing waste generated on a factory production line. The system provides multiple functions, including waste classification, accumulation measurement, collection date prediction, and real-time monitoring.

[0804] System Configuration

[0805] 1. Terminal (waste disposal robot and camera)

[0806] It is equipped with a robot that manages the disposal of waste and uses sensors to detect waste.

[0807] When waste is detected, the camera captures an image and sends that image data to a cloud server.

[0808] 2. Cloud Server

[0809] The received image data is passed to a "generating artificial intelligence means" to identify the type of waste.

[0810] The identified waste types are cross-referenced with the "Domestic Municipal Waste Identification Database."

[0811] Based on the matching results, provide feedback to the user.

[0812] If there are errors in the feedback results, an alert will be issued and an interactive concierge will guide you to the correct solution.

[0813] The system measures the amount of waste accumulated, predicts the next collection date, and notifies the user.

[0814] 3. Users (using smart glasses or dedicated applications)

[0815] You can check the status of your waste disposal in real time through smart glasses.

[0816] Receive notifications and alerts from the cloud server through a dedicated application.

[0817] It is possible to monitor the status of waste disposal remotely.

[0818] Program processing

[0819] The program for this system is processed as follows:

[0820] 1. Waste disposal detection and image capture

[0821] When the device's sensor detects the dropping off of waste, the camera automatically captures an image.

[0822] The acquired image data is sent to a cloud server.

[0823] For example, we will utilize OpenCV as the hardware to be used.

[0824] 2. Image Analysis

[0825] The cloud server receives image data and passes it to a "generative AI model" to identify the type of waste.

[0826] The analysis algorithm uses artificial intelligence models such as TensorFlow and PyTorch.

[0827] 3. Matching and Feedback

[0828] The analysis results generated by artificial intelligence are compared with the "garbage identification database of domestic municipalities."

[0829] Based on the matching results, feedback is provided to the user.

[0830] Database matching is performed using database management systems such as SQL.

[0831] 4. Alerts and interactive concierge

[0832] If an error in sorting is detected, the server will issue an alert, and an interactive system will guide the user on the correct sorting method.

[0833] The conversational concierge will utilize conversational services such as Google Dialogflow and AWS Lex.

[0834] 5. Measurement of accumulated amount and prediction of the next collection date

[0835] Sensors measure the amount of waste accumulated and transmit that information to a cloud server.

[0836] The server predicts the next collection date based on the accumulated amount and collection schedule, and notifies the user.

[0837] Notifications are delivered via a dedicated application or smart glasses.

[0838] Specific example

[0839] For example, when a worker in a factory drops off metal waste, a sensor detects the drop and a camera captures an image. This image is sent to a cloud server and recognized as metal by a generative AI model. This recognition result is then compared with a waste identification database of local municipalities to confirm the correct sorting method. The worker is then notified via smart glasses that "this is the correct sorting."

[0840] Example of a prompt

[0841] "This is a captured image of waste. Identify the type of waste and select one of the following categories: [Combustible waste, Non-combustible waste, Plastic, Metal, Bottles, Paper]"

[0842] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0843] Step 1:

[0844] The device uses a sensor to detect the placement of waste. The sensor detects that waste has been thrown into the trash can and notifies the device of the event. Based on this information, the camera is activated and an image is captured. The input is the waste placement event, and the output is the captured image data.

[0845] Step 2:

[0846] The device sends the captured image data to the cloud server. The image data is uploaded to the cloud server via the network and prepared for further analysis. The input is the captured image data, and the output is the transmission of the image data to the cloud server.

[0847] Step 3:

[0848] The server passes the received image data to an artificial intelligence (AI) generator. The server inputs the image data into the AI ​​model and performs analysis to identify the type of waste. The input is image data uploaded to the cloud server, and the output is the analyzed waste type information.

[0849] Step 4:

[0850] The server compares the identified waste type with the waste identification database of domestic municipalities. Database matching is performed using a database management system such as SQL. The input is waste type information analyzed by AI, and the output is the matching result.

[0851] Step 5:

[0852] The server provides feedback to the user based on the matching results. This feedback can be received by the user through smart glasses or a dedicated application. The input is the matching result, and the output is the feedback information for the user.

[0853] Step 6:

[0854] If the server detects an error in the feedback result, it issues an alert, and an interactive concierge guides the user through the correct sorting process. The concierge uses tools such as Google Dialogflow or AWS Lex. The input is the error information from the feedback result, and the output is the user's alert and interactive guidance.

[0855] Step 7:

[0856] A sensor measures the amount of waste accumulated and sends this information to a cloud server. The server predicts the next collection date based on the accumulated amount and collection schedule, and notifies the user. The input is the measured amount of waste, and the output is the predicted next collection date.

[0857] Step 8:

[0858] Users can check the real-time status of their waste disposal through smart glasses or a dedicated application. Based on this information, users can take appropriate disposal actions. The input is the waste disposal status, and the output is the real-time displayed disposal information.

[0859] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0860] This invention is a system that supports waste sorting using an IoT device attached to a household trash can, and further includes a function to recognize the user's emotions and adjust the feedback accordingly. The details and specific examples of the system's program processing are described below.

[0861] System Configuration

[0862] 1. Device (trash can camera and sensors)

[0863] It is equipped with a sensor that detects when trash is dropped in and a camera that captures images of the dropped-in trash.

[0864] It has the function to send captured image data to a cloud server.

[0865] 2. Cloud Server

[0866] The system receives image data and uses artificial intelligence (AI) to identify the type of waste.

[0867] The analysis results are compared with the waste identification database of local governments in Japan, and feedback is provided to users.

[0868] If an incorrect sorting is detected, an alert will be issued and an interactive concierge will be activated.

[0869] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[0870] It features an emotion recognition engine that recognizes user emotions and appropriately adjusts feedback and guidance.

[0871] 3. Users (using a dedicated application)

[0872] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[0873] It is possible to monitor the status of garbage disposal remotely.

[0874] Program processing

[0875] This program will be executed in the following steps.

[0876] 1. Detecting and capturing images of trash being dropped off.

[0877] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[0878] The device sends the captured image data to the cloud server.

[0879] 2. Image Analysis

[0880] The server receives image data and passes it to the artificial intelligence model for generating data.

[0881] An AI model analyzes the image and identifies the type of waste.

[0882] 3. Matching and Feedback

[0883] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0884] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[0885] 4. Alerts and interactive concierge

[0886] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[0887] 5. Emotion Recognition and Adaptive Feedback

[0888] The server's emotion recognition engine detects the user's emotional state.

[0889] If a user is feeling stressed or confused, an interactive concierge will provide gentle and courteous guidance.

[0890] 6. Prediction of accumulated amount and next collection date

[0891] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[0892] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[0893] 7. Remote Monitoring

[0894] Users can monitor the status of their trash collection in real time through a dedicated application.

[0895] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[0896] Specific example

[0897] 1. In the case of correct sorting

[0898] The user throws the plastic bottle into the trash can.

[0899] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[0900] The server analyzes the image and identifies it as plastic.

[0901] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[0902] 2. In the case of incorrect sorting

[0903] A user throws a metal can into a trash can, but the cloud server's analysis and matching results detect an error (for example, classifying it as combustible waste).

[0904] The server uses an emotion recognition engine to check the user's emotional state.

[0905] An alert is issued, and an interactive concierge calmly instructs, "This is metal waste. Please sort it correctly."

[0906] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[0907] The following describes the processing flow.

[0908] Step 1:

[0909] Terminal: "Detects the dropping off of trash"

[0910] A sensor installed in the trash can detects when trash is placed inside.

[0911] When trash is detected being dropped in, the sensor activates the camera.

[0912] Step 2:

[0913] Device: "Capture image"

[0914] The camera captures an image of the trash that has been dropped off.

[0915] The captured image data is temporarily stored inside the device.

[0916] Step 3:

[0917] Terminal: "Send image data to the cloud server"

[0918] The device sends the captured image data to the cloud server.

[0919] Once the transmission is complete, the device waits for a response from the cloud server.

[0920] Step 4:

[0921] Server: "Receive image data"

[0922] The cloud server receives the image data.

[0923] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[0924] Step 5:

[0925] Server: "Perform image analysis"

[0926] The AI ​​model analyzes the image data.

[0927] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[0928] Step 6:

[0929] Server: "Compare analysis results with the waste identification database."

[0930] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[0931] Through verification, we determine whether the waste has been sorted appropriately.

[0932] Step 7:

[0933] Server: "Provide feedback to users"

[0934] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[0935] Feedback is sent to users through a dedicated application.

[0936] Step 8:

[0937] Device: "Receive and display feedback"

[0938] The terminal receives feedback from the server.

[0939] Display the received feedback to the user.

[0940] Step 9:

[0941] Server: "Recognize emotions"

[0942] The server's emotion recognition engine detects the user's response to the feedback.

[0943] Evaluate whether users are experiencing stress or confusion.

[0944] Step 10:

[0945] Server: "Perform feedback adjustments"

[0946] Based on evaluations by the emotion recognition engine, the interactive concierge system adjusts the feedback content.

[0947] If the user is feeling stressed, provide detailed instructions in a gentle tone.

[0948] Step 11:

[0949] Server: "Issues an alert in case of incorrect sorting."

[0950] If incorrect classification is detected, the server will immediately issue an alert.

[0951] The alert will point out the error to the user and provide the correct sorting method.

[0952] Step 12:

[0953] Terminal: "Launch the interactive concierge"

[0954] When an alert is issued, the device activates its interactive concierge function.

[0955] The concierge function provides users with guidance and support in an interactive format.

[0956] Step 13:

[0957] Terminal: "Measure accumulated amount"

[0958] A sensor inside the trash can measures the amount of waste accumulated.

[0959] The measured data is sent to the cloud server.

[0960] Step 14:

[0961] Server: "Predict the next collection date"

[0962] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[0963] The prediction results are notified to the user through a dedicated application.

[0964] Step 15:

[0965] User: "Perform remote monitoring"

[0966] A dedicated application is used to monitor the status of garbage disposal in real time.

[0967] Users can assist elderly individuals with sorting waste as needed.

[0968] The above describes the specific flow of program processing in the system of the present invention.

[0969] (Example 2)

[0970] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0971] In modern society, waste sorting is a crucial environmental issue, but many people are confused about sorting methods, leading to frequent missorting. Elderly people and those unfamiliar with sorting are particularly prone to stress and confusion during the process. Furthermore, managing the amount of waste accumulated in households and knowing the next collection date is difficult, making timely disposal challenging. Additionally, family members and caregivers living far away have limited means of monitoring waste sorting status. A system is needed to address these challenges and improve the efficiency and accuracy of waste sorting.

[0972] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0973] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for transmitting the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to identify the type of waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an emotion recognition means for detecting the user's emotional state using an emotion recognition engine, an adaptive feedback means for adjusting guidance based on the emotion recognition means, an interactive concierge means for providing guidance to the user in real time, a storage amount measurement means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. As a result, the accuracy of waste sorting is improved, the efficiency of waste processing is increased, user stress is reduced, and sorting work can be carried out smoothly. Furthermore, it becomes easier to provide support because family members or caregivers in remote locations can monitor the mailing status.

[0974] A "sensor device" is a device used to detect waste placed in a trash can.

[0975] A "camera device" is a photographic device used to capture images of the discarded waste that has been submitted.

[0976] "Image transmission means" refers to a communication device for sending captured image data to a cloud server.

[0977] "Generative artificial intelligence means" refers to artificial intelligence technology used to analyze received image data and identify the type of waste.

[0978] "Verification means" refers to a device or software for verifying analysis results against a waste identification database of a domestic municipality.

[0979] A "feedback mechanism" is a device or system that provides the user with the correct sorting method based on the matching results.

[0980] An "alert means" is a device or system that issues an alert when it detects an error in the feedback provided by the feedback means.

[0981] "Emotion recognition means" refers to a device or software that uses an emotion recognition engine to detect the user's emotional state.

[0982] "Adaptive feedback means" refers to a device or system for adjusting guidance based on the emotional state detected by the emotion recognition means.

[0983] An "interactive concierge system" is a dialogue system that provides users with real-time sorting and guidance.

[0984] A "storage volume measuring device" is a sensor device used to measure the amount of waste accumulated.

[0985] "Prediction means" refers to a device or software for predicting the next collection date based on the accumulated amount.

[0986] "Notification means" refers to a device or system for notifying the user of the prediction results.

[0987] "Remote monitoring means" refers to a device or system that allows a user to remotely monitor the status of waste disposal.

[0988] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. Furthermore, it has a function to recognize the user's emotions and adjust the feedback accordingly. The embodiments of this system are described in detail below.

[0989] System Configuration

[0990] 1. Device (trash can camera and sensors)

[0991] A sensor attached to the trash can detects when trash is placed inside.

[0992] When a mail item is detected, the camera automatically captures an image and sends the image data to a cloud server. This device has a built-in communication module that allows it to send data to the cloud server.

[0993] 2. Cloud Server

[0994] The cloud server receives image data sent from the terminal. The received image data is then passed to a generative AI model on the server for analysis.

[0995] The generating AI model analyzes the images and identifies the type of waste. The analysis results are then compared against a waste identification database used by local municipalities in Japan.

[0996] Based on the matching results, users are provided with feedback on correct waste sorting. This feedback is presented to the user via notification or in real time.

[0997] If incorrect sorting is detected, an alert is issued, and an interactive concierge guides you through the correct sorting method.

[0998] The server also uses an emotion recognition engine to detect the user's emotional state. If the user is feeling stressed or confused, an interactive concierge will provide gentle and helpful guidance.

[0999] Sensors measure the amount of trash accumulated in the trash cans, and this data is sent to a cloud server. The cloud server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[1000] 3. User (using a dedicated application)

[1001] Users can receive feedback and alert notifications, as well as guidance from an interactive concierge, through a dedicated application.

[1002] The application allows for remote monitoring of waste disposal status. This enables family members and caregivers to remotely assist elderly individuals with their waste sorting.

[1003] Specific example

[1004] 1. In the case of correct sorting

[1005] The user throws the plastic bottle into the trash can.

[1006] The device's sensor detects the insertion of the item, and the camera captures an image of the inserted plastic bottle. The captured image data is sent to a cloud server.

[1007] The cloud server passes the image data to a generating AI model, which identifies it as a plastic bottle. A verification method confirms the accuracy of the analysis result.

[1008] The user will be notified via a feedback system that "this is the correct sorting."

[1009] 2. In the case of incorrect sorting

[1010] A user throws a metal can into a trash can, but the analysis results incorrectly classify it as combustible waste.

[1011] The cloud server's emotion recognition engine checks the user's emotional state. If the user notices an error and is feeling stressed, an alert is issued.

[1012] Through an interactive concierge system, you'll be gently guided with instructions like, "This is metal waste. Please classify it correctly."

[1013] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[1014] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1015] Step 1:

[1016] Garbage disposal detection and image capture

[1017] The device uses a sensor to detect when waste is placed in the trash can. When a user places waste in the trash can, the sensor detects this action. At that time, the sensor emits a trigger signal indicating "Trash has been placed in the trash can." Subsequently, the camera that receives this trigger signal automatically captures an image. Captured image data is then generated.

[1018] Input: Garbage disposal action

[1019] Output: Image data of the discarded trash

[1020] Step 2:

[1021] Sending image data to a cloud server

[1022] The device sends the captured image data to the cloud server. The captured image data is uploaded to the cloud server via a communication module. During this process, necessary metadata (such as the time of posting) is also added to the image data.

[1023] Input: Captured image data

[1024] Output: Image data uploaded to the cloud server

[1025] Step 3:

[1026] Image analysis

[1027] The server passes the received image data to a generative AI model. The cloud server inputs the received image data into the AI ​​model for analysis. This generative AI model analyzes the images and identifies the type of waste. This analysis yields classification results such as "plastic bottles" or "metal cans."

[1028] Input: Image data uploaded to a cloud server

[1029] Output: Analysis results (type of waste)

[1030] Step 4:

[1031] Matching and Feedback

[1032] The server compares the analysis results with the waste identification database of local municipalities in Japan. The analyzed data is compared with the database to confirm the correct sorting method. A feedback system notifies the user about the correct sorting method. This notification is provided through a dedicated application.

[1033] Input: Analysis results

[1034] Output: Feedback on waste sorting

[1035] Step 5:

[1036] Alerts and interactive concierge

[1037] The server issues an alert if it detects incorrect sorting. When the matching system finds incorrect sorting, it sends a warning to the user through the notification system. Simultaneously, an interactive concierge is activated to guide the user on the correct sorting method.

[1038] Input: Matching results (error detection)

[1039] Output: Alert notifications and interactive concierge feedback

[1040] Step 6:

[1041] Emotion recognition and adaptive feedback

[1042] The server uses an emotion recognition engine to detect the user's emotional state. It analyzes the user's responses to feedback and alerts, and if stress or confusion is detected, the conversational concierge adjusts the tone and content of the feedback.

[1043] Input: User response data

[1044] Output: Feedback adapted to emotional state

[1045] Step 7:

[1046] Accumulated amount and prediction of the next collection date

[1047] A sensor measures the amount of trash in the bins and sends the data to a server. The server predicts the next collection date based on the amount of trash and the local government's collection schedule. The prediction result is then notified to the user through a dedicated application.

[1048] Input: Accumulated data

[1049] Output: Predicted notification of the next collection date

[1050] Step 8:

[1051] Remote monitoring

[1052] Users can monitor the status of their garbage disposal in real time through a dedicated application. The monitoring function within the application allows users and their families to remotely check the disposal status. This function makes it possible to remotely assist elderly people with their garbage sorting.

[1053] Input: Garbage disposal data

[1054] Output: Real-time mailing status display

[1055] (Application Example 2)

[1056] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1057] Traditional waste sorting methods often failed to properly separate waste, resulting in reduced recycling efficiency and potentially exacerbating social and environmental problems. Furthermore, elderly and disabled individuals frequently encountered difficulties with sorting, and there was a lack of support for proper sorting, particularly in public facilities and retail stores. Additionally, managing waste bin levels and timely collection proved challenging, leading to frequent hygiene issues. A solution was needed to address these problems while reducing the burden on users.

[1058] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing guidance to the user in real time, an emotion recognition means for recognizing the user's emotions and adjusting the feedback, a notification means for visually or audibly guiding the user on how to sort waste, and a monitoring means for monitoring the status of waste disposal with a selected device. This enables users to sort waste accurately and easily, improving the accuracy of the sorting system and reducing the environmental burden in physical stores, etc. Furthermore, it becomes possible to manage the amount of waste accumulated in trash cans and predict the timing of collection, thus resolving hygiene issues.

[1059] "Image capture means for detecting deposited waste" refers to a device that captures deposited waste as an image using a camera or other imaging device.

[1060] "Image transmission means for sending image data to a cloud server" refers to a device or mechanism for transmitting acquired image data to a cloud server via the internet.

[1061] "Generative artificial intelligence means" refers to a means that uses artificial intelligence technology to analyze acquired image data and automatically recognize and classify the type of waste.

[1062] "Verification method" refers to a method of verifying the results classified by artificial intelligence against the waste identification database of local municipalities within Japan.

[1063] A "feedback mechanism" is a device or mechanism that informs the user of the correct disposal method or areas for improvement based on the verification results.

[1064] An "alert device" is a device or mechanism that warns the user or system when an error in the feedback is detected.

[1065] An "interactive concierge service" is an interactive support method that provides users with real-time guidance on things like how to sort their waste.

[1066] "Emotion recognition means" refers to technology that recognizes a user's facial expressions, voice, etc., and analyzes their emotional state.

[1067] "Notification means for guiding users on how to sort waste visually or audibly" refers to means of instructing users on how to sort waste visually or audibly using displays or speakers.

[1068] A "monitoring device" refers to a device or mechanism that continuously monitors the status of waste disposal and the amount of waste accumulated in the trash cans using a selected device.

[1069] "Accumulation amount measuring means" refers to sensors or devices used to measure the amount of waste in a trash can.

[1070] A "prediction tool" is a device or program used to calculate the next collection date based on data on the amount of waste accumulated.

[1071] A "notification means" is a device or mechanism for informing users or administrators of predicted results or important information.

[1072] "Remote monitoring means" refers to technologies and devices that allow users to monitor the status of garbage disposal from a remote location.

[1073] System Configuration

[1074] This invention is a system that detects deposited waste and supports users in proper waste sorting. This system consists of the following main hardware and software components.

[1075] hardware

[1076] 1. Camera (e.g., Logitech C920)

[1077] A device for capturing images of garbage that has been dropped off.

[1078] 2. Pressure sensor (e.g., Phidgets 3132)

[1079] A device for detecting the dropping off of trash.

[1080] 3. Display monitor (Example: LG 24MP88HV-S)

[1081] A device for providing visual feedback to users.

[1082] 4. Smartphone (e.g., iPhone 12)

[1083] A mobile device for running dedicated applications.

[1084] software

[1085] 1. Cloud server (e.g., Amazon Web Services)

[1086] A server that receives and analyzes image data to recognize the type of waste.

[1087] 2. Image analysis models (e.g., TensorFlow)

[1088] An artificial intelligence model for identifying the type of waste using acquired image data.

[1089] 3. Emotion recognition engine (e.g., IBM Watson Emotion Analysis)

[1090] An engine that analyzes user emotions and provides appropriate feedback.

[1091] 4. Database (e.g., PostgreSQL)

[1092] It will be used as a waste identification database and compared with the analysis results on the cloud server.

[1093] 5. Dedicated application (e.g., developed using Swift or Kotlin)

[1094] An application for users to monitor their waste sorting status.

[1095] System operation

[1096] 1. Detecting and capturing images of trash being dropped off.

[1097] When the device's pressure sensor detects the dropping off of trash, the camera automatically captures an image. This image data is then sent to a cloud server.

[1098] 2. Image Analysis

[1099] The cloud server passes the received image data to a TensorFlow model, which analyzes the type of garbage.

[1100] 3. Matching and Feedback

[1101] The analysis results are compared with a PostgreSQL database to provide users with feedback on correct waste sorting. This feedback is displayed visually on a display monitor or smartphone.

[1102] 4. Alerts and interactive concierge

[1103] If incorrect sorting is detected, the system will issue an alert and activate an interactive concierge to provide detailed guidance.

[1104] 5. Emotion Recognition and Adaptive Feedback

[1105] IBM Watson's emotion recognition engine analyzes the user's facial expressions and voice, providing appropriate feedback based on their emotions.

[1106] 6. Prediction of accumulated amount and next collection date

[1107] A pressure sensor periodically measures the amount of trash in the trash can and sends the data to a cloud server. The cloud server predicts the next collection date based on the amount of trash and the collection schedule, and notifies the user.

[1108] 7. Remote Monitoring

[1109] Through a dedicated application, users can remotely monitor the status of their trash disposal in real time.

[1110] Specific example

[1111] 1. In the case of correct sorting

[1112] The user throws the plastic bottle into the recycling bin.

[1113] A sensor detects the mail being dropped in, and a camera takes a picture.

[1114] A cloud server analyzes the image and identifies it as a plastic bottle.

[1115] The display shows "This is recyclable waste, thank you!"

[1116] 2. In the case of incorrect sorting

[1117] The user throws the aluminum can into the recycling bin.

[1118] Analysis of the cloud server identified this as metal waste.

[1119] The display shows, "This is metal waste. Please dispose of it in the correct trash can."

[1120] If the user makes a displeased expression, the emotion recognition engine will detect it and gently say, "Recycling is important, let's try again!"

[1121] Example of a prompt

[1122] The following is an example of prompt statements used when analyzing waste using a generative AI model.

[1123] prompt:

[1124] New image data has been received. Please identify the type of trash shown in this image.

[1125] Image data:

[1126] [Base64 encoded image data]

[1127] Database reference:

[1128] Please use the latest database for classifying waste types.

[1129] This system enables users to sort waste easily and accurately, improving the precision of sorting procedures and reducing environmental impact in physical stores and other locations. Furthermore, it allows for management of waste bin volume and prediction of collection timing, resolving hygiene issues.

[1130] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1131] Step 1:

[1132] The device uses a pressure sensor to detect the dropping of trash. When trash is detected, the camera automatically captures an image and sends the image data to a cloud server. The input is the dropping of trash, and the output is the image data sent to the cloud server.

[1133] Step 2:

[1134] The server passes the received image data to a generating AI model for image analysis. Specifically, it uses a TensorFlow model to identify the type of waste. In this process, the input is image data, and the output is the analysis result indicating the type of waste.

[1135] Step 3:

[1136] The server compares the analysis results with the PostgreSQL database to verify the correct classification. The input is the analysis results and database information, and the output is the comparison result. If the comparison result is correct, feedback is provided to the user based on that information.

[1137] Step 4:

[1138] The feedback mechanism provides users with visual feedback via a display monitor or smartphone. The input is the matching result from a cloud server, and the output is a feedback message to the user. For example, it might display, "This is recyclable waste, thank you!"

[1139] Step 5:

[1140] If incorrect sorting is detected, the server issues an alert, and an interactive concierge begins providing detailed instructions. The input is the error detection result, and the output is the alert notification and the activation of the interactive concierge. The concierge displays on the screen, "This is metal waste. Please dispose of it in the correct trash can."

[1141] Step 6:

[1142] The server uses IBM Watson's emotion recognition engine to analyze the user's facial expressions and voice to determine their emotional state. Input is the user's facial expression data and voice, and output is the emotion analysis result. Appropriate feedback is provided based on the user's emotions.

[1143] Step 7:

[1144] The server periodically collects data from pressure sensors to measure the amount of waste in the trash cans. The input is pressure sensor data, and the output is the amount of waste. Based on this data, the server predicts the next collection date and notifies users and administrators.

[1145] Step 8:

[1146] Users can remotely monitor the status of their trash collection in real time through a dedicated application. Input is data from a cloud server, and output is the collection status displayed on the application. This makes it possible to manage trash even from a distance.

[1147] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1148] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1149] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1150] [Third Embodiment]

[1151] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1152] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1153] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1154] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1155] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1156] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1157] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1158] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1159] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1160] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1161] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1162] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1163] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. The details of the system's program processing and specific examples are described below.

[1164] System Configuration

[1165] 1. Device (trash can camera and sensors)

[1166] It is equipped with a sensor that detects when trash is dropped in and a camera that captures images of the dropped-in trash.

[1167] It has the function to send captured image data to a cloud server.

[1168] 2. Cloud Server

[1169] The system receives image data and uses artificial intelligence (AI) to identify the type of waste.

[1170] The analysis results are compared with the waste identification database of local governments in Japan, and feedback is provided to users.

[1171] If an incorrect sorting is detected, an alert will be issued and an interactive concierge will be activated.

[1172] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[1173] 3. Users (using a dedicated application)

[1174] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[1175] It is possible to monitor the status of garbage disposal remotely.

[1176] Program processing

[1177] This program will be executed in the following steps.

[1178] 1. Detecting and capturing images of trash being dropped off.

[1179] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[1180] The device sends the captured image data to the cloud server.

[1181] 2. Image Analysis

[1182] The server receives image data and passes it to the artificial intelligence model for generating data.

[1183] An AI model analyzes the image and identifies the type of waste.

[1184] 3. Matching and Feedback

[1185] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[1186] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[1187] 4. Alerts and interactive concierge

[1188] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[1189] 5. Prediction of accumulated amount and next collection date

[1190] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[1191] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[1192] 6. Remote Monitoring

[1193] Users can monitor the status of their trash collection in real time through a dedicated application.

[1194] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[1195] Specific example

[1196] 1. In the case of correct sorting

[1197] The user throws the plastic bottle into the trash can.

[1198] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[1199] The server analyzes the image and identifies it as plastic.

[1200] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[1201] 2. In the case of incorrect sorting

[1202] A user threw a metal can into a trash can, but the cloud server analysis and comparison results detected an error (for example, classifying it as combustible waste).

[1203] An alert system issues an alert, and an interactive concierge guides the user on the correct way to sort metal cans for disposal.

[1204] This invention will improve the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, it will enable family members and caregivers living separately to monitor waste sorting status in real time and provide appropriate support.

[1205] The following describes the processing flow.

[1206] Step 1:

[1207] Terminal: "Detects the dropping off of trash"

[1208] A sensor installed in the trash can detects when trash is placed inside.

[1209] When trash is detected being dropped in, the sensor activates the camera.

[1210] Step 2:

[1211] Device: "Capture image"

[1212] The camera captures an image of the trash that has been dropped off.

[1213] The captured image data is temporarily stored inside the device.

[1214] Step 3:

[1215] Terminal: "Send image data to the cloud server"

[1216] The device sends the captured image data to the cloud server.

[1217] Once the transmission is complete, the device waits for a response from the cloud server.

[1218] Step 4:

[1219] Server: "Receive image data"

[1220] The cloud server receives the image data.

[1221] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[1222] Step 5:

[1223] Server: "Perform image analysis"

[1224] The AI ​​model analyzes the image data.

[1225] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[1226] Step 6:

[1227] Server: "Compare analysis results with the waste identification database."

[1228] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[1229] Through verification, we determine whether the waste has been sorted appropriately.

[1230] Step 7:

[1231] Server: "Provide feedback to users"

[1232] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[1233] Feedback is sent to users through a dedicated application.

[1234] Step 8:

[1235] Device: "Receive and display feedback"

[1236] The terminal receives feedback from the server.

[1237] Display the received feedback to the user.

[1238] Step 9:

[1239] Server: "Issues an alert in case of incorrect sorting."

[1240] If incorrect classification is detected, the server will immediately issue an alert.

[1241] The alert will point out the error to the user and provide the correct sorting method.

[1242] Step 10:

[1243] Terminal: "Launch the interactive concierge"

[1244] When an alert is issued, the device activates its interactive concierge function.

[1245] The concierge function provides users with guidance and support in an interactive format.

[1246] Step 11:

[1247] Terminal: "Measure accumulated amount"

[1248] A sensor inside the trash can measures the amount of waste accumulated.

[1249] The measured data is sent to the cloud server.

[1250] Step 12:

[1251] Server: "Predict the next collection date"

[1252] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[1253] The prediction results are notified to the user through a dedicated application.

[1254] Step 13:

[1255] User: "Perform remote monitoring"

[1256] A dedicated application is used to monitor the status of garbage disposal in real time.

[1257] Users can assist elderly individuals with sorting waste as needed.

[1258] The above describes the specific flow of program processing in the system of the present invention.

[1259] (Example 1)

[1260] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1261] Proper waste sorting within the home is difficult for many people, and mistakes are frequent, especially among the elderly and busy households. As a result, the efficiency and accuracy of waste disposal decline, making it difficult for local governments to collect and recycle waste. On the other hand, there are limited ways for family members living far away to support elderly people with waste sorting. There is a need to solve this problem and improve the accuracy of waste sorting.

[1262] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1263] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for sending the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a storage amount measuring means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. This improves the accuracy of waste sorting, streamlines waste disposal, and enables support from remote locations.

[1264] A "sensor means" refers to a device or function used to detect the dropping of trash.

[1265] "Camera means" refers to a photographic device or function used to capture images of the trash that has been dropped off.

[1266] "Image transmission means" refers to a device or function for transmitting captured image data to a cloud server.

[1267] "Generative artificial intelligence means" refers to AI technology and functions used to analyze image data and classify waste.

[1268] "Verification means" refers to a device or function for verifying analysis results against a waste identification database.

[1269] A "feedback mechanism" refers to a device or function that provides feedback to the user based on the analysis results.

[1270] An "alert mechanism" is a device or function that issues an alert when an error in the feedback is detected.

[1271] "Interactive concierge services" refer to conversational systems and functions that provide real-time guidance to users.

[1272] "Accumulation amount measuring means" refers to a device or function for measuring the accumulated amount of waste.

[1273] A "prediction tool" is a function for predicting the next collection date based on the measured amount of accumulated waste.

[1274] "Notification means" refers to devices or functions used to notify the user of the prediction results.

[1275] "Remote monitoring means" refers to devices or functions that allow users to remotely monitor the status of waste disposal.

[1276] This invention provides an IoT system that assists with household waste sorting, and in particular, provides a configuration for achieving accurate waste classification and sorting. The system detects the waste that is dropped off, analyzes its image to identify the type of waste, and notifies the user of the correct sorting method.

[1277] System Configuration

[1278] This system includes the following main components and means:

[1279] 1. Device (trash can sensor and camera)

[1280] Sensors: Installed to detect the dropping of trash. Infrared sensors and ultrasonic sensors are used.

[1281] Camera method: Capture images of the dropped-off trash. A high-resolution camera is used to photograph the dropped-off trash.

[1282] 2. Cloud Server

[1283] Image transmission method: Captured image data is sent to a cloud server. This utilizes a Wi-Fi module and transmits data using the HTTP protocol.

[1284] Generative AI method: Analyzes received image data to identify the type of waste. For this purpose, AI technologies such as the Google Cloud Vision API are used.

[1285] Verification method: The analysis results are compared with a waste identification database. Based on the verification results, the correct waste sorting method is confirmed.

[1286] Feedback method: Notify the user of the feedback results. For example, use push notifications to notify a dedicated application.

[1287] Alerting method: An alert is issued when an error in feedback is detected. The alert is sent to the user via push notification.

[1288] Interactive concierge system: A dialogue system that provides real-time guidance to users. This may utilize services such as Amazon Lex.

[1289] 3. Managing the amount of trash in the trash cans and the collection schedule.

[1290] Accumulation amount measurement method: Measures the amount of accumulated waste in the trash can. The amount of waste is measured using an ultrasonic sensor or similar device.

[1291] Prediction method: Predict the next collection date based on the accumulated amount and the local government's collection schedule. Based on the prediction results, determine the next collection date.

[1292] Notification method: Notify the user of the prediction results. For example, send a notification such as, "The next collection date is in 2 days."

[1293] 4. User Interface

[1294] Remote monitoring method: A dedicated application allows users to monitor the status of garbage disposal in real time. Family members and caregivers can also assist elderly individuals with their garbage sorting.

[1295] Specific example

[1296] In the case of correct sorting

[1297] The user throws the plastic bottle into the trash can.

[1298] The device's sensors detect the mail being dropped in, and the camera captures an image.

[1299] The device sends image data to the cloud server.

[1300] The server analyzes the image and identifies it as plastic.

[1301] If the matching result is correct, the server notifies the user with "Correct classification."

[1302] In case of incorrect sorting

[1303] The user throws the metal can into the trash can.

[1304] The device's sensors detect the mail being dropped in, and the camera captures an image.

[1305] The device sends image data to the cloud server.

[1306] The server analyzes the image to identify incorrect sorting (for example, classifying it as combustible waste).

[1307] The server issues an alert, activates an interactive concierge, and guides the user through the correct sorting method.

[1308] Example of a prompt

[1309] 1. Scenario: Notifying people about the correct sorting of plastic bottles.

[1310] "A plastic bottle has been placed in the trash can. Please ensure proper sorting."

[1311] 2. Scenario: Notification of incorrect sorting of metal cans.

[1312] "A metal can was placed in the trash can, but it was detected as incorrectly sorted. Please provide instructions on the correct sorting method."

[1313] With the above configuration, this system can improve the accuracy of waste sorting and provide users with quick and appropriate support.

[1314] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1315] Detailed explanation of the system's program processing.

[1316] Step 1: Detecting trash and capturing images

[1317] Input: Garbage is placed in the trash can.

[1318] Operation: The device's sensors detect the placement of trash. These sensors use infrared or ultrasonic sensors. When trash is detected, the device's camera automatically captures an image. The image is saved in high-resolution JPEG format and a timestamp is added.

[1319] Output: Captured image data of the trash.

[1320] Step 2: Sending image data to the cloud

[1321] Input: Image data of trash captured on a device.

[1322] Operation: The device sends captured image data to the cloud server. This data is transmitted via the HTTP protocol using the Wi-Fi module. The data is then uploaded to the cloud server via a REST API.

[1323] Output: Image data stored on a cloud server.

[1324] Step 3: Image Analysis

[1325] Input: Image data stored on a cloud server.

[1326] Operation: The server receives image data and passes it to an artificial intelligence model. This AI model uses the Google Cloud Vision API to analyze objects in the image and identify the type of waste. The analysis uses information such as shape, color, and text.

[1327] Output: Type of waste (e.g., plastic, metal, paper, etc.).

[1328] Step 4: Matching and Feedback

[1329] Input: The type of waste analyzed.

[1330] Operation: The server compares the analysis results with the waste identification database of domestic municipalities. A REST API is used to communicate with the database and verify the accuracy of the analysis results.

[1331] If the matching results are accurate, the server uses a dedicated application to send feedback to the user about the correct sorting.

[1332] Output: Feedback notification such as "Correct sorting."

[1333] Step 5: Alerts and Interactive Concierge

[1334] Input: Matching result when an incorrect classification is detected.

[1335] Operation: When the server detects incorrect sorting, it issues an alert. The alert is pushed to a dedicated application. Additionally, an interactive concierge is activated to teach the user the correct sorting method. Amazon Lex is used for the interactive concierge, providing support via voice and text.

[1336] Output: Alert notifications such as "Incorrect sorting detected," and guidance on sorting methods from an interactive concierge.

[1337] Step 6: Measure the amount of trash in the bins and predict the next collection date.

[1338] Input: Data on the amount of waste bin contents.

[1339] Operation: The device's sensors measure the amount of trash accumulated in the trash can. An ultrasonic sensor is used for this, and the data is sent to a cloud server. The server uses this information to compare it with the local government's collection schedule and predict the next collection date.

[1340] Output: Notification of prediction results such as "The next collection date is in 2 days."

[1341] Step 7: Remote Monitoring

[1342] Input: Real-time data on trash disposal and trash can contents.

[1343] Operation: Users monitor the status of their garbage disposal in real time through a dedicated application. The application displays data from a cloud server and allows users to check past disposal history. In addition, family members and caregivers can remotely assist elderly individuals with their garbage sorting.

[1344] Output: Real-time monitoring data display and notification.

[1345] The above outlines the specific steps of the system's program processing and the detailed operation of each step. This system allows users to obtain accurate waste sorting information in real time, improving the efficiency and accuracy of waste disposal.

[1346] (Application Example 1)

[1347] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1348] Sorting waste generated on factory production lines efficiently and accurately is difficult. Traditional manual waste sorting is prone to human error, and further automation and real-time monitoring are needed. In addition, a convenient display system is required so that workers can constantly check the status of waste sorting.

[1349] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1350] In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a display means for displaying the waste deposit status in real time on a display device worn by the user, and a mechanical means for automating waste deposit. This enables accurate sorting and real-time monitoring of waste.

[1351] "Image capture means" refers to a device that takes pictures of the discarded waste that has been submitted and acquires the image data.

[1352] "Image transmission means" refers to a device or function that transmits acquired image data to a cloud server.

[1353] "Generative artificial intelligence means" refers to an artificial intelligence model that analyzes received image data to identify the type of waste.

[1354] "Verification means" refers to a method of comparing and verifying the analyzed data with the waste identification database of local municipalities within Japan.

[1355] A "feedback method" is a means of notifying the user of the sorting results based on the matching results.

[1356] An "alert system" is a means of issuing a warning when an error in classification is detected.

[1357] An "interactive concierge service" is an interactive system that guides users in real time on the correct sorting methods.

[1358] "Display means" refers to a means of displaying the status of waste disposal in real time on a display device worn by the user.

[1359] "Mechanical means" refers to a device or function for automating the disposal of waste.

[1360] A "storage unit measurement means" is a means of measuring the amount of waste using a sensor.

[1361] A "prediction method" is a means of predicting the next collection date based on the accumulated amount and collection schedule.

[1362] "Notification means" refers to a means of notifying users of information such as the predicted collection date.

[1363] A "remote monitoring method" is a means that allows users to remotely monitor the status of waste disposal.

[1364] This invention is a system for accurately sorting and managing waste generated on a factory production line. The system provides multiple functions, including waste classification, accumulation measurement, collection date prediction, and real-time monitoring.

[1365] System Configuration

[1366] 1. Terminal (waste disposal robot and camera)

[1367] It is equipped with a robot that manages the disposal of waste and uses sensors to detect waste.

[1368] When waste is detected, the camera captures an image and sends that image data to a cloud server.

[1369] 2. Cloud Server

[1370] The received image data is passed to a "generating artificial intelligence means" to identify the type of waste.

[1371] The identified waste types are cross-referenced with the "Domestic Municipal Waste Identification Database."

[1372] Based on the matching results, provide feedback to the user.

[1373] If there are errors in the feedback results, an alert will be issued and an interactive concierge will guide you to the correct solution.

[1374] The system measures the amount of waste accumulated, predicts the next collection date, and notifies the user.

[1375] 3. Users (using smart glasses or dedicated applications)

[1376] You can check the status of your waste disposal in real time through smart glasses.

[1377] Receive notifications and alerts from the cloud server through a dedicated application.

[1378] It is possible to monitor the status of waste disposal remotely.

[1379] Program processing

[1380] The program for this system is processed as follows:

[1381] 1. Waste disposal detection and image capture

[1382] When the device's sensor detects the dropping off of waste, the camera automatically captures an image.

[1383] The acquired image data is sent to a cloud server.

[1384] For example, we will utilize OpenCV as the hardware to be used.

[1385] 2. Image Analysis

[1386] The cloud server receives image data and passes it to a "generative AI model" to identify the type of waste.

[1387] The analysis algorithm uses artificial intelligence models such as TensorFlow and PyTorch.

[1388] 3. Matching and Feedback

[1389] The analysis results generated by artificial intelligence are compared with the "garbage identification database of domestic municipalities."

[1390] Based on the matching results, feedback is provided to the user.

[1391] Database matching is performed using database management systems such as SQL.

[1392] 4. Alerts and interactive concierge

[1393] If an error in sorting is detected, the server will issue an alert, and an interactive system will guide the user on the correct sorting method.

[1394] The conversational concierge will utilize conversational services such as Google Dialogflow and AWS Lex.

[1395] 5. Measurement of accumulated amount and prediction of the next collection date

[1396] Sensors measure the amount of waste accumulated and transmit that information to a cloud server.

[1397] The server predicts the next collection date based on the accumulated amount and collection schedule, and notifies the user.

[1398] Notifications are delivered via a dedicated application or smart glasses.

[1399] Specific example

[1400] For example, when a worker in a factory drops off metal waste, a sensor detects the drop and a camera captures an image. This image is sent to a cloud server and recognized as metal by a generative AI model. This recognition result is then compared with a waste identification database of local municipalities to confirm the correct sorting method. The worker is then notified via smart glasses that "this is the correct sorting."

[1401] Example of a prompt

[1402] "This is a captured image of waste. Identify the type of waste and select one of the following categories: [Combustible waste, Non-combustible waste, Plastic, Metal, Bottles, Paper]"

[1403] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1404] Step 1:

[1405] The device uses a sensor to detect the placement of waste. The sensor detects that waste has been thrown into the trash can and notifies the device of the event. Based on this information, the camera is activated and an image is captured. The input is the waste placement event, and the output is the captured image data.

[1406] Step 2:

[1407] The device sends the captured image data to the cloud server. The image data is uploaded to the cloud server via the network and prepared for further analysis. The input is the captured image data, and the output is the transmission of the image data to the cloud server.

[1408] Step 3:

[1409] The server passes the received image data to an artificial intelligence (AI) generator. The server inputs the image data into the AI ​​model and performs analysis to identify the type of waste. The input is image data uploaded to the cloud server, and the output is the analyzed waste type information.

[1410] Step 4:

[1411] The server compares the identified waste type with the waste identification database of domestic municipalities. Database matching is performed using a database management system such as SQL. The input is waste type information analyzed by AI, and the output is the matching result.

[1412] Step 5:

[1413] The server provides feedback to the user based on the matching results. This feedback can be received by the user through smart glasses or a dedicated application. The input is the matching result, and the output is the feedback information for the user.

[1414] Step 6:

[1415] If the server detects an error in the feedback result, it issues an alert, and an interactive concierge guides the user through the correct sorting process. The concierge uses tools such as Google Dialogflow or AWS Lex. The input is the error information from the feedback result, and the output is the user's alert and interactive guidance.

[1416] Step 7:

[1417] A sensor measures the amount of waste accumulated and sends this information to a cloud server. The server predicts the next collection date based on the accumulated amount and collection schedule, and notifies the user. The input is the measured amount of waste, and the output is the predicted next collection date.

[1418] Step 8:

[1419] Users can check the real-time status of their waste disposal through smart glasses or a dedicated application. Based on this information, users can take appropriate disposal actions. The input is the waste disposal status, and the output is the real-time displayed disposal information.

[1420] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1421] This invention is a system that supports waste sorting using an IoT device attached to a household trash can, and further includes a function to recognize the user's emotions and adjust the feedback accordingly. The details and specific examples of the system's program processing are described below.

[1422] System Configuration

[1423] 1. Device (trash can camera and sensors)

[1424] It is equipped with a sensor that detects when trash is dropped in and a camera that captures images of the dropped-in trash.

[1425] It has the function to send captured image data to a cloud server.

[1426] 2. Cloud Server

[1427] The system receives image data and uses artificial intelligence (AI) to identify the type of waste.

[1428] The analysis results are compared with the waste identification database of local governments in Japan, and feedback is provided to users.

[1429] If an incorrect sorting is detected, an alert will be issued and an interactive concierge will be activated.

[1430] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[1431] It features an emotion recognition engine that recognizes user emotions and appropriately adjusts feedback and guidance.

[1432] 3. Users (using a dedicated application)

[1433] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[1434] It is possible to monitor the status of garbage disposal remotely.

[1435] Program processing

[1436] This program will be executed in the following steps.

[1437] 1. Detecting and capturing images of trash being dropped off.

[1438] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[1439] The device sends the captured image data to the cloud server.

[1440] 2. Image Analysis

[1441] The server receives image data and passes it to the artificial intelligence model for generating data.

[1442] An AI model analyzes the image and identifies the type of waste.

[1443] 3. Matching and Feedback

[1444] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[1445] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[1446] 4. Alerts and interactive concierge

[1447] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[1448] 5. Emotion Recognition and Adaptive Feedback

[1449] The server's emotion recognition engine detects the user's emotional state.

[1450] If a user is feeling stressed or confused, an interactive concierge will provide gentle and courteous guidance.

[1451] 6. Prediction of accumulated amount and next collection date

[1452] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[1453] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[1454] 7. Remote Monitoring

[1455] Users can monitor the status of their trash collection in real time through a dedicated application.

[1456] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[1457] Specific example

[1458] 1. In the case of correct sorting

[1459] The user throws the plastic bottle into the trash can.

[1460] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[1461] The server analyzes the image and identifies it as plastic.

[1462] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[1463] 2. In the case of incorrect sorting

[1464] A user throws a metal can into a trash can, but the cloud server's analysis and matching results detect an error (for example, classifying it as combustible waste).

[1465] The server uses an emotion recognition engine to check the user's emotional state.

[1466] An alert is issued, and an interactive concierge calmly instructs, "This is metal waste. Please sort it correctly."

[1467] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[1468] The following describes the processing flow.

[1469] Step 1:

[1470] Terminal: "Detects the dropping off of trash"

[1471] A sensor installed in the trash can detects when trash is placed inside.

[1472] When trash is detected being dropped in, the sensor activates the camera.

[1473] Step 2:

[1474] Device: "Capture image"

[1475] The camera captures an image of the trash that has been dropped off.

[1476] The captured image data is temporarily stored inside the device.

[1477] Step 3:

[1478] Terminal: "Send image data to the cloud server"

[1479] The device sends the captured image data to the cloud server.

[1480] Once the transmission is complete, the device waits for a response from the cloud server.

[1481] Step 4:

[1482] Server: "Receive image data"

[1483] The cloud server receives the image data.

[1484] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[1485] Step 5:

[1486] Server: "Perform image analysis"

[1487] The AI ​​model analyzes the image data.

[1488] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[1489] Step 6:

[1490] Server: "Compare analysis results with the waste identification database."

[1491] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[1492] Through verification, we determine whether the waste has been sorted appropriately.

[1493] Step 7:

[1494] Server: "Provide feedback to users"

[1495] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[1496] Feedback is sent to users through a dedicated application.

[1497] Step 8:

[1498] Device: "Receive and display feedback"

[1499] The terminal receives feedback from the server.

[1500] Display the received feedback to the user.

[1501] Step 9:

[1502] Server: "Recognize emotions"

[1503] The server's emotion recognition engine detects the user's response to the feedback.

[1504] Evaluate whether users are experiencing stress or confusion.

[1505] Step 10:

[1506] Server: "Perform feedback adjustments"

[1507] Based on evaluations by the emotion recognition engine, the interactive concierge system adjusts the feedback content.

[1508] If the user is feeling stressed, provide detailed instructions in a gentle tone.

[1509] Step 11:

[1510] Server: "Issues an alert in case of incorrect sorting."

[1511] If incorrect classification is detected, the server will immediately issue an alert.

[1512] The alert will point out the error to the user and provide the correct sorting method.

[1513] Step 12:

[1514] Terminal: "Launch the interactive concierge"

[1515] When an alert is issued, the device activates its interactive concierge function.

[1516] The concierge function provides users with guidance and support in an interactive format.

[1517] Step 13:

[1518] Terminal: "Measure accumulated amount"

[1519] A sensor inside the trash can measures the amount of waste accumulated.

[1520] The measured data is sent to the cloud server.

[1521] Step 14:

[1522] Server: "Predict the next collection date"

[1523] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[1524] The prediction results are notified to the user through a dedicated application.

[1525] Step 15:

[1526] User: "Perform remote monitoring"

[1527] A dedicated application is used to monitor the status of garbage disposal in real time.

[1528] Users can assist elderly individuals with sorting waste as needed.

[1529] The above describes the specific flow of program processing in the system of the present invention.

[1530] (Example 2)

[1531] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1532] In modern society, waste sorting is a crucial environmental issue, but many people are confused about sorting methods, leading to frequent missorting. Elderly people and those unfamiliar with sorting are particularly prone to stress and confusion during the process. Furthermore, managing the amount of waste accumulated in households and knowing the next collection date is difficult, making timely disposal challenging. Additionally, family members and caregivers living far away have limited means of monitoring waste sorting status. A system is needed to address these challenges and improve the efficiency and accuracy of waste sorting.

[1533] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1534] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for transmitting the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to identify the type of waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an emotion recognition means for detecting the user's emotional state using an emotion recognition engine, an adaptive feedback means for adjusting guidance based on the emotion recognition means, an interactive concierge means for providing guidance to the user in real time, a storage amount measurement means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. As a result, the accuracy of waste sorting is improved, the efficiency of waste processing is increased, user stress is reduced, and sorting work can be carried out smoothly. Furthermore, it becomes easier to provide support because family members or caregivers in remote locations can monitor the mailing status.

[1535] A "sensor device" is a device used to detect waste placed in a trash can.

[1536] A "camera device" is a photographic device used to capture images of the discarded waste that has been submitted.

[1537] "Image transmission means" refers to a communication device for sending captured image data to a cloud server.

[1538] "Generative artificial intelligence means" refers to artificial intelligence technology used to analyze received image data and identify the type of waste.

[1539] "Verification means" refers to a device or software for verifying analysis results against a waste identification database of a domestic municipality.

[1540] A "feedback mechanism" is a device or system that provides the user with the correct sorting method based on the matching results.

[1541] An "alert means" is a device or system that issues an alert when it detects an error in the feedback provided by the feedback means.

[1542] "Emotion recognition means" refers to a device or software that uses an emotion recognition engine to detect the user's emotional state.

[1543] "Adaptive feedback means" refers to a device or system for adjusting guidance based on the emotional state detected by the emotion recognition means.

[1544] An "interactive concierge system" is a dialogue system that provides users with real-time sorting and guidance.

[1545] A "storage volume measuring device" is a sensor device used to measure the amount of waste accumulated.

[1546] "Prediction means" refers to a device or software for predicting the next collection date based on the accumulated amount.

[1547] "Notification means" refers to a device or system for notifying the user of the prediction results.

[1548] "Remote monitoring means" refers to a device or system that allows a user to remotely monitor the status of waste disposal.

[1549] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. Furthermore, it has a function to recognize the user's emotions and adjust the feedback accordingly. The embodiments of this system are described in detail below.

[1550] System Configuration

[1551] 1. Device (trash can camera and sensors)

[1552] A sensor attached to the trash can detects when trash is placed inside.

[1553] When a mail item is detected, the camera automatically captures an image and sends the image data to a cloud server. This device has a built-in communication module that allows it to send data to the cloud server.

[1554] 2. Cloud Server

[1555] The cloud server receives image data sent from the terminal. The received image data is then passed to a generative AI model on the server for analysis.

[1556] The generating AI model analyzes the images and identifies the type of waste. The analysis results are then compared against a waste identification database used by local municipalities in Japan.

[1557] Based on the matching results, users are provided with feedback on correct waste sorting. This feedback is presented to the user via notification or in real time.

[1558] If incorrect sorting is detected, an alert is issued, and an interactive concierge guides you through the correct sorting method.

[1559] The server also uses an emotion recognition engine to detect the user's emotional state. If the user is feeling stressed or confused, an interactive concierge will provide gentle and helpful guidance.

[1560] Sensors measure the amount of trash accumulated in the trash cans, and this data is sent to a cloud server. The cloud server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[1561] 3. User (using a dedicated application)

[1562] Users can receive feedback and alert notifications, as well as guidance from an interactive concierge, through a dedicated application.

[1563] The application allows for remote monitoring of waste disposal status. This enables family members and caregivers to remotely assist elderly individuals with their waste sorting.

[1564] Specific example

[1565] 1. In the case of correct sorting

[1566] The user throws the plastic bottle into the trash can.

[1567] The device's sensor detects the insertion of the item, and the camera captures an image of the inserted plastic bottle. The captured image data is sent to a cloud server.

[1568] The cloud server passes the image data to a generating AI model, which identifies it as a plastic bottle. A verification method confirms the accuracy of the analysis result.

[1569] The user will be notified via a feedback system that "this is the correct sorting."

[1570] 2. In the case of incorrect sorting

[1571] A user throws a metal can into a trash can, but the analysis results incorrectly classify it as combustible waste.

[1572] The cloud server's emotion recognition engine checks the user's emotional state. If the user notices an error and is feeling stressed, an alert is issued.

[1573] Through an interactive concierge system, you'll be gently guided with instructions like, "This is metal waste. Please classify it correctly."

[1574] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[1575] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1576] Step 1:

[1577] Garbage disposal detection and image capture

[1578] The device uses a sensor to detect when waste is placed in the trash can. When a user places waste in the trash can, the sensor detects this action. At that time, the sensor emits a trigger signal indicating "Trash has been placed in the trash can." Subsequently, the camera that receives this trigger signal automatically captures an image. Captured image data is then generated.

[1579] Input: Garbage disposal action

[1580] Output: Image data of the discarded trash

[1581] Step 2:

[1582] Sending image data to a cloud server

[1583] The device sends the captured image data to the cloud server. The captured image data is uploaded to the cloud server via a communication module. During this process, necessary metadata (such as the time of posting) is also added to the image data.

[1584] Input: Captured image data

[1585] Output: Image data uploaded to the cloud server

[1586] Step 3:

[1587] Image analysis

[1588] The server passes the received image data to a generative AI model. The cloud server inputs the received image data into the AI ​​model for analysis. This generative AI model analyzes the images and identifies the type of waste. This analysis yields classification results such as "plastic bottles" or "metal cans."

[1589] Input: Image data uploaded to a cloud server

[1590] Output: Analysis results (type of waste)

[1591] Step 4:

[1592] Matching and Feedback

[1593] The server compares the analysis results with the waste identification database of local municipalities in Japan. The analyzed data is compared with the database to confirm the correct sorting method. A feedback system notifies the user about the correct sorting method. This notification is provided through a dedicated application.

[1594] Input: Analysis results

[1595] Output: Feedback on waste sorting

[1596] Step 5:

[1597] Alerts and interactive concierge

[1598] The server issues an alert if it detects incorrect sorting. When the matching system finds incorrect sorting, it sends a warning to the user through the notification system. Simultaneously, an interactive concierge is activated to guide the user on the correct sorting method.

[1599] Input: Matching results (error detection)

[1600] Output: Alert notifications and interactive concierge feedback

[1601] Step 6:

[1602] Emotion recognition and adaptive feedback

[1603] The server uses an emotion recognition engine to detect the user's emotional state. It analyzes the user's responses to feedback and alerts, and if stress or confusion is detected, the conversational concierge adjusts the tone and content of the feedback.

[1604] Input: User response data

[1605] Output: Feedback adapted to emotional state

[1606] Step 7:

[1607] Accumulated amount and prediction of the next collection date

[1608] A sensor measures the amount of trash in the bins and sends the data to a server. The server predicts the next collection date based on the amount of trash and the local government's collection schedule. The prediction result is then notified to the user through a dedicated application.

[1609] Input: Accumulated data

[1610] Output: Predicted notification of the next collection date

[1611] Step 8:

[1612] Remote monitoring

[1613] Users can monitor the status of their garbage disposal in real time through a dedicated application. The monitoring function within the application allows users and their families to remotely check the disposal status. This function makes it possible to remotely assist elderly people with their garbage sorting.

[1614] Input: Garbage disposal data

[1615] Output: Real-time mailing status display

[1616] (Application Example 2)

[1617] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1618] Traditional waste sorting methods often failed to properly separate waste, resulting in reduced recycling efficiency and potentially exacerbating social and environmental problems. Furthermore, elderly and disabled individuals frequently encountered difficulties with sorting, and there was a lack of support for proper sorting, particularly in public facilities and retail stores. Additionally, managing waste bin levels and timely collection proved challenging, leading to frequent hygiene issues. A solution was needed to address these problems while reducing the burden on users.

[1619] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing guidance to the user in real time, an emotion recognition means for recognizing the user's emotions and adjusting the feedback, a notification means for visually or audibly guiding the user on how to sort waste, and a monitoring means for monitoring the status of waste disposal with a selected device. This enables users to sort waste accurately and easily, improving the accuracy of the sorting system and reducing the environmental burden in physical stores, etc. Furthermore, it becomes possible to manage the amount of waste accumulated in trash cans and predict the timing of collection, thus resolving hygiene issues.

[1620] "Image capture means for detecting deposited waste" refers to a device that captures deposited waste as an image using a camera or other imaging device.

[1621] "Image transmission means for sending image data to a cloud server" refers to a device or mechanism for transmitting acquired image data to a cloud server via the internet.

[1622] "Generative artificial intelligence means" refers to a means that uses artificial intelligence technology to analyze acquired image data and automatically recognize and classify the type of waste.

[1623] "Verification method" refers to a method of verifying the results classified by artificial intelligence against the waste identification database of local municipalities within Japan.

[1624] A "feedback mechanism" is a device or mechanism that informs the user of the correct disposal method or areas for improvement based on the verification results.

[1625] An "alert device" is a device or mechanism that warns the user or system when an error in the feedback is detected.

[1626] An "interactive concierge service" is an interactive support method that provides users with real-time guidance on things like how to sort their waste.

[1627] "Emotion recognition means" refers to technology that recognizes a user's facial expressions, voice, etc., and analyzes their emotional state.

[1628] "Notification means for guiding users on how to sort waste visually or audibly" refers to means of instructing users on how to sort waste visually or audibly using displays or speakers.

[1629] A "monitoring device" refers to a device or mechanism that continuously monitors the status of waste disposal and the amount of waste accumulated in the trash cans using a selected device.

[1630] "Accumulation amount measuring means" refers to sensors or devices used to measure the amount of waste in a trash can.

[1631] A "prediction tool" is a device or program used to calculate the next collection date based on data on the amount of waste accumulated.

[1632] A "notification means" is a device or mechanism for informing users or administrators of predicted results or important information.

[1633] "Remote monitoring means" refers to technologies and devices that allow users to monitor the status of garbage disposal from a remote location.

[1634] System Configuration

[1635] This invention is a system that detects deposited waste and supports users in proper waste sorting. This system consists of the following main hardware and software components.

[1636] hardware

[1637] 1. Camera (e.g., Logitech C920)

[1638] A device for capturing images of garbage that has been dropped off.

[1639] 2. Pressure sensor (e.g., Phidgets 3132)

[1640] A device for detecting the dropping off of trash.

[1641] 3. Display monitor (Example: LG 24MP88HV-S)

[1642] A device for providing visual feedback to users.

[1643] 4. Smartphone (e.g., iPhone 12)

[1644] A mobile device for running dedicated applications.

[1645] software

[1646] 1. Cloud server (e.g., Amazon Web Services)

[1647] A server that receives and analyzes image data to recognize the type of waste.

[1648] 2. Image analysis models (e.g., TensorFlow)

[1649] An artificial intelligence model for identifying the type of waste using acquired image data.

[1650] 3. Emotion recognition engine (e.g., IBM Watson Emotion Analysis)

[1651] An engine that analyzes user emotions and provides appropriate feedback.

[1652] 4. Database (e.g., PostgreSQL)

[1653] It will be used as a waste identification database and compared with the analysis results on the cloud server.

[1654] 5. Dedicated application (e.g., developed using Swift or Kotlin)

[1655] An application for users to monitor their waste sorting status.

[1656] System operation

[1657] 1. Detecting and capturing images of trash being dropped off.

[1658] When the device's pressure sensor detects the dropping off of trash, the camera automatically captures an image. This image data is then sent to a cloud server.

[1659] 2. Image Analysis

[1660] The cloud server passes the received image data to a TensorFlow model, which analyzes the type of garbage.

[1661] 3. Matching and Feedback

[1662] The analysis results are compared with a PostgreSQL database to provide users with feedback on correct waste sorting. This feedback is displayed visually on a display monitor or smartphone.

[1663] 4. Alerts and interactive concierge

[1664] If incorrect sorting is detected, the system will issue an alert and activate an interactive concierge to provide detailed guidance.

[1665] 5. Emotion Recognition and Adaptive Feedback

[1666] IBM Watson's emotion recognition engine analyzes the user's facial expressions and voice, providing appropriate feedback based on their emotions.

[1667] 6. Prediction of accumulated amount and next collection date

[1668] A pressure sensor periodically measures the amount of trash in the trash can and sends the data to a cloud server. The cloud server predicts the next collection date based on the amount of trash and the collection schedule, and notifies the user.

[1669] 7. Remote Monitoring

[1670] Through a dedicated application, users can remotely monitor the status of their trash disposal in real time.

[1671] Specific example

[1672] 1. In the case of correct sorting

[1673] The user throws the plastic bottle into the recycling bin.

[1674] A sensor detects the mail being dropped in, and a camera takes a picture.

[1675] A cloud server analyzes the image and identifies it as a plastic bottle.

[1676] The display shows "This is recyclable waste, thank you!"

[1677] 2. In the case of incorrect sorting

[1678] The user throws the aluminum can into the recycling bin.

[1679] Analysis of the cloud server identified this as metal waste.

[1680] The display shows, "This is metal waste. Please dispose of it in the correct trash can."

[1681] If the user makes a displeased expression, the emotion recognition engine will detect it and gently say, "Recycling is important, let's try again!"

[1682] Example of a prompt

[1683] The following is an example of prompt statements used when analyzing waste using a generative AI model.

[1684] prompt:

[1685] New image data has been received. Please identify the type of trash shown in this image.

[1686] Image data:

[1687] [Base64 encoded image data]

[1688] Database reference:

[1689] Please use the latest database for classifying waste types.

[1690] This system enables users to sort waste easily and accurately, improving the precision of sorting procedures and reducing environmental impact in physical stores and other locations. Furthermore, it allows for management of waste bin volume and prediction of collection timing, resolving hygiene issues.

[1691] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1692] Step 1:

[1693] The device uses a pressure sensor to detect the dropping of trash. When trash is detected, the camera automatically captures an image and sends the image data to a cloud server. The input is the dropping of trash, and the output is the image data sent to the cloud server.

[1694] Step 2:

[1695] The server passes the received image data to a generating AI model for image analysis. Specifically, it uses a TensorFlow model to identify the type of waste. In this process, the input is image data, and the output is the analysis result indicating the type of waste.

[1696] Step 3:

[1697] The server compares the analysis results with the PostgreSQL database to verify the correct classification. The input is the analysis results and database information, and the output is the comparison result. If the comparison result is correct, feedback is provided to the user based on that information.

[1698] Step 4:

[1699] The feedback mechanism provides users with visual feedback via a display monitor or smartphone. The input is the matching result from a cloud server, and the output is a feedback message to the user. For example, it might display, "This is recyclable waste, thank you!"

[1700] Step 5:

[1701] If incorrect sorting is detected, the server issues an alert, and an interactive concierge begins providing detailed instructions. The input is the error detection result, and the output is the alert notification and the activation of the interactive concierge. The concierge displays on the screen, "This is metal waste. Please dispose of it in the correct trash can."

[1702] Step 6:

[1703] The server uses IBM Watson's emotion recognition engine to analyze the user's facial expressions and voice to determine their emotional state. Input is the user's facial expression data and voice, and output is the emotion analysis result. Appropriate feedback is provided based on the user's emotions.

[1704] Step 7:

[1705] The server periodically collects data from pressure sensors to measure the amount of waste in the trash cans. The input is pressure sensor data, and the output is the amount of waste. Based on this data, the server predicts the next collection date and notifies users and administrators.

[1706] Step 8:

[1707] Users can remotely monitor the status of their trash collection in real time through a dedicated application. Input is data from a cloud server, and output is the collection status displayed on the application. This makes it possible to manage trash even from a distance.

[1708] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1709] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1710] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1711] [Fourth Embodiment]

[1712] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1713] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1714] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1715] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1716] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1717] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1718] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1719] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1720] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1721] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1722] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1723] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1724] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1725] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. The details of the system's program processing and specific examples are described below.

[1726] System Configuration

[1727] 1. Device (trash can camera and sensors)

[1728] It is equipped with a sensor that detects when trash is dropped in and a camera that captures images of the dropped-in trash.

[1729] It has the function to send captured image data to a cloud server.

[1730] 2. Cloud Server

[1731] The system receives image data and uses artificial intelligence (AI) to identify the type of waste.

[1732] The analysis results are compared with the waste identification database of local governments in Japan, and feedback is provided to users.

[1733] If an incorrect sorting is detected, an alert will be issued and an interactive concierge will be activated.

[1734] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[1735] 3. Users (using a dedicated application)

[1736] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[1737] It is possible to monitor the status of garbage disposal remotely.

[1738] Program processing

[1739] This program will be executed in the following steps.

[1740] 1. Detecting and capturing images of trash being dropped off.

[1741] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[1742] The device sends the captured image data to the cloud server.

[1743] 2. Image Analysis

[1744] The server receives image data and passes it to the artificial intelligence model for generating data.

[1745] An AI model analyzes the image and identifies the type of waste.

[1746] 3. Matching and Feedback

[1747] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[1748] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[1749] 4. Alerts and interactive concierge

[1750] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[1751] 5. Prediction of accumulated amount and next collection date

[1752] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[1753] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[1754] 6. Remote Monitoring

[1755] Users can monitor the status of their trash collection in real time through a dedicated application.

[1756] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[1757] Specific example

[1758] 1. In the case of correct sorting

[1759] The user throws the plastic bottle into the trash can.

[1760] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[1761] The server analyzes the image and identifies it as plastic.

[1762] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[1763] 2. In the case of incorrect sorting

[1764] A user threw a metal can into a trash can, but the cloud server analysis and comparison results detected an error (for example, classifying it as combustible waste).

[1765] An alert system issues an alert, and an interactive concierge guides the user on the correct way to sort metal cans for disposal.

[1766] This invention will improve the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, it will enable family members and caregivers living separately to monitor waste sorting status in real time and provide appropriate support.

[1767] The following describes the processing flow.

[1768] Step 1:

[1769] Terminal: "Detects the dropping off of trash"

[1770] A sensor installed in the trash can detects when trash is placed inside.

[1771] When trash is detected being dropped in, the sensor activates the camera.

[1772] Step 2:

[1773] Device: "Capture image"

[1774] The camera captures an image of the trash that has been dropped off.

[1775] The captured image data is temporarily stored inside the device.

[1776] Step 3:

[1777] Terminal: "Send image data to the cloud server"

[1778] The device sends the captured image data to the cloud server.

[1779] Once the transmission is complete, the device waits for a response from the cloud server.

[1780] Step 4:

[1781] Server: "Receive image data"

[1782] The cloud server receives the image data.

[1783] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[1784] Step 5:

[1785] Server: "Perform image analysis"

[1786] The AI ​​model analyzes the image data.

[1787] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[1788] Step 6:

[1789] Server: "Compare analysis results with the waste identification database."

[1790] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[1791] Through verification, we determine whether the waste has been sorted appropriately.

[1792] Step 7:

[1793] Server: "Provide feedback to users"

[1794] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[1795] Feedback is sent to users through a dedicated application.

[1796] Step 8:

[1797] Device: "Receive and display feedback"

[1798] The terminal receives feedback from the server.

[1799] Display the received feedback to the user.

[1800] Step 9:

[1801] Server: "Issues an alert in case of incorrect sorting."

[1802] If incorrect classification is detected, the server will immediately issue an alert.

[1803] The alert will point out the error to the user and provide the correct sorting method.

[1804] Step 10:

[1805] Terminal: "Launch the interactive concierge"

[1806] When an alert is issued, the device activates its interactive concierge function.

[1807] The concierge function provides users with guidance and support in an interactive format.

[1808] Step 11:

[1809] Terminal: "Measure accumulated amount"

[1810] A sensor inside the trash can measures the amount of waste accumulated.

[1811] The measured data is sent to the cloud server.

[1812] Step 12:

[1813] Server: "Predict the next collection date"

[1814] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[1815] The prediction results are notified to the user through a dedicated application.

[1816] Step 13:

[1817] User: "Perform remote monitoring"

[1818] A dedicated application is used to monitor the status of garbage disposal in real time.

[1819] Users can assist elderly individuals with sorting waste as needed.

[1820] The above describes the specific flow of program processing in the system of the present invention.

[1821] (Example 1)

[1822] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1823] Proper waste sorting within the home is difficult for many people, and mistakes are frequent, especially among the elderly and busy households. As a result, the efficiency and accuracy of waste disposal decline, making it difficult for local governments to collect and recycle waste. On the other hand, there are limited ways for family members living far away to support elderly people with waste sorting. There is a need to solve this problem and improve the accuracy of waste sorting.

[1824] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1825] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for sending the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a storage amount measuring means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. This improves the accuracy of waste sorting, streamlines waste disposal, and enables support from remote locations.

[1826] A "sensor means" refers to a device or function used to detect the dropping of trash.

[1827] "Camera means" refers to a photographic device or function used to capture images of the trash that has been dropped off.

[1828] "Image transmission means" refers to a device or function for transmitting captured image data to a cloud server.

[1829] "Generative artificial intelligence means" refers to AI technology and functions used to analyze image data and classify waste.

[1830] "Verification means" refers to a device or function for verifying analysis results against a waste identification database.

[1831] A "feedback mechanism" refers to a device or function that provides feedback to the user based on the analysis results.

[1832] An "alert mechanism" is a device or function that issues an alert when an error in the feedback is detected.

[1833] "Interactive concierge services" refer to conversational systems and functions that provide real-time guidance to users.

[1834] "Accumulation amount measuring means" refers to a device or function for measuring the accumulated amount of waste.

[1835] A "prediction tool" is a function for predicting the next collection date based on the measured amount of accumulated waste.

[1836] "Notification means" refers to devices or functions used to notify the user of the prediction results.

[1837] "Remote monitoring means" refers to devices or functions that allow users to remotely monitor the status of waste disposal.

[1838] This invention provides an IoT system that assists with household waste sorting, and in particular, provides a configuration for achieving accurate waste classification and sorting. The system detects the waste that is dropped off, analyzes its image to identify the type of waste, and notifies the user of the correct sorting method.

[1839] System Configuration

[1840] This system includes the following main components and means:

[1841] 1. Device (trash can sensor and camera)

[1842] Sensors: Installed to detect the dropping of trash. Infrared sensors and ultrasonic sensors are used.

[1843] Camera method: Capture images of the dropped-off trash. A high-resolution camera is used to photograph the dropped-off trash.

[1844] 2. Cloud Server

[1845] Image transmission method: Captured image data is sent to a cloud server. This utilizes a Wi-Fi module and transmits data using the HTTP protocol.

[1846] Generative AI method: Analyzes received image data to identify the type of waste. For this purpose, AI technologies such as the Google Cloud Vision API are used.

[1847] Verification method: The analysis results are compared with a waste identification database. Based on the verification results, the correct waste sorting method is confirmed.

[1848] Feedback method: Notify the user of the feedback results. For example, use push notifications to notify a dedicated application.

[1849] Alerting method: An alert is issued when an error in feedback is detected. The alert is sent to the user via push notification.

[1850] Interactive concierge system: A dialogue system that provides real-time guidance to users. This may utilize services such as Amazon Lex.

[1851] 3. Managing the amount of trash in the trash cans and the collection schedule.

[1852] Accumulation amount measurement method: Measures the amount of accumulated waste in the trash can. The amount of waste is measured using an ultrasonic sensor or similar device.

[1853] Prediction method: Predict the next collection date based on the accumulated amount and the local government's collection schedule. Based on the prediction results, determine the next collection date.

[1854] Notification method: Notify the user of the prediction results. For example, send a notification such as, "The next collection date is in 2 days."

[1855] 4. User Interface

[1856] Remote monitoring method: A dedicated application allows users to monitor the status of garbage disposal in real time. Family members and caregivers can also assist elderly individuals with their garbage sorting.

[1857] Specific example

[1858] In the case of correct sorting

[1859] The user throws the plastic bottle into the trash can.

[1860] The device's sensors detect the mail being dropped in, and the camera captures an image.

[1861] The device sends image data to the cloud server.

[1862] The server analyzes the image and identifies it as plastic.

[1863] If the matching result is correct, the server notifies the user with "Correct classification."

[1864] In case of incorrect sorting

[1865] The user throws the metal can into the trash can.

[1866] The device's sensors detect the mail being dropped in, and the camera captures an image.

[1867] The device sends image data to the cloud server.

[1868] The server analyzes the image to identify incorrect sorting (for example, classifying it as combustible waste).

[1869] The server issues an alert, activates an interactive concierge, and guides the user through the correct sorting method.

[1870] Example of a prompt

[1871] 1. Scenario: Notifying people about the correct sorting of plastic bottles.

[1872] "A plastic bottle has been placed in the trash can. Please ensure proper sorting."

[1873] 2. Scenario: Notification of incorrect sorting of metal cans.

[1874] "A metal can was placed in the trash can, but it was detected as incorrectly sorted. Please provide instructions on the correct sorting method."

[1875] With the above configuration, this system can improve the accuracy of waste sorting and provide users with quick and appropriate support.

[1876] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1877] Detailed explanation of the system's program processing.

[1878] Step 1: Detecting trash and capturing images

[1879] Input: Garbage is placed in the trash can.

[1880] Operation: The device's sensors detect the placement of trash. These sensors use infrared or ultrasonic sensors. When trash is detected, the device's camera automatically captures an image. The image is saved in high-resolution JPEG format and a timestamp is added.

[1881] Output: Captured image data of the trash.

[1882] Step 2: Sending image data to the cloud

[1883] Input: Image data of trash captured on a device.

[1884] Operation: The device sends captured image data to the cloud server. This data is transmitted via the HTTP protocol using the Wi-Fi module. The data is then uploaded to the cloud server via a REST API.

[1885] Output: Image data stored on a cloud server.

[1886] Step 3: Image Analysis

[1887] Input: Image data stored on a cloud server.

[1888] Operation: The server receives image data and passes it to an artificial intelligence model. This AI model uses the Google Cloud Vision API to analyze objects in the image and identify the type of waste. The analysis uses information such as shape, color, and text.

[1889] Output: Type of waste (e.g., plastic, metal, paper, etc.).

[1890] Step 4: Matching and Feedback

[1891] Input: The type of waste analyzed.

[1892] Operation: The server compares the analysis results with the waste identification database of domestic municipalities. A REST API is used to communicate with the database and verify the accuracy of the analysis results.

[1893] If the matching results are accurate, the server uses a dedicated application to send feedback to the user about the correct sorting.

[1894] Output: Feedback notification such as "Correct sorting."

[1895] Step 5: Alerts and Interactive Concierge

[1896] Input: Matching result when an incorrect classification is detected.

[1897] Operation: When the server detects incorrect sorting, it issues an alert. The alert is pushed to a dedicated application. Additionally, an interactive concierge is activated to teach the user the correct sorting method. Amazon Lex is used for the interactive concierge, providing support via voice and text.

[1898] Output: Alert notifications such as "Incorrect sorting detected," and guidance on sorting methods from an interactive concierge.

[1899] Step 6: Measure the amount of trash in the bins and predict the next collection date.

[1900] Input: Data on the amount of waste bin contents.

[1901] Operation: The device's sensors measure the amount of trash accumulated in the trash can. An ultrasonic sensor is used for this, and the data is sent to a cloud server. The server uses this information to compare it with the local government's collection schedule and predict the next collection date.

[1902] Output: Notification of prediction results such as "The next collection date is in 2 days."

[1903] Step 7: Remote Monitoring

[1904] Input: Real-time data on trash disposal and trash can contents.

[1905] Operation: Users monitor the status of their garbage disposal in real time through a dedicated application. The application displays data from a cloud server and allows users to check past disposal history. In addition, family members and caregivers can remotely assist elderly individuals with their garbage sorting.

[1906] Output: Real-time monitoring data display and notification.

[1907] The above outlines the specific steps of the system's program processing and the detailed operation of each step. This system allows users to obtain accurate waste sorting information in real time, improving the efficiency and accuracy of waste disposal.

[1908] (Application Example 1)

[1909] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1910] Sorting waste generated on factory production lines efficiently and accurately is difficult. Traditional manual waste sorting is prone to human error, and further automation and real-time monitoring are needed. In addition, a convenient display system is required so that workers can constantly check the status of waste sorting.

[1911] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1912] In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing real-time guidance to the user, a display means for displaying the waste deposit status in real time on a display device worn by the user, and a mechanical means for automating waste deposit. This enables accurate sorting and real-time monitoring of waste.

[1913] "Image capture means" refers to a device that takes pictures of the discarded waste that has been submitted and acquires the image data.

[1914] "Image transmission means" refers to a device or function that transmits acquired image data to a cloud server.

[1915] "Generative artificial intelligence means" refers to an artificial intelligence model that analyzes received image data to identify the type of waste.

[1916] "Verification means" refers to a method of comparing and verifying the analyzed data with the waste identification database of local municipalities within Japan.

[1917] A "feedback method" is a means of notifying the user of the sorting results based on the matching results.

[1918] An "alert system" is a means of issuing a warning when an error in classification is detected.

[1919] An "interactive concierge service" is an interactive system that guides users in real time on the correct sorting methods.

[1920] "Display means" refers to a means of displaying the status of waste disposal in real time on a display device worn by the user.

[1921] "Mechanical means" refers to a device or function for automating the disposal of waste.

[1922] A "storage unit measurement means" is a means of measuring the amount of waste using a sensor.

[1923] A "prediction method" is a means of predicting the next collection date based on the accumulated amount and collection schedule.

[1924] "Notification means" refers to a means of notifying users of information such as the predicted collection date.

[1925] A "remote monitoring method" is a means that allows users to remotely monitor the status of waste disposal.

[1926] This invention is a system for accurately sorting and managing waste generated on a factory production line. The system provides multiple functions, including waste classification, accumulation measurement, collection date prediction, and real-time monitoring.

[1927] System Configuration

[1928] 1. Terminal (waste disposal robot and camera)

[1929] It is equipped with a robot that manages the disposal of waste and uses sensors to detect waste.

[1930] When waste is detected, the camera captures an image and sends that image data to a cloud server.

[1931] 2. Cloud Server

[1932] The received image data is passed to a "generating artificial intelligence means" to identify the type of waste.

[1933] The identified waste types are cross-referenced with the "Domestic Municipal Waste Identification Database."

[1934] Based on the matching results, provide feedback to the user.

[1935] If there are errors in the feedback results, an alert will be issued and an interactive concierge will guide you to the correct solution.

[1936] The system measures the amount of waste accumulated, predicts the next collection date, and notifies the user.

[1937] 3. Users (using smart glasses or dedicated applications)

[1938] You can check the status of your waste disposal in real time through smart glasses.

[1939] Receive notifications and alerts from the cloud server through a dedicated application.

[1940] It is possible to monitor the status of waste disposal remotely.

[1941] Program processing

[1942] The program for this system is processed as follows:

[1943] 1. Waste disposal detection and image capture

[1944] When the device's sensor detects the dropping off of waste, the camera automatically captures an image.

[1945] The acquired image data is sent to a cloud server.

[1946] For example, we will utilize OpenCV as the hardware to be used.

[1947] 2. Image Analysis

[1948] The cloud server receives image data and passes it to a "generative AI model" to identify the type of waste.

[1949] The analysis algorithm uses artificial intelligence models such as TensorFlow and PyTorch.

[1950] 3. Matching and Feedback

[1951] The analysis results generated by artificial intelligence are compared with the "garbage identification database of domestic municipalities."

[1952] Based on the matching results, feedback is provided to the user.

[1953] Database matching is performed using database management systems such as SQL.

[1954] 4. Alerts and interactive concierge

[1955] If an error in sorting is detected, the server will issue an alert, and an interactive system will guide the user on the correct sorting method.

[1956] The conversational concierge will utilize conversational services such as Google Dialogflow and AWS Lex.

[1957] 5. Measurement of accumulated amount and prediction of the next collection date

[1958] Sensors measure the amount of waste accumulated and transmit that information to a cloud server.

[1959] The server predicts the next collection date based on the accumulated amount and collection schedule, and notifies the user.

[1960] Notifications are delivered via a dedicated application or smart glasses.

[1961] Specific example

[1962] For example, when a worker in a factory drops off metal waste, a sensor detects the drop and a camera captures an image. This image is sent to a cloud server and recognized as metal by a generative AI model. This recognition result is then compared with a waste identification database of local municipalities to confirm the correct sorting method. The worker is then notified via smart glasses that "this is the correct sorting."

[1963] Example of a prompt

[1964] "This is a captured image of waste. Identify the type of waste and select one of the following categories: [Combustible waste, Non-combustible waste, Plastic, Metal, Bottles, Paper]"

[1965] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1966] Step 1:

[1967] The device uses a sensor to detect the placement of waste. The sensor detects that waste has been thrown into the trash can and notifies the device of the event. Based on this information, the camera is activated and an image is captured. The input is the waste placement event, and the output is the captured image data.

[1968] Step 2:

[1969] The device sends the captured image data to the cloud server. The image data is uploaded to the cloud server via the network and prepared for further analysis. The input is the captured image data, and the output is the transmission of the image data to the cloud server.

[1970] Step 3:

[1971] The server passes the received image data to an artificial intelligence (AI) generator. The server inputs the image data into the AI ​​model and performs analysis to identify the type of waste. The input is image data uploaded to the cloud server, and the output is the analyzed waste type information.

[1972] Step 4:

[1973] The server compares the identified waste type with the waste identification database of domestic municipalities. Database matching is performed using a database management system such as SQL. The input is waste type information analyzed by AI, and the output is the matching result.

[1974] Step 5:

[1975] The server provides feedback to the user based on the matching results. This feedback can be received by the user through smart glasses or a dedicated application. The input is the matching result, and the output is the feedback information for the user.

[1976] Step 6:

[1977] If the server detects an error in the feedback result, it issues an alert, and an interactive concierge guides the user through the correct sorting process. The concierge uses tools such as Google Dialogflow or AWS Lex. The input is the error information from the feedback result, and the output is the user's alert and interactive guidance.

[1978] Step 7:

[1979] A sensor measures the amount of waste accumulated and sends this information to a cloud server. The server predicts the next collection date based on the accumulated amount and collection schedule, and notifies the user. The input is the measured amount of waste, and the output is the predicted next collection date.

[1980] Step 8:

[1981] Users can check the real-time status of their waste disposal through smart glasses or a dedicated application. Based on this information, users can take appropriate disposal actions. The input is the waste disposal status, and the output is the real-time displayed disposal information.

[1982] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1983] This invention is a system that supports waste sorting using an IoT device attached to a household trash can, and further includes a function to recognize the user's emotions and adjust the feedback accordingly. The details and specific examples of the system's program processing are described below.

[1984] System Configuration

[1985] 1. Device (trash can camera and sensors)

[1986] It is equipped with a sensor that detects when trash is dropped in and a camera that captures images of the dropped-in trash.

[1987] It has the function to send captured image data to a cloud server.

[1988] 2. Cloud Server

[1989] The system receives image data and uses artificial intelligence (AI) to identify the type of waste.

[1990] The analysis results are compared with the waste identification database of local governments in Japan, and feedback is provided to users.

[1991] If an incorrect sorting is detected, an alert will be issued and an interactive concierge will be activated.

[1992] The system predicts the amount of trash in the bins and the next collection date, and notifies the user.

[1993] It features an emotion recognition engine that recognizes user emotions and appropriately adjusts feedback and guidance.

[1994] 3. Users (using a dedicated application)

[1995] Receive feedback and alert notifications, and get guidance from an interactive concierge.

[1996] It is possible to monitor the status of garbage disposal remotely.

[1997] Program processing

[1998] This program will be executed in the following steps.

[1999] 1. Detecting and capturing images of trash being dropped off.

[2000] When the device's sensor detects that trash has been dropped in, the camera automatically captures an image.

[2001] The device sends the captured image data to the cloud server.

[2002] 2. Image Analysis

[2003] The server receives image data and passes it to the artificial intelligence model for generating data.

[2004] An AI model analyzes the image and identifies the type of waste.

[2005] 3. Matching and Feedback

[2006] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[2007] Based on the matching results, the system provides users with feedback on how to properly sort their waste.

[2008] 4. Alerts and interactive concierge

[2009] If incorrect sorting is detected, the server will issue an alert and guide the user through an interactive concierge to the correct sorting method.

[2010] 5. Emotion Recognition and Adaptive Feedback

[2011] The server's emotion recognition engine detects the user's emotional state.

[2012] If a user is feeling stressed or confused, an interactive concierge will provide gentle and courteous guidance.

[2013] 6. Prediction of accumulated amount and next collection date

[2014] A sensor measures the amount of trash accumulated in the trash can and sends that information to a server.

[2015] The server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[2016] 7. Remote Monitoring

[2017] Users can monitor the status of their trash collection in real time through a dedicated application.

[2018] Family members and caregivers can remotely assist elderly individuals with their waste sorting practices.

[2019] Specific example

[2020] 1. In the case of correct sorting

[2021] The user throws the plastic bottle into the trash can.

[2022] The device's sensors detect the mail being dropped in, and the camera captures an image and sends it to a cloud server.

[2023] The server analyzes the image and identifies it as plastic.

[2024] If the matching result is correct, a feedback message will be sent stating, "Correct sorting."

[2025] 2. In the case of incorrect sorting

[2026] A user throws a metal can into a trash can, but the cloud server's analysis and matching results detect an error (for example, classifying it as combustible waste).

[2027] The server uses an emotion recognition engine to check the user's emotional state.

[2028] An alert is issued, and an interactive concierge calmly instructs, "This is metal waste. Please sort it correctly."

[2029] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[2030] The following describes the processing flow.

[2031] Step 1:

[2032] Terminal: "Detects the dropping off of trash"

[2033] A sensor installed in the trash can detects when trash is placed inside.

[2034] When trash is detected being dropped in, the sensor activates the camera.

[2035] Step 2:

[2036] Device: "Capture image"

[2037] The camera captures an image of the trash that has been dropped off.

[2038] The captured image data is temporarily stored inside the device.

[2039] Step 3:

[2040] Terminal: "Send image data to the cloud server"

[2041] The device sends the captured image data to the cloud server.

[2042] Once the transmission is complete, the device waits for a response from the cloud server.

[2043] Step 4:

[2044] Server: "Receive image data"

[2045] The cloud server receives the image data.

[2046] After receiving confirmation, the image data is provided to the generated artificial intelligence (AI) model.

[2047] Step 5:

[2048] Server: "Perform image analysis"

[2049] The AI ​​model analyzes the image data.

[2050] Through analysis, the type of waste (e.g., plastic, metal, paper, etc.) is identified.

[2051] Step 6:

[2052] Server: "Compare analysis results with the waste identification database."

[2053] The analysis results are compared with the waste identification databases of local municipalities within Japan.

[2054] Through verification, we determine whether the waste has been sorted appropriately.

[2055] Step 7:

[2056] Server: "Provide feedback to users"

[2057] Based on the matching results, the system will notify the user if the sorting is correct, or instruct them to correct it if it is incorrect.

[2058] Feedback is sent to users through a dedicated application.

[2059] Step 8:

[2060] Device: "Receive and display feedback"

[2061] The terminal receives feedback from the server.

[2062] Display the received feedback to the user.

[2063] Step 9:

[2064] Server: "Recognize emotions"

[2065] The server's emotion recognition engine detects the user's response to the feedback.

[2066] Evaluate whether users are experiencing stress or confusion.

[2067] Step 10:

[2068] Server: "Perform feedback adjustments"

[2069] Based on evaluations by the emotion recognition engine, the interactive concierge system adjusts the feedback content.

[2070] If the user is feeling stressed, provide detailed instructions in a gentle tone.

[2071] Step 11:

[2072] Server: "Issues an alert in case of incorrect sorting."

[2073] If incorrect classification is detected, the server will immediately issue an alert.

[2074] The alert will point out the error to the user and provide the correct sorting method.

[2075] Step 12:

[2076] Terminal: "Launch the interactive concierge"

[2077] When an alert is issued, the device activates its interactive concierge function.

[2078] The concierge function provides users with guidance and support in an interactive format.

[2079] Step 13:

[2080] Terminal: "Measure accumulated amount"

[2081] A sensor inside the trash can measures the amount of waste accumulated.

[2082] The measured data is sent to the cloud server.

[2083] Step 14:

[2084] Server: "Predict the next collection date"

[2085] Based on accumulated data and the collection schedule of local governments, the next collection date is predicted.

[2086] The prediction results are notified to the user through a dedicated application.

[2087] Step 15:

[2088] User: "Perform remote monitoring"

[2089] A dedicated application is used to monitor the status of garbage disposal in real time.

[2090] Users can assist elderly individuals with sorting waste as needed.

[2091] The above describes the specific flow of program processing in the system of the present invention.

[2092] (Example 2)

[2093] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2094] In modern society, waste sorting is a crucial environmental issue, but many people are confused about sorting methods, leading to frequent missorting. Elderly people and those unfamiliar with sorting are particularly prone to stress and confusion during the process. Furthermore, managing the amount of waste accumulated in households and knowing the next collection date is difficult, making timely disposal challenging. Additionally, family members and caregivers living far away have limited means of monitoring waste sorting status. A system is needed to address these challenges and improve the efficiency and accuracy of waste sorting.

[2095] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2096] In this invention, the server includes a sensor means for detecting deposited waste, a camera means for capturing images of the deposited waste, an image transmission means for transmitting the captured image data to a cloud server, a generative artificial intelligence means for analyzing the image data to identify the type of waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an emotion recognition means for detecting the user's emotional state using an emotion recognition engine, an adaptive feedback means for adjusting guidance based on the emotion recognition means, an interactive concierge means for providing guidance to the user in real time, a storage amount measurement means for measuring the amount of accumulated waste, a prediction means for predicting the next collection date based on the storage amount, a notification means for notifying the user of the prediction results, and a remote monitoring means for the user to remotely monitor the status of waste deposits. As a result, the accuracy of waste sorting is improved, the efficiency of waste processing is increased, user stress is reduced, and sorting work can be carried out smoothly. Furthermore, it becomes easier to provide support because family members or caregivers in remote locations can monitor the mailing status.

[2097] A "sensor device" is a device used to detect waste placed in a trash can.

[2098] A "camera device" is a photographic device used to capture images of the discarded waste that has been submitted.

[2099] "Image transmission means" refers to a communication device for sending captured image data to a cloud server.

[2100] "Generative artificial intelligence means" refers to artificial intelligence technology used to analyze received image data and identify the type of waste.

[2101] "Verification means" refers to a device or software for verifying analysis results against a waste identification database of a domestic municipality.

[2102] A "feedback mechanism" is a device or system that provides the user with the correct sorting method based on the matching results.

[2103] An "alert means" is a device or system that issues an alert when it detects an error in the feedback provided by the feedback means.

[2104] "Emotion recognition means" refers to a device or software that uses an emotion recognition engine to detect the user's emotional state.

[2105] "Adaptive feedback means" refers to a device or system for adjusting guidance based on the emotional state detected by the emotion recognition means.

[2106] An "interactive concierge system" is a dialogue system that provides users with real-time sorting and guidance.

[2107] A "storage volume measuring device" is a sensor device used to measure the amount of waste accumulated.

[2108] "Prediction means" refers to a device or software for predicting the next collection date based on the accumulated amount.

[2109] "Notification means" refers to a device or system for notifying the user of the prediction results.

[2110] "Remote monitoring means" refers to a device or system that allows a user to remotely monitor the status of waste disposal.

[2111] This invention is a system that supports waste sorting using an IoT device attached to a household trash can. Furthermore, it has a function to recognize the user's emotions and adjust the feedback accordingly. The embodiments of this system are described in detail below.

[2112] System Configuration

[2113] 1. Device (trash can camera and sensors)

[2114] A sensor attached to the trash can detects when trash is placed inside.

[2115] When a mail item is detected, the camera automatically captures an image and sends the image data to a cloud server. This device has a built-in communication module that allows it to send data to the cloud server.

[2116] 2. Cloud Server

[2117] The cloud server receives image data sent from the terminal. The received image data is then passed to a generative AI model on the server for analysis.

[2118] The generating AI model analyzes the images and identifies the type of waste. The analysis results are then compared against a waste identification database used by local municipalities in Japan.

[2119] Based on the matching results, users are provided with feedback on correct waste sorting. This feedback is presented to the user via notification or in real time.

[2120] If incorrect sorting is detected, an alert is issued, and an interactive concierge guides you through the correct sorting method.

[2121] The server also uses an emotion recognition engine to detect the user's emotional state. If the user is feeling stressed or confused, an interactive concierge will provide gentle and helpful guidance.

[2122] Sensors measure the amount of trash accumulated in the trash cans, and this data is sent to a cloud server. The cloud server predicts the next collection date based on the accumulated amount and the local government's collection schedule, and notifies the user.

[2123] 3. User (using a dedicated application)

[2124] Users can receive feedback and alert notifications, as well as guidance from an interactive concierge, through a dedicated application.

[2125] The application allows for remote monitoring of waste disposal status. This enables family members and caregivers to remotely assist elderly individuals with their waste sorting.

[2126] Specific example

[2127] 1. In the case of correct sorting

[2128] The user throws the plastic bottle into the trash can.

[2129] The device's sensor detects the insertion of the item, and the camera captures an image of the inserted plastic bottle. The captured image data is sent to a cloud server.

[2130] The cloud server passes the image data to a generating AI model, which identifies it as a plastic bottle. A verification method confirms the accuracy of the analysis result.

[2131] The user will be notified via a feedback system that "this is the correct sorting."

[2132] 2. In the case of incorrect sorting

[2133] A user throws a metal can into a trash can, but the analysis results incorrectly classify it as combustible waste.

[2134] The cloud server's emotion recognition engine checks the user's emotional state. If the user notices an error and is feeling stressed, an alert is issued.

[2135] Through an interactive concierge system, you'll be gently guided with instructions like, "This is metal waste. Please classify it correctly."

[2136] This invention improves the accuracy of waste sorting among the elderly, thereby enhancing the efficiency and accuracy of waste disposal for society as a whole. Furthermore, by providing appropriate feedback and guidance that responds to the user's emotions, it reduces the stress of sorting and improves user convenience.

[2137] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2138] Step 1:

[2139] Garbage disposal detection and image capture

[2140] The device uses a sensor to detect when waste is placed in the trash can. When a user places waste in the trash can, the sensor detects this action. At that time, the sensor emits a trigger signal indicating "Trash has been placed in the trash can." Subsequently, the camera that receives this trigger signal automatically captures an image. Captured image data is then generated.

[2141] Input: Garbage disposal action

[2142] Output: Image data of the discarded trash

[2143] Step 2:

[2144] Sending image data to a cloud server

[2145] The device sends the captured image data to the cloud server. The captured image data is uploaded to the cloud server via a communication module. During this process, necessary metadata (such as the time of posting) is also added to the image data.

[2146] Input: Captured image data

[2147] Output: Image data uploaded to the cloud server

[2148] Step 3:

[2149] Image analysis

[2150] The server passes the received image data to a generative AI model. The cloud server inputs the received image data into the AI ​​model for analysis. This generative AI model analyzes the images and identifies the type of waste. This analysis yields classification results such as "plastic bottles" or "metal cans."

[2151] Input: Image data uploaded to a cloud server

[2152] Output: Analysis results (type of waste)

[2153] Step 4:

[2154] Matching and Feedback

[2155] The server compares the analysis results with the waste identification database of local municipalities in Japan. The analyzed data is compared with the database to confirm the correct sorting method. A feedback system notifies the user about the correct sorting method. This notification is provided through a dedicated application.

[2156] Input: Analysis results

[2157] Output: Feedback on waste sorting

[2158] Step 5:

[2159] Alerts and interactive concierge

[2160] The server issues an alert if it detects incorrect sorting. When the matching system finds incorrect sorting, it sends a warning to the user through the notification system. Simultaneously, an interactive concierge is activated to guide the user on the correct sorting method.

[2161] Input: Matching results (error detection)

[2162] Output: Alert notifications and interactive concierge feedback

[2163] Step 6:

[2164] Emotion recognition and adaptive feedback

[2165] The server uses an emotion recognition engine to detect the user's emotional state. It analyzes the user's responses to feedback and alerts, and if stress or confusion is detected, the conversational concierge adjusts the tone and content of the feedback.

[2166] Input: User response data

[2167] Output: Feedback adapted to emotional state

[2168] Step 7:

[2169] Accumulated amount and prediction of the next collection date

[2170] A sensor measures the amount of trash in the bins and sends the data to a server. The server predicts the next collection date based on the amount of trash and the local government's collection schedule. The prediction result is then notified to the user through a dedicated application.

[2171] Input: Accumulated data

[2172] Output: Predicted notification of the next collection date

[2173] Step 8:

[2174] Remote monitoring

[2175] Users can monitor the status of their garbage disposal in real time through a dedicated application. The monitoring function within the application allows users and their families to remotely check the disposal status. This function makes it possible to remotely assist elderly people with their garbage sorting.

[2176] Input: Garbage disposal data

[2177] Output: Real-time mailing status display

[2178] (Application Example 2)

[2179] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2180] Traditional waste sorting methods often failed to properly separate waste, resulting in reduced recycling efficiency and potentially exacerbating social and environmental problems. Furthermore, elderly and disabled individuals frequently encountered difficulties with sorting, and there was a lack of support for proper sorting, particularly in public facilities and retail stores. Additionally, managing waste bin levels and timely collection proved challenging, leading to frequent hygiene issues. A solution was needed to address these problems while reducing the burden on users.

[2181] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes an image capture means for detecting deposited waste, an image transmission means for transmitting the image data of the deposited waste to a cloud server, a generation artificial intelligence means for analyzing the image data to classify the waste, a matching means for comparing the analysis results with a waste identification database of domestic municipalities, a feedback means for providing feedback to the user based on the matching results, an alert means for issuing an alert when an error in the feedback by the feedback means is detected, an interactive concierge means for providing guidance to the user in real time, an emotion recognition means for recognizing the user's emotions and adjusting the feedback, a notification means for visually or audibly guiding the user on how to sort waste, and a monitoring means for monitoring the status of waste disposal with a selected device. This enables users to sort waste accurately and easily, improving the accuracy of the sorting system and reducing the environmental burden in physical stores, etc. Furthermore, it becomes possible to manage the amount of waste accumulated in trash cans and predict the timing of collection, thus resolving hygiene issues.

[2182] "Image capture means for detecting deposited waste" refers to a device that captures deposited waste as an image using a camera or other imaging device.

[2183] "Image transmission means for sending image data to a cloud server" refers to a device or mechanism for transmitting acquired image data to a cloud server via the internet.

[2184] "Generative artificial intelligence means" refers to a means that uses artificial intelligence technology to analyze acquired image data and automatically recognize and classify the type of waste.

[2185] "Verification method" refers to a method of verifying the results classified by artificial intelligence against the waste identification database of local municipalities within Japan.

[2186] A "feedback mechanism" is a device or mechanism that informs the user of the correct disposal method or areas for improvement based on the verification results.

[2187] An "alert device" is a device or mechanism that warns the user or system when an error in the feedback is detected.

[2188] An "interactive concierge service" is an interactive support method that provides users with real-time guidance on things like how to sort their waste.

[2189] "Emotion recognition means" refers to technology that recognizes a user's facial expressions, voice, etc., and analyzes their emotional state.

[2190] "Notification means for guiding users on how to sort waste visually or audibly" refers to means of instructing users on how to sort waste visually or audibly using displays or speakers.

[2191] A "monitoring device" refers to a device or mechanism that continuously monitors the status of waste disposal and the amount of waste accumulated in the trash cans using a selected device.

[2192] "Accumulation amount measuring means" refers to sensors or devices used to measure the amount of waste in a trash can.

[2193] A "prediction tool" is a device or program used to calculate the next collection date based on data on the amount of waste accumulated.

[2194] A "notification means" is a device or mechanism for informing users or administrators of predicted results or important information.

[2195] "Remote monitoring means" refers to technologies and devices that allow users to monitor the status of garbage disposal from a remote location.

[2196] System Configuration

[2197] This invention is a system that detects deposited waste and supports users in proper waste sorting. This system consists of the following main hardware and software components.

[2198] hardware

[2199] 1. Camera (e.g., Logitech C920)

[2200] A device for capturing images of garbage that has been dropped off.

[2201] 2. Pressure sensor (e.g., Phidgets 3132)

[2202] A device for detecting the dropping off of trash.

[2203] 3. Display monitor (Example: LG 24MP88HV-S)

[2204] A device for providing visual feedback to users.

[2205] 4. Smartphone (e.g., iPhone 12)

[2206] A mobile device for running dedicated applications.

[2207] software

[2208] 1. Cloud server (e.g., Amazon Web Services)

[2209] A server that receives and analyzes image data to recognize the type of waste.

[2210] 2. Image analysis models (e.g., TensorFlow)

[2211] An artificial intelligence model for identifying the type of waste using acquired image data.

[2212] 3. Emotion recognition engine (e.g., IBM Watson Emotion Analysis)

[2213] An engine that analyzes user emotions and provides appropriate feedback.

[2214] 4. Database (e.g., PostgreSQL)

[2215] It will be used as a waste identification database and compared with the analysis results on the cloud server.

[2216] 5. Dedicated application (e.g., developed using Swift or Kotlin)

[2217] An application for users to monitor their waste sorting status.

[2218] System operation

[2219] 1. Detecting and capturing images of trash being dropped off.

[2220] When the device's pressure sensor detects the dropping off of trash, the camera automatically captures an image. This image data is then sent to a cloud server.

[2221] 2. Image Analysis

[2222] The cloud server passes the received image data to a TensorFlow model, which analyzes the type of garbage.

[2223] 3. Matching and Feedback

[2224] The analysis results are compared with a PostgreSQL database to provide users with feedback on correct waste sorting. This feedback is displayed visually on a display monitor or smartphone.

[2225] 4. Alerts and interactive concierge

[2226] If incorrect sorting is detected, the system will issue an alert and activate an interactive concierge to provide detailed guidance.

[2227] 5. Emotion Recognition and Adaptive Feedback

[2228] IBM Watson's emotion recognition engine analyzes the user's facial expressions and voice, providing appropriate feedback based on their emotions.

[2229] 6. Prediction of accumulated amount and next collection date

[2230] A pressure sensor periodically measures the amount of trash in the trash can and sends the data to a cloud server. The cloud server predicts the next collection date based on the amount of trash and the collection schedule, and notifies the user.

[2231] 7. Remote Monitoring

[2232] Through a dedicated application, users can remotely monitor the status of their trash disposal in real time.

[2233] Specific example

[2234] 1. In the case of correct sorting

[2235] The user throws the plastic bottle into the recycling bin.

[2236] A sensor detects the mail being dropped in, and a camera takes a picture.

[2237] A cloud server analyzes the image and identifies it as a plastic bottle.

[2238] The display shows "This is recyclable waste, thank you!"

[2239] 2. In the case of incorrect sorting

[2240] The user throws the aluminum can into the recycling bin.

[2241] Analysis of the cloud server identified this as metal waste.

[2242] The display shows, "This is metal waste. Please dispose of it in the correct trash can."

[2243] If the user makes a displeased expression, the emotion recognition engine will detect it and gently say, "Recycling is important, let's try again!"

[2244] Example of a prompt

[2245] The following is an example of prompt statements used when analyzing waste using a generative AI model.

[2246] prompt:

[2247] New image data has been received. Please identify the type of trash shown in this image.

[2248] Image data:

[2249] [Base64 encoded image data]

[2250] Database reference:

[2251] Please use the latest database for classifying waste types.

[2252] This system enables users to sort waste easily and accurately, improving the precision of sorting procedures and reducing environmental impact in physical stores and other locations. Furthermore, it allows for management of waste bin volume and prediction of collection timing, resolving hygiene issues.

[2253] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2254] Step 1:

[2255] The device uses a pressure sensor to detect the dropping of trash. When trash is detected, the camera automatically captures an image and sends the image data to a cloud server. The input is the dropping of trash, and the output is the image data sent to the cloud server.

[2256] Step 2:

[2257] The server passes the received image data to a generating AI model for image analysis. Specifically, it uses a TensorFlow model to identify the type of waste. In this process, the input is image data, and the output is the analysis result indicating the type of waste.

[2258] Step 3:

[2259] The server compares the analysis results with the PostgreSQL database to verify the correct classification. The input is the analysis results and database information, and the output is the comparison result. If the comparison result is correct, feedback is provided to the user based on that information.

[2260] Step 4:

[2261] The feedback mechanism provides users with visual feedback via a display monitor or smartphone. The input is the matching result from a cloud server, and the output is a feedback message to the user. For example, it might display, "This is recyclable waste, thank you!"

[2262] Step 5:

[2263] If incorrect sorting is detected, the server issues an alert, and an interactive concierge begins providing detailed instructions. The input is the error detection result, and the output is the alert notification and the activation of the interactive concierge. The concierge displays on the screen, "This is metal waste. Please dispose of it in the correct trash can."

[2264] Step 6:

[2265] The server uses IBM Watson's emotion recognition engine to analyze the user's facial expressions and voice to determine their emotional state. Input is the user's facial expression data and voice, and output is the emotion analysis result. Appropriate feedback is provided based on the user's emotions.

[2266] Step 7:

[2267] The server periodically collects data from pressure sensors to measure the amount of waste in the trash cans. The input is pressure sensor data, and the output is the amount of waste. Based on this data, the server predicts the next collection date and notifies users and administrators.

[2268] Step 8:

[2269] Users can remotely monitor the status of their trash collection in real time through a dedicated application. Input is data from a cloud server, and output is the collection status displayed on the application. This makes it possible to manage trash even from a distance.

[2270] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2271] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2272] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2273] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2274] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2275] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2276] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2277] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2278] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[2279] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2280] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2281] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2282] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2283] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2284] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2285] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2286] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2287] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2288] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2289] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2290] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[2291] The following is further disclosed regarding the embodiments described above.

[2292] (Claim 1)

[2293] An image capture means for detecting discarded waste,

[2294] An image transmission means for sending image data of the above-mentioned discarded waste to a cloud server,

[2295] A generating artificial intelligence means for classifying waste by analyzing the above image data,

[2296] A matching means for comparing the above analysis results with a waste identification database of a domestic municipality,

[2297] A feedback mechanism that provides feedback to the user based on the above matching results,

[2298] An alert mechanism that issues an alert when it detects an error in the feedback provided by the above-mentioned feedback mechanism,

[2299] A system equipped with an interactive concierge service that provides real-time guidance to users.

[2300] (Claim 2)

[2301] A means for measuring th...

Claims

1. An image capture means for detecting discarded waste, An image transmission means for sending image data of the above-mentioned discarded waste to a cloud server, A generating artificial intelligence means for classifying waste by analyzing the above image data, A matching means for comparing the above analysis results with a waste identification database of a domestic municipality, A feedback mechanism that provides feedback to the user based on the above matching results, An alert mechanism that issues an alert when it detects an error in the feedback provided by the above-mentioned feedback mechanism, A system equipped with an interactive concierge service that provides real-time guidance to users.

2. A means for measuring the amount of the above-mentioned waste to measure the amount of accumulated waste, A prediction means for predicting the next collection date based on the above accumulated amount, The system according to claim 1, including a notification means for notifying the user of the above prediction results.

3. The system according to claim 1, comprising remote monitoring means for a user to remotely monitor the status of waste disposal.

Images