System

Abstract

A system is provided.SOLUTION: A system comprising: means for uploading an image captured or selected by a user; means for receiving and temporarily storing the uploaded image; means for analyzing the temporarily stored image to identify a type of waste; means for retrieving a sorting method from a database based on the identified type of waste; and means for providing appropriate instructions to the user based on the retrieved sorting method.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When sorting waste, it is difficult for many users to accurately identify which category a particular waste falls into. As a result, improper sorting occurs, leading to reduced recycling efficiency and adverse effects on the environment. The present invention aims to assist users in recycling appropriately by providing a simple and accurate method for sorting waste. [Means for solving the problem]

[0005] The present invention is a waste sorting support system that includes a means for uploading images taken or selected by a user, a means for receiving and temporarily storing the uploaded images, a means for analyzing the temporarily stored images to identify the type of waste, a means for retrieving a sorting method from a database based on the identified type of waste, and a means for providing appropriate instructions to the user based on the retrieved sorting method. In particular, the system is characterized by the use of a machine learning model to identify the type of waste and natural language generation technology to generate appropriate instructions to provide to the user.

[0006] "Uploading" is the act of sending an image file taken or selected by a user to a server via the Internet.

[0007] "Temporary storage" refers to the act of holding an image file received by a server for a short period of time or until the next processing.

[0008] "Analyzing an image" refers to the act of using a computer program to extract information from uploaded image data and identify specific objects or features.

[0009] "Identifying the type of waste" refers to the act of recognizing the type and classification of waste from the analyzed image data.

[0010] A "database" is a collection of information that systematically stores methods for separating waste according to type and other related information.

[0011] A "machine learning model" is an algorithm or mathematical model that is trained using past data to make predictions or classifications for new data.

[0012] "Natural language generation technology" is a technology that uses a computer program to automatically generate text in a natural language that is easy for humans to understand.

[0013] "Appropriate instructions" are messages that provide specific steps or methods that users need to follow to properly separate their waste. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate method for sorting waste.

[0036] System configuration

[0037] The system of the present invention mainly includes the following components:

[0038] 1. User device: This is the device that users use to take photos of trash and upload them to the server via the application. This can be a smartphone or tablet.

[0039] 2. Server: This is the central processing unit that receives and temporarily stores image data, and also performs the necessary processing for the image recognition module and database.

[0040] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0041] 4. Database: An information repository that stores the methods for separating waste according to type, providing information on how to separate identified waste.

[0042] 5. AI inference engine: Generates specific instructions for the user based on the results obtained from the image recognition module and information retrieved from the database.

[0043] 6. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0044] Program processing

[0045] 1. User uploads a photo

[0046] Users use a smartphone app to take or select an image of the trash.

[0047] The terminal transmits this image file to the server.

[0048] 2. The server receives the photo

[0049] The server receives the image file sent from the terminal and temporarily stores it.

[0050] 3. The image recognition module analyzes the photo

[0051] The server passes the temporarily stored image to the image recognition module.

[0052] An image recognition module analyzes the image and identifies the type of waste.

[0053] 4. Check against the database

[0054] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[0055] 5. AI generates instructions

[0056] Based on the classification method obtained, the server uses natural language generation technology to generate appropriate instructions for the user.

[0057] 6. Notify users

[0058] The server transmits the generated instructions to the user terminal.

[0059] The device will then display the received instructions to the user within the app.

[0060] Specific examples

[0061] Scenario: A user wants to dispose of a plastic bottle.

[0062] 1. User uploads a photo

[0063] Users take a photo of the plastic bottle using their smartphone camera and send it to the server via the app.

[0064] 2. The server receives the photo

[0065] The server receives this photo and temporarily stores it.

[0066] 3. The image recognition module analyzes the photo

[0067] The server sends the photo to an image recognition module for analysis.

[0068] The module analyzes the photo and identifies it as a plastic bottle.

[0069] 4. Check against the database

[0070] The server searches the database based on the results identified as a plastic bottle.

[0071] The database sends the result to the server: "Remove the cap and label and separate the bottle part as plastic."

[0072] 5. AI generates instructions

[0073] The AI ​​inference engine uses the results to generate instructions for the user.

[0074] The generated instructions are: "Remove the cap and label and separate the bottle as plastic."

[0075] 6. Notify users

[0076] The server sends this instruction to the user terminal.

[0077] The device will display these instructions to the user within the app, so they know how to properly separate their waste.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] A user launches the smartphone app and taps the "Upload Photo" button. The user then takes a photo of the trash or selects one from an existing photo gallery.

[0081] Step 2:

[0082] The device acquires the selected or taken photo data and generates an API request to send it to the server. The device then sends this request to the server via the Internet.

[0083] Step 3:

[0084] The server receives the API request, analyzes the uploaded photo data, and stores the photo data in temporary storage.

[0085] Step 4:

[0086] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[0087] Step 5:

[0088] The server receives the analysis results from the image recognition module, which identifies the trash in the photo as a plastic bottle.

[0089] Step 6:

[0090] Based on the recognition results, the server generates an SQL query to retrieve the appropriate classification method from the database, and sends this query to the database.

[0091] Step 7:

[0092] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[0093] Step 8:

[0094] The server generates instructions for the user based on the sorting method data returned from the database, using natural language generation technology.

[0095] Step 9:

[0096] The server creates an API response containing the generated instruction text and sends it to the device.

[0097] Step 10:

[0098] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app displays the message "Please remove the cap and label and separate the bottle part as plastic." on the app interface.

[0099] Step 11:

[0100] The user follows the instructions provided to separate the garbage, making it easy for the user to separate the garbage accurately.

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] Current waste sorting systems have problems, such as users finding it difficult to understand how to sort their waste, making it difficult to dispose of waste efficiently. Furthermore, manually identifying the type of waste and identifying the appropriate sorting method based on that is time-consuming and inconvenient for users.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for generating appropriate instructions for the user based on the retrieved sorting method, and means for notifying the generated instructions to a user terminal. This allows the user to easily learn the appropriate method for sorting waste, enabling efficient and accurate waste disposal.

[0106] "User" means an individual or organization that uses the system to separate waste.

[0107] "Image" refers to photographic data of waste taken or selected by the user.

[0108] "Uploading means" refers to a function or device for transmitting image data from a user terminal to a server.

[0109] "Means for receiving and temporarily storing" refers to a function or device that allows the server to receive image data sent from a user terminal and temporarily store it.

[0110] "Means for analyzing" refers to a system or function that allows the server to analyze temporarily stored images and automatically identify the type of waste.

[0111] A "machine learning model" is an algorithm or statistical model that the server uses to identify types of waste, and has knowledge learned from image data.

[0112] "Database" refers to an information repository that stores sorting methods for different types of waste.

[0113] "Means for obtaining a sorting method" refers to the function of the server to search and obtain an appropriate sorting method from the database based on the type of waste identified.

[0114] "Natural language generation technology" refers to technology that generates easy-to-understand instructions for users based on the classification method acquired by the server.

[0115] "Means for generating" refers to the function of generating specific instructions for the user based on the sorting method obtained by the server.

[0116] "Means for notifying" refers to the function of sending instructions generated by the server to the user terminal and notifying the user.

[0117] "User terminal" refers to the device that a user uses to take or select images of waste and upload them to the server. This could be a smartphone or tablet.

[0118] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate waste sorting method. Specific embodiments for implementing this system will be described below.

[0119] The system mainly includes the following components:

[0120] 1. User Device:

[0121] This is a device that allows users to take photos of waste and upload them to a server via an application. Specifically, it is a smartphone or tablet. The user device has a camera function and an internet connection, so the captured images can be quickly sent to the server.

[0122] 2. Server:

[0123] The server receives and temporarily stores image data. It is also a central processing unit that performs the necessary processing for the image recognition module and database. The server runs on a web server and efficiently manages the reception and processing of data.

[0124] 3. Image Recognition Module:

[0125] This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models (e.g., TensorFlow or PyTorch) to perform image analysis. The image recognition module uses a trained model to detect the characteristics of the waste and classify it into a specific category.

[0126] 4. Database:

[0127] The database is an information repository that stores waste separation methods according to the type of waste. It provides information on the separation method for the identified waste. The database contains the appropriate separation method for each type of waste and has an index for searching them.

[0128] 5. AI inference engine:

[0129] The AI ​​inference engine generates specific instructions for the user based on the results from the image recognition module and information retrieved from the database, using natural language generation techniques (e.g., generative AI models such as GPT-3) to generate instructions in a form that is easy for humans to understand.

[0130] 6. Notification System:

[0131] The notification system sends instructions generated by the server to the user's device and provides instructions on how to sort the waste. The notification system uses in-app notifications and push notifications to quickly and reliably inform users.

[0132] Specific examples

[0133] Scenario: A user wants to dispose of a plastic bottle.

[0134] 1. User uploads a photo:

[0135] Users take a photo of the plastic bottle with their smartphone camera and send it to the server via the app.

[0136] 2. The server receives and temporarily stores the photos:

[0137] The server receives this photo and temporarily stores it in the "uploads" folder.

[0138] 3. The image recognition module analyzes the photo:

[0139] The server sends the photo to an image recognition module, which uses TensorFlow to identify it as a plastic bottle.

[0140] 4. The server checks the database and obtains the classification method:

[0141] The server queries the database and obtains the sorting method: "Remove the cap and label, and separate the bottle part as plastic."

[0142] 5. The AI ​​inference engine generates instructions:

[0143] Based on the results obtained, the AI ​​inference engine generates instructions such as, "Remove the cap and label and separate the bottle part as plastic."

[0144] 6. The server notifies the user:

[0145] The server sends the generated instructions to the user's device, and the user receives the instructions within the app.

[0146] Example prompt statement

[0147] Input: A user-taken image of a plastic bottle

[0148] Output: Instructions based on image analysis results: "Remove the cap and label and separate the bottle as plastic."

[0149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0150] Step 1:

[0151] User uploads a photo

[0152] Input: Images of waste taken by the user with a smartphone camera

[0153] Specific operation: The user launches the app on their smartphone and takes a photo of the waste or selects an existing image. The taken or selected image is displayed in the app. The user presses the "Upload" button, and the image file is sent to the server.

[0154] Output: Image data sent from the device to the server

[0155] Step 2:

[0156] The server receives the photo and temporarily stores it.

[0157] Input: Image data sent from the device

[0158] Specific operation: The server receives the image data sent from the device and temporarily stores it in memory or storage. Once the server has received the image data, it registers the image metadata in a database. This metadata includes the image storage location, the date and time of transmission, etc.

[0159] Output: Temporarily saved image data and its metadata

[0160] Step 3:

[0161] Image recognition module analyzes the photo

[0162] Input: Temporarily saved image data

[0163] Specific operation: The server sends the temporarily stored image data to the image recognition module. The image recognition module analyzes the image using a machine learning model (e.g., TensorFlow or PyTorch). The model extracts waste characteristics from the image and identifies the type of waste. An analysis result (e.g., "plastic bottle") is generated.

[0164] Output: Analysis results identifying the type of waste

[0165] Step 4:

[0166] The server checks the database and obtains the sorting method.

[0167] Input: Analysis results from the image recognition module (type of waste)

[0168] Specific operation: The server searches a database based on the analysis results obtained from the image recognition module. The database stores sorting methods for each type of waste. The server retrieves the sorting method corresponding to the type of waste from the database.

[0169] Output: The obtained sorting method (e.g., "Remove the cap and label, and sort the bottle part as plastic")

[0170] Step 5:

[0171] AI inference engine generates instructions

[0172] Input: Obtained sorting method

[0173] Specific operation: The AI ​​inference engine uses natural language generation technology (for example, a generative AI model such as GPT-3) to generate instructions for the user based on the acquired sorting method. The generated instructions are expressed in a form that is easy for humans to understand. For example, the generated instruction might be, "Remove the cap and label, and separate the bottle part as plastic."

[0174] Output: Generated directives

[0175] Step 6:

[0176] The server notifies the user

[0177] Input: Generated instructions

[0178] Specific operation: The server sends the generated instruction text to the user's device. Using the notification system, the in-app notification function or push notification function is used to quickly and reliably convey information to the user.

[0179] Output: Instructions displayed on the user's terminal

[0180] This allows users to easily understand and practice proper waste separation methods.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] Existing systems that provide users with proper waste sorting methods often use fixed devices and lack real-time operability. This creates a problem when users want to know how to sort their waste immediately, especially in physical stores and public places. Furthermore, there are only a limited number of systems that support waste sorting rules that vary from region to region, making it difficult for users to accurately sort their waste in accordance with their local rules.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for providing appropriate instructions to the user based on the retrieved sorting method, means for displaying the sorting method on a device held by the user in real time, and means for customizing the sorting method based on the sorting rules of the area to which the user belongs. This allows the user to obtain an appropriate sorting method in real time, enabling accurate waste sorting suited to their area.

[0186] "Means for uploading images taken or selected by the user" refers to a function that allows a user to capture images of waste using a device such as a smartphone or smart glasses and send the images to the server.

[0187] "Means for receiving and temporarily storing uploaded images" refers to a function by which the server receives image data sent by a user and temporarily stores it for analysis.

[0188] "Means for analyzing temporarily stored images and identifying the type of waste" refers to a function that analyzes temporarily stored image data using machine learning models and image recognition algorithms to determine the type of waste indicated by the image.

[0189] "Means for retrieving sorting methods from a database based on the type of waste identified" is a function that searches for and retrieves appropriate sorting methods from a database based on the type of waste identified through analysis.

[0190] "Means for providing appropriate instructions to the user based on the obtained sorting method" is a function for generating and providing specific instructions to the user based on the sorting method obtained from the database.

[0191] "Means for displaying the sorting method in real time on a device held by the user" refers to a function for instantly displaying the acquired sorting method on a user device such as a smartphone or smart glasses.

[0192] "Means for customizing sorting methods based on the sorting rules of the area to which the user belongs" is a function for customizing appropriate sorting methods in accordance with the specific sorting rules of the area in which the user resides and providing that information.

[0193] This invention is a system that analyzes images of waste taken or selected by a user and provides sorting methods in real time. The system includes a user terminal, a server, an image recognition module, a database, an AI inference engine, and a notification system.

[0194] The system functions as follows:

[0195] User Device

[0196] The user device refers to a smartphone or smart glasses, and is responsible for allowing users to take images of waste and upload them to the server. When users take images of waste using the device's camera, the image data is saved on the device.

[0197] server

[0198] The server receives image data sent from the user terminal and temporarily stores it. It functions as a central processing unit for the image recognition module, database, AI inference engine, etc. After receiving the image data, the server sends the image data to the image recognition module.

[0199] Image Recognition Module

[0200] The image recognition module uses machine learning models to analyze image data submitted by users and identify the type of waste. This module uses generative AI models that have been trained on a wide range of waste data to perform highly accurate image analysis.

[0201] Database

[0202] The database stores the correct method for separating waste based on the analysis results. The server searches the database based on the results obtained from the image recognition module and obtains information on the corresponding separation method.

[0203] AI inference engine

[0204] Based on the classification methods retrieved from the database, the AI ​​inference engine uses natural language generation technology to generate specific instructions for the user, which are presented in a format that is easy for the user to understand.

[0205] Notification System

[0206] The notification system sends instructions generated by the server to the user's device, where the instructions are displayed in real time and users can receive guidance on how to properly separate their waste.

[0207] Specific examples

[0208] Here is a specific example using a smartphone.

[0209] When a user disposes of a plastic bottle purchased at a supermarket, they take a photo of the bottle with their smartphone camera. The photo is then sent to a server via the app. The server receives the photo and temporarily stores it. Next, an image recognition module analyzes the photo and identifies it as a plastic bottle. Based on this identification, the server searches a database for the corresponding sorting method, and uses an AI inference engine to generate instructions such as "Remove the cap and label, and separate the bottle portion as plastic." These instructions are sent to the smartphone, allowing the user to learn the correct sorting method in real time.

[0210] Prompt Sentence Examples

[0211] An example prompt for uploading an image of a plastic bottle is as follows:

[0212] "I'll upload a picture of a plastic bottle. How do I dispose of this plastic bottle?"

[0213] This allows users to easily obtain accurate sorting instructions in accordance with local sorting rules.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Step 1:

[0216] Users take pictures of the waste using their smartphone camera and upload them to a server via the app.

[0217] Input: Image of waste taken with a smartphone

[0218] Output: Image data sent to the server

[0219] Specific operation: The user launches the camera app on their smartphone, takes a photo of the waste, and then presses the upload button to send the image to the server.

[0220] Step 2:

[0221] The server receives the image data sent by the user and temporarily stores it.

[0222] Input: Image data sent by the user

[0223] Output: Image data temporarily stored on the server

[0224] Specific operation: The server receives the image data, temporarily stores it in a database or file system, and records a receipt confirmation in a log.

[0225] Step 3:

[0226] The server passes the temporarily stored image data to an image recognition module to identify the type of waste.

[0227] Input: Temporarily saved image data

[0228] Output: Analysis results (type of waste)

[0229] How it works: The server sends image data to the image recognition module, which analyzes the image using a generative AI model. The image recognition module identifies the type of waste and returns that information to the server.

[0230] Step 4:

[0231] The server searches the database based on the type of waste obtained from the image recognition module to obtain the corresponding sorting method.

[0232] Input: Analysis results (type of waste)

[0233] Output: Sorting method

[0234] Specific operation: The server searches the database to obtain the classification method corresponding to the analysis result, and then extracts the appropriate classification method by referring to the rules and instructions in the database.

[0235] Step 5:

[0236] Based on the sorting method obtained by the server, the AI ​​inference engine generates specific instructions for the user in natural language.

[0237] Input: Sorting method

[0238] Output: Separation instructions for the user

[0239] How it works: The AI ​​inference engine analyzes the sorting method and uses a generative AI model to generate instructions in natural language that are easy for the user to understand, such as "Remove the cap and label, and separate the bottle part as plastic."

[0240] Step 6:

[0241] The server sends the generated instructions to the user's smartphone and displays them in real time.

[0242] Input: Separation instructions for the user

[0243] Output: Instructions displayed on the smartphone

[0244] Specific operation: The server sends the generated instruction text to the user's smartphone via the notification system. The smartphone app receives the instruction text and displays it on the screen in real time.

[0245] This processing flow allows users to learn in real time how to properly separate the waste they have photographed, enabling them to dispose of it correctly in accordance with their local rules.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. By using the emotion engine, it becomes possible to provide more flexible and appropriate instructions according to the user's emotional state.

[0248] System configuration

[0249] The system of the present invention mainly includes the following components:

[0250] 1. User device: A device where a user takes a photo of the garbage and uploads it to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[0251] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[0252] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0253] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[0254] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[0255] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[0256] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0257] Program processing

[0258] 1. User uploads a photo

[0259] Using a smartphone app, users take or select an image of the trash, which also captures facial expressions and audio data.

[0260] The terminal transmits the image file and emotion data to the server.

[0261] 2. The server receives the photo

[0262] The server receives the image file and emotion data sent from the terminal and temporarily stores them.

[0263] 3. The image recognition module analyzes the photo

[0264] The server passes the temporarily stored image data to the image recognition module.

[0265] An image recognition module analyzes the image and identifies the type of waste.

[0266] 4. Emotion engine analyzes emotions

[0267] The server passes the temporarily stored emotional data (facial expressions and voice) to the emotion engine.

[0268] The emotion engine analyzes this data to determine the user's emotional state.

[0269] 5. Check against the database

[0270] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[0271] 6. AI generates instructions

[0272] The server generates instructions for the user based on the image recognition results, the emotion engine results, and classification methods from the database. These instructions take into account the user's emotional state.

[0273] 7. Notify users

[0274] The server transmits the generated instruction text to the user terminal.

[0275] The device will then display the received instructions to the user within the app.

[0276] Specific examples

[0277] Scenario: A user wants to dispose of a plastic bottle.

[0278] 1. User uploads a photo

[0279] Users take a photo of the bottle using their smartphone camera and send it to the server via the app, which also captures their facial expressions and voice.

[0280] 2. The server receives the photo

[0281] The server receives the photo and emotion data and temporarily stores them.

[0282] 3. The image recognition module analyzes the photo

[0283] The server sends the photo to an image recognition module for analysis.

[0284] The module analyzes the photo and identifies it as a plastic bottle.

[0285] 4. Emotion engine analyzes emotions

[0286] The server sends the emotion data to the emotion engine for analysis.

[0287] The emotion engine identifies that the user is stressed.

[0288] 5. Check against the database

[0289] The server searches the database based on the results identified as a plastic bottle.

[0290] The database returns the data: "Remove the cap and label, and separate the bottle part as plastic."

[0291] 6. AI generates instructions

[0292] The AI ​​inference engine uses these results to generate instructions for the user. Based on the emotion engine's results, the instructions are adjusted to "Recycle in a few easy steps. Remove the cap and label and separate the bottle part as plastic."

[0293] 7. Notify users

[0294] The server sends this instruction to the user terminal.

[0295] The device will display these instructions within the app, making it easy for users to follow the proper sorting method.

[0296] The processing flow will be explained below.

[0297] Step 1:

[0298] A user launches the smartphone app and taps the "Upload Photo" button. They can then take a photo of the litter or select one from their existing photo gallery. The app also simultaneously captures the user's facial expressions and voice.

[0299] Step 2:

[0300] The device acquires the selected or taken photo data and the captured facial expression and voice data, generates an API request to send this data to the server, and sends it to the server via the Internet.

[0301] Step 3:

[0302] The server receives the API request, analyzes the uploaded photo data and emotion data, and temporarily stores this data.

[0303] Step 4:

[0304] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[0305] Step 5:

[0306] The server receives the analysis results from the image recognition module and identifies the garbage in the photo as a plastic bottle.

[0307] Step 6:

[0308] The server sends the temporarily stored facial expression and voice data to the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., stress, comfort).

[0309] Step 7:

[0310] The server generates an SQL query to retrieve an appropriate sorting method from the database based on the recognition result (plastic bottle) and emotion data (e.g., stress state). The server then sends this query to the database.

[0311] Step 8:

[0312] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[0313] Step 9:

[0314] The server generates instructions for the user based on the sorting method data returned from the database. At this time, the server uses natural language generation technology to adjust the instructions to take into account the user's emotional state. For example, "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic."

[0315] Step 10:

[0316] The server creates an API response containing the generated instruction and sends it to the user's device.

[0317] Step 11:

[0318] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app interface displays the message, "Recycling is easy. Remove the cap and label, and separate the bottle part as plastic."

[0319] Step 12:

[0320] Users follow the instructions provided to them to separate their garbage, making it easy for them to separate their garbage accurately.

[0321] Example 2

[0322] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] Conventional garbage sorting systems not only do not provide sufficient instructions for users to accurately sort waste types, but also lack the flexibility to take into account the user's emotional state, making it difficult for users to sort their waste efficiently and without stress.

[0324] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and temporarily storing images and emotional data taken or selected by the user, means for analyzing the temporarily stored images to identify the type of waste, means for analyzing the temporarily stored emotional data to identify the user's emotional state, means for acquiring a sorting method from a database based on the identified type of waste and the user's emotional state, means for using an AI inference engine to generate appropriate instructions that take into account the user's emotional state based on the acquired sorting method, and means for notifying the user terminal of the generated instructions. This allows the user to receive flexible and appropriate sorting instructions that take into account their own emotional state, allowing for stress-free and efficient sorting work.

[0325] "User" refers to a person who uses the system to take or select images of waste and receive instructions on how to sort it.

[0326] "Device" refers to a device used by a User, such as a smartphone or tablet, that is capable of taking and sending images, capturing emotional data, and receiving notifications.

[0327] "Server" refers to a computer system that receives image data and emotion data, temporarily stores them, and analyzes them.

[0328] "Image recognition module" refers to a software module that runs on a server and has the function of analyzing received image data using a machine learning model and identifying the type of waste.

[0329] An "emotion engine" refers to a deep learning model that runs on a server and has the ability to analyze received facial expressions and voice data to identify the user's emotional state.

[0330] "Database" refers to an information repository that systematically stores waste sorting methods according to type and other related information.

[0331] An "AI inference engine" refers to an algorithm or program that generates instructions for users based on the analysis results of the image recognition module and emotion engine and information from a database.

[0332] "Notification system" refers to a module that has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0333] "Emotional Data" refers to a user's facial expressions and voice data, and means data used to analyze a user's emotional state.

[0334] A "machine learning model" refers to statistical methods and algorithms that learn patterns from data and perform image recognition and emotion analysis.

[0335] A "deep learning model" refers to an algorithm that uses artificial neuron networks to perform more advanced data analysis.

[0336] "Temporary storage" means that the image data and emotion data received by the server are temporarily stored until analysis is complete.

[0337] "Sorting method" refers to the appropriate disposal method for each type of waste, and is information obtained from the database.

[0338] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. This system makes it possible to provide flexible and appropriate instructions according to the user's emotional state.

[0339] System configuration

[0340] The system of the present invention mainly includes the following components:

[0341] 1. User device: This is a device that users use to take photos of garbage and upload them to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[0342] 2. Server: Receives and temporarily stores image data and emotion data, and performs the necessary processing for the image recognition module, emotion engine, database, and AI inference engine.

[0343] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0344] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[0345] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[0346] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[0347] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0348] System operation explanation

[0349] Users take photos of waste using their smartphone camera and send them to the server via the application. During the transmission, facial expressions and voice data are also captured. The server receives and temporarily stores this data.

[0350] The server then passes the temporarily stored image data to an image recognition module, which uses a machine learning model to identify the type of waste. At the same time, it also passes the emotion data to an emotion engine, which uses a deep learning model to analyze the user's emotional state.

[0351] The analysis results are compared with the database, and the server uses an AI inference engine to generate instructions for the user. These instructions are based on the analyzed emotional data and take into account the user's emotional state, such as "Recycle in a few easy steps." Finally, the generated instructions are sent to the user's device and displayed within the app.

[0352] Specific examples

[0353] For example, if a user wants to throw away a plastic bottle, the following actions occur:

[0354] 1. The user takes a photo of a plastic bottle using their smartphone camera and sends the photo and emotional data to the server via the app.

[0355] 2. The server receives the photo and emotion data and temporarily stores them.

[0356] 3. The server sends the image data to the image recognition module, which identifies it as a plastic bottle.

[0357] 4. The server sends the emotion data to the emotion engine, which determines that the user is stressed.

[0358] 5. The server searches the database to find out how to separate plastic bottles. The database returns the information that says, "Remove the cap and label, and separate the bottle as plastic."

[0359] 6. The AI ​​inference engine generates the following instruction: "Recycle in a few simple steps. Remove the cap and label, and separate the bottle part as plastic."

[0360] 7. The server sends this instruction to the user's device, which displays the instruction within the app, allowing the user to view the notification and take appropriate action.

[0361] Prompt Sentence Examples

[0362] Please explain the process by which a user takes a photo of the garbage they want to dispose of, sends emotional data to the server, and receives sorting instructions based on their emotional state.

[0363] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0364] Step 1:

[0365] The user uploads a photo.

[0366] How it works: The user takes a photo of the waste using their smartphone camera, which also captures facial expressions and voice data. The user then presses the upload button within the app, which sends this data to the server.

[0367] Input: Garbage images, facial expression data, and audio data.

[0368] Output: Image files and emotion data sent to the server.

[0369] Step 2:

[0370] The server receives the photos and temporarily stores them.

[0371] Specific operation: The server receives the image file and emotion data sent from the device and temporarily stores them. Upon receiving the data, it checks for consistency and whether there are any missing data.

[0372] Input: Image file and emotion data sent from the user device.

[0373] Output: Image files and emotion data saved in temporary storage.

[0374] Step 3:

[0375] The image recognition module analyzes the photo.

[0376] How it works: The server passes the image data stored in the temporary storage area to the image recognition module, which uses a pre-trained machine learning model to analyze the image and identify the type of waste.

[0377] Input: Image data retrieved from temporary storage.

[0378] Output: Identified waste type (e.g., PET bottles, paper, plastic).

[0379] Step 4:

[0380] The emotion engine analyzes emotions.

[0381] How it works: The server passes the emotional data (facial expressions and voice data) stored in the temporary storage area to the emotion engine, which then uses a deep learning model to analyze the data and identify the user's emotional state.

[0382] Input: Emotion data retrieved from temporary storage.

[0383] Output: Identified user emotional state (e.g., stressed, relaxed, angry).

[0384] Step 5:

[0385] The server checks the database.

[0386] Specific operation: The server searches the database based on the type of waste obtained from the image recognition module. The database stores appropriate sorting methods and related information for each type of waste. The server retrieves the corresponding sorting method.

[0387] Input: Identified waste type.

[0388] Output: The corresponding sorting method (e.g. remove cap and label, and sort the bottle as plastic).

[0389] Step 6:

[0390] An AI inference engine generates instructions.

[0391] Specific operation: The server uses an AI inference engine to generate instructions for the user based on the image recognition results, the emotion engine results, and the classification method retrieved from the database. These instructions take into account the user's emotional state.

[0392] Input: Image recognition results, emotion engine results, classification methods from the database.

[0393] Output: Instructions for the user (e.g., "Recycle in a few easy steps. Remove the cap and label and separate the bottle for plastic.").

[0394] Step 7:

[0395] Notify the user.

[0396] Specific operation: The server sends the generated instruction text to the user's device, which then displays the instruction within the app and suggests appropriate sorting methods to the user.

[0397] Input: Instructions sent by the server.

[0398] Output: Instructions displayed on the user's terminal.

[0399] (Application example 2)

[0400] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] In recent years, with growing interest in environmental issues, accurate waste separation is becoming increasingly important. However, many users do not properly understand the types of waste and how to separate them, leading to incorrect separation. Furthermore, users' emotional state can sometimes affect their waste separation behavior. This situation hinders effective waste management. The present invention aims to solve these problems and provide a system that supports users in easily separating waste appropriately.

[0402] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken or selected by the user, means for acquiring and analyzing the user's emotional data, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for acquiring a sorting method from a database based on the identified type of waste, and means for providing the user with appropriate instructions based on the acquired sorting method and the user's emotional state. This makes it possible to provide flexible and appropriate sorting instructions based on the type of waste and the user's emotional state.

[0403] A "user" is an individual who uses the system to take or select images of waste and learn how to separate it.

[0404] "Waste" means items that are no longer needed or useful and that need to be disposed of in an environmentally friendly manner.

[0405] "Means for uploading images" refers to a function or device for transmitting image data taken or selected by a user to a server.

[0406] "Emotion data" is information that indicates the emotional state of the user, and is acquired based on facial expressions and voice data.

[0407] The "means for receiving and temporarily storing images" refers to a function or device that temporarily stores uploaded image data.

[0408] The "means for analyzing images and identifying the type of waste" refers to a function or device for identifying the type of waste from temporarily stored image data using a machine learning model.

[0409] The "database" is an information repository that systematically stores waste sorting methods according to type and other related information.

[0410] The "means for obtaining a sorting method" is a function or device that searches for and obtains an appropriate sorting method from a database based on the type of waste identified.

[0411] The "means for providing appropriate instructions" is a function or device that generates and conveys easy-to-understand instructions to the user based on the acquired discernment method and the user's emotional state.

[0412] This invention relates to an application that helps users properly separate waste generated during food delivery. Specifically, it provides a system that analyzes images of waste and provides guidance on the appropriate separation method based on the user's emotional state.

[0413] System configuration

[0414] The system of the present invention includes the following components:

[0415] 1. User device: This is the device on which the user takes a picture of the trash and acquires emotion data. This device can be a smartphone or tablet.

[0416] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[0417] 3. Image Recognition Module: This module analyzes uploaded images and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0418] 4. Emotion Engine: This engine has the ability to analyze the user's emotional state. It uses a deep learning model to perform emotion analysis.

[0419] 5. Database: Contains information on how to separate waste by type and related information.

[0420] 6. AI Inference Engine: Based on the results of the image recognition module and emotion engine, it generates specific and emotion-sensitive instructions for the user.

[0421] 7. Notification system: The generated instructions are sent to the user's terminal, and the user is informed of how to sort the waste.

[0422] System Operation

[0423] 1. User device operation:

[0424] Users use their smartphone camera to take pictures of the waste and capture facial expressions and voice as emotional data.

[0425] The data is then uploaded to the server through the application.

[0426] 2. Receiving and temporary storage on the server:

[0427] The server temporarily stores the uploaded image data and emotion data.

[0428] 3. Image Recognition Module Processing:

[0429] The temporarily stored image data is analyzed by an image recognition module to identify the type of waste.

[0430] 4. Emotion Engine Processing:

[0431] The emotion data is analyzed by an emotion engine to determine the user's emotional state.

[0432] 5. Database verification:

[0433] The server searches a database based on the identified type of waste and obtains the appropriate sorting method.

[0434] 6. Generate instructions:

[0435] The AI ​​inference engine generates easy-to-understand instructions based on the type of waste and the user's emotional state.

[0436] 7. Notice to Users:

[0437] The server then sends the generated instructions to the user's device and displays them within the app, allowing the user to learn how to properly separate their trash.

[0438] Specific examples

[0439] For example, consider the case where a user is about to dispose of a plastic bottle. The user takes a picture of the bottle, captures facial expressions and voice as emotional data, and uploads it to the server. The server uses an image recognition module to recognize the bottle, and an emotion engine to identify that the user is feeling stressed. The database is then retrieved to provide instructions such as "Remove the cap and label, and separate the bottle part as plastic," and the AI ​​inference engine generates instructions such as "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic." These instructions are displayed on the user's device, allowing the user to properly separate their waste.

[0440] Prompt Sentence Examples

[0441] 1. "waste_image.jpg" is an image of a plastic bottle taken by a user.

[0442] 2. User's face photo "user_image.jpg".

[0443] 3. User's audio file "user_audio.wav".

[0444] Using this data, the system suggests appropriate ways to dispose of plastic bottles, taking into account the user's emotional state.

[0445] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0446] Step 1:

[0447] Users take images of waste using their smartphone cameras and capture facial expressions and voice as emotional data. Users then upload these data to a server via an application. The input is waste images, facial expressions, and voice data, and the output is the uploaded data.

[0448] Step 2:

[0449] The server receives and temporarily stores image data and emotion data uploaded by the user. The input of this step is the uploaded image data and emotion data, and the output is the temporarily stored data.

[0450] Step 3:

[0451] The server passes the temporarily stored image data to an image recognition module, which uses machine learning models to analyze these images and identify the waste type. The input is the temporarily stored image data, and the output is the identified waste type.

[0452] Step 4:

[0453] The server passes the stored emotional data to the emotion engine, which uses a deep learning model to analyze the data and identify the user's emotional state. The input is the stored emotional data, and the output is the user's emotional state.

[0454] Step 5:

[0455] The server searches the database based on the identified waste type to obtain the corresponding sorting method. The input is the identified waste type, and the output is the corresponding sorting method.

[0456] Step 6:

[0457] The server uses an AI inference engine to generate specific and emotion-sensitive instructions for users based on the image recognition results, the emotion engine results, and the sorting method from the database. The input is the type of waste, the user's emotional state, and the sorting method, and the output is the generated instructions.

[0458] Step 7:

[0459] The server sends the generated instructions to the user's device, which then displays the received instructions to the user within the app. The input is the generated instructions, and the output is the sorting instructions displayed to the user.

[0460] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0461] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0462] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0463] [Second embodiment]

[0464] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0465] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0466] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0467] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0468] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0469] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0470] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0471] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0472] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0473] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0474] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0475] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0476] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate method for sorting waste.

[0477] System configuration

[0478] The system of the present invention mainly includes the following components:

[0479] 1. User device: This is the device that users use to take photos of trash and upload them to the server via the application. This can be a smartphone or tablet.

[0480] 2. Server: This is the central processing unit that receives and temporarily stores image data, and also performs the necessary processing for the image recognition module and database.

[0481] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0482] 4. Database: An information repository that stores the methods for separating waste according to type, providing information on how to separate identified waste.

[0483] 5. AI inference engine: Generates specific instructions for the user based on the results obtained from the image recognition module and information retrieved from the database.

[0484] 6. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0485] Program processing

[0486] 1. User uploads a photo

[0487] Users use a smartphone app to take or select an image of the trash.

[0488] The terminal transmits this image file to the server.

[0489] 2. The server receives the photo

[0490] The server receives the image file sent from the terminal and temporarily stores it.

[0491] 3. The image recognition module analyzes the photo

[0492] The server passes the temporarily stored image to the image recognition module.

[0493] An image recognition module analyzes the image and identifies the type of waste.

[0494] 4. Check against the database

[0495] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[0496] 5. AI generates instructions

[0497] Based on the classification method obtained, the server uses natural language generation technology to generate appropriate instructions for the user.

[0498] 6. Notify users

[0499] The server transmits the generated instructions to the user terminal.

[0500] The device will then display the received instructions to the user within the app.

[0501] Specific examples

[0502] Scenario: A user wants to dispose of a plastic bottle.

[0503] 1. User uploads a photo

[0504] Users take a photo of the plastic bottle using their smartphone camera and send it to the server via the app.

[0505] 2. The server receives the photo

[0506] The server receives this photo and temporarily stores it.

[0507] 3. The image recognition module analyzes the photo

[0508] The server sends the photo to an image recognition module for analysis.

[0509] The module analyzes the photo and identifies it as a plastic bottle.

[0510] 4. Check against the database

[0511] The server searches the database based on the results identified as a plastic bottle.

[0512] The database sends the result to the server: "Remove the cap and label and separate the bottle part as plastic."

[0513] 5. AI generates instructions

[0514] The AI ​​inference engine uses the results to generate instructions for the user.

[0515] The generated instructions are: "Remove the cap and label and separate the bottle as plastic."

[0516] 6. Notify users

[0517] The server sends this instruction to the user terminal.

[0518] The device will display these instructions to the user within the app, so they know how to properly separate their waste.

[0519] The processing flow will be explained below.

[0520] Step 1:

[0521] A user launches the smartphone app and taps the "Upload Photo" button. The user then takes a photo of the trash or selects one from an existing photo gallery.

[0522] Step 2:

[0523] The device acquires the selected or taken photo data and generates an API request to send it to the server. The device then sends this request to the server via the Internet.

[0524] Step 3:

[0525] The server receives the API request, analyzes the uploaded photo data, and stores the photo data in temporary storage.

[0526] Step 4:

[0527] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[0528] Step 5:

[0529] The server receives the analysis results from the image recognition module, which identifies the trash in the photo as a plastic bottle.

[0530] Step 6:

[0531] Based on the recognition results, the server generates an SQL query to retrieve the appropriate classification method from the database, and sends this query to the database.

[0532] Step 7:

[0533] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[0534] Step 8:

[0535] The server generates instructions for the user based on the sorting method data returned from the database, using natural language generation technology.

[0536] Step 9:

[0537] The server creates an API response containing the generated instruction text and sends it to the device.

[0538] Step 10:

[0539] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app displays the message "Please remove the cap and label and separate the bottle part as plastic." on the app interface.

[0540] Step 11:

[0541] The user follows the instructions provided to separate the garbage, making it easy for the user to separate the garbage accurately.

[0542] Example 1

[0543] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0544] Current waste sorting systems have problems, such as users finding it difficult to understand how to sort their waste, making it difficult to dispose of waste efficiently. Furthermore, manually identifying the type of waste and identifying the appropriate sorting method based on that is time-consuming and inconvenient for users.

[0545] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0546] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for generating appropriate instructions for the user based on the retrieved sorting method, and means for notifying the generated instructions to a user terminal. This allows the user to easily learn the appropriate method for sorting waste, enabling efficient and accurate waste disposal.

[0547] "User" means an individual or organization that uses the system to separate waste.

[0548] "Image" refers to photographic data of waste taken or selected by the user.

[0549] "Uploading means" refers to a function or device for transmitting image data from a user terminal to a server.

[0550] "Means for receiving and temporarily storing" refers to a function or device that allows the server to receive image data sent from a user terminal and temporarily store it.

[0551] "Means for analyzing" refers to a system or function that allows the server to analyze temporarily stored images and automatically identify the type of waste.

[0552] A "machine learning model" is an algorithm or statistical model that the server uses to identify types of waste, and has knowledge learned from image data.

[0553] "Database" refers to an information repository that stores sorting methods for different types of waste.

[0554] "Means for obtaining a sorting method" refers to the function of the server to search and obtain an appropriate sorting method from the database based on the type of waste identified.

[0555] "Natural language generation technology" refers to technology that generates easy-to-understand instructions for users based on the classification method acquired by the server.

[0556] "Means for generating" refers to the function of generating specific instructions for the user based on the sorting method obtained by the server.

[0557] "Means for notifying" refers to the function of sending instructions generated by the server to the user terminal and notifying the user.

[0558] "User terminal" refers to the device that a user uses to take or select images of waste and upload them to the server. This could be a smartphone or tablet.

[0559] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate waste sorting method. Specific embodiments for implementing this system will be described below.

[0560] The system mainly includes the following components:

[0561] 1. User Device:

[0562] This is a device that allows users to take photos of waste and upload them to a server via an application. Specifically, it is a smartphone or tablet. The user device has a camera function and an internet connection, so the captured images can be quickly sent to the server.

[0563] 2. Server:

[0564] The server receives and temporarily stores image data. It is also a central processing unit that performs the necessary processing for the image recognition module and database. The server runs on a web server and efficiently manages the reception and processing of data.

[0565] 3. Image Recognition Module:

[0566] This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models (e.g., TensorFlow or PyTorch) to perform image analysis. The image recognition module uses a trained model to detect the characteristics of the waste and classify it into a specific category.

[0567] 4. Database:

[0568] The database is an information repository that stores waste separation methods according to the type of waste. It provides information on the separation method for the identified waste. The database contains the appropriate separation method for each type of waste and has an index for searching them.

[0569] 5. AI inference engine:

[0570] The AI ​​inference engine generates specific instructions for the user based on the results from the image recognition module and information retrieved from the database, using natural language generation techniques (e.g., generative AI models such as GPT-3) to generate instructions in a form that is easy for humans to understand.

[0571] 6. Notification System:

[0572] The notification system sends instructions generated by the server to the user's device and provides instructions on how to sort the waste. The notification system uses in-app notifications and push notifications to quickly and reliably inform users.

[0573] Specific examples

[0574] Scenario: A user wants to dispose of a plastic bottle.

[0575] 1. User uploads a photo:

[0576] Users take a photo of the plastic bottle with their smartphone camera and send it to the server via the app.

[0577] 2. The server receives and temporarily stores the photos:

[0578] The server receives this photo and temporarily stores it in the "uploads" folder.

[0579] 3. The image recognition module analyzes the photo:

[0580] The server sends the photo to an image recognition module, which uses TensorFlow to identify it as a plastic bottle.

[0581] 4. The server checks the database and obtains the classification method:

[0582] The server queries the database and obtains the sorting method: "Remove the cap and label, and separate the bottle part as plastic."

[0583] 5. The AI ​​inference engine generates instructions:

[0584] Based on the results obtained, the AI ​​inference engine generates instructions such as, "Remove the cap and label and separate the bottle part as plastic."

[0585] 6. The server notifies the user:

[0586] The server sends the generated instructions to the user's device, and the user receives the instructions within the app.

[0587] Example prompt statement

[0588] Input: A user-taken image of a plastic bottle

[0589] Output: Instructions based on image analysis results: "Remove the cap and label and separate the bottle as plastic."

[0590] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0591] Step 1:

[0592] User uploads a photo

[0593] Input: Images of waste taken by the user with a smartphone camera

[0594] Specific operation: The user launches the app on their smartphone and takes a photo of the waste or selects an existing image. The taken or selected image is displayed in the app. The user presses the "Upload" button, and the image file is sent to the server.

[0595] Output: Image data sent from the device to the server

[0596] Step 2:

[0597] The server receives the photo and temporarily stores it.

[0598] Input: Image data sent from the device

[0599] Specific operation: The server receives the image data sent from the device and temporarily stores it in memory or storage. Once the server has received the image data, it registers the image metadata in a database. This metadata includes the image storage location, the date and time of transmission, etc.

[0600] Output: Temporarily saved image data and its metadata

[0601] Step 3:

[0602] Image recognition module analyzes the photo

[0603] Input: Temporarily saved image data

[0604] Specific operation: The server sends the temporarily stored image data to the image recognition module. The image recognition module analyzes the image using a machine learning model (e.g., TensorFlow or PyTorch). The model extracts waste characteristics from the image and identifies the type of waste. An analysis result (e.g., "plastic bottle") is generated.

[0605] Output: Analysis results identifying the type of waste

[0606] Step 4:

[0607] The server checks the database and obtains the sorting method.

[0608] Input: Analysis results from the image recognition module (type of waste)

[0609] Specific operation: The server searches a database based on the analysis results obtained from the image recognition module. The database stores sorting methods for each type of waste. The server retrieves the sorting method corresponding to the type of waste from the database.

[0610] Output: The obtained sorting method (e.g., "Remove the cap and label, and sort the bottle part as plastic")

[0611] Step 5:

[0612] AI inference engine generates instructions

[0613] Input: Obtained sorting method

[0614] Specific operation: The AI ​​inference engine uses natural language generation technology (for example, a generative AI model such as GPT-3) to generate instructions for the user based on the acquired sorting method. The generated instructions are expressed in a form that is easy for humans to understand. For example, the generated instruction might be, "Remove the cap and label, and separate the bottle part as plastic."

[0615] Output: Generated directives

[0616] Step 6:

[0617] The server notifies the user

[0618] Input: Generated instructions

[0619] Specific operation: The server sends the generated instruction text to the user's device. Using the notification system, the in-app notification function or push notification function is used to quickly and reliably convey information to the user.

[0620] Output: Instructions displayed on the user's terminal

[0621] This allows users to easily understand and practice proper waste separation methods.

[0622] (Application example 1)

[0623] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0624] Existing systems that provide users with proper waste sorting methods often use fixed devices and lack real-time operability. This creates a problem when users want to know how to sort their waste immediately, especially in physical stores and public places. Furthermore, there are only a limited number of systems that support waste sorting rules that vary from region to region, making it difficult for users to accurately sort their waste in accordance with their local rules.

[0625] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0626] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for providing appropriate instructions to the user based on the retrieved sorting method, means for displaying the sorting method on a device held by the user in real time, and means for customizing the sorting method based on the sorting rules of the area to which the user belongs. This allows the user to obtain an appropriate sorting method in real time, enabling accurate waste sorting suited to their area.

[0627] "Means for uploading images taken or selected by the user" refers to a function that allows a user to capture images of waste using a device such as a smartphone or smart glasses and send the images to the server.

[0628] "Means for receiving and temporarily storing uploaded images" refers to a function by which the server receives image data sent by a user and temporarily stores it for analysis.

[0629] "Means for analyzing temporarily stored images and identifying the type of waste" refers to a function that analyzes temporarily stored image data using machine learning models and image recognition algorithms to determine the type of waste indicated by the image.

[0630] "Means for retrieving sorting methods from a database based on the type of waste identified" is a function that searches for and retrieves appropriate sorting methods from a database based on the type of waste identified through analysis.

[0631] "Means for providing appropriate instructions to the user based on the obtained sorting method" is a function for generating and providing specific instructions to the user based on the sorting method obtained from the database.

[0632] "Means for displaying the sorting method in real time on a device held by the user" refers to a function for instantly displaying the acquired sorting method on a user device such as a smartphone or smart glasses.

[0633] "Means for customizing sorting methods based on the sorting rules of the area to which the user belongs" is a function for customizing appropriate sorting methods in accordance with the specific sorting rules of the area in which the user resides and providing that information.

[0634] This invention is a system that analyzes images of waste taken or selected by a user and provides sorting methods in real time. The system includes a user terminal, a server, an image recognition module, a database, an AI inference engine, and a notification system.

[0635] The system functions as follows:

[0636] User Device

[0637] The user device refers to a smartphone or smart glasses, and is responsible for allowing users to take images of waste and upload them to the server. When users take images of waste using the device's camera, the image data is saved on the device.

[0638] server

[0639] The server receives image data sent from the user terminal and temporarily stores it. It functions as a central processing unit for the image recognition module, database, AI inference engine, etc. After receiving the image data, the server sends the image data to the image recognition module.

[0640] Image Recognition Module

[0641] The image recognition module uses machine learning models to analyze image data submitted by users and identify the type of waste. This module uses generative AI models that have been trained on a wide range of waste data to perform highly accurate image analysis.

[0642] Database

[0643] The database stores the correct method for separating waste based on the analysis results. The server searches the database based on the results obtained from the image recognition module and obtains information on the corresponding separation method.

[0644] AI inference engine

[0645] Based on the classification methods retrieved from the database, the AI ​​inference engine uses natural language generation technology to generate specific instructions for the user, which are presented in a format that is easy for the user to understand.

[0646] Notification System

[0647] The notification system sends instructions generated by the server to the user's device, where the instructions are displayed in real time and users can receive guidance on how to properly separate their waste.

[0648] Specific examples

[0649] Here is a specific example using a smartphone.

[0650] When a user disposes of a plastic bottle purchased at a supermarket, they take a photo of the bottle with their smartphone camera. The photo is then sent to a server via the app. The server receives the photo and temporarily stores it. Next, an image recognition module analyzes the photo and identifies it as a plastic bottle. Based on this identification, the server searches a database for the corresponding sorting method, and uses an AI inference engine to generate instructions such as "Remove the cap and label, and separate the bottle portion as plastic." These instructions are sent to the smartphone, allowing the user to learn the correct sorting method in real time.

[0651] Prompt Sentence Examples

[0652] An example prompt for uploading an image of a plastic bottle is as follows:

[0653] "I'll upload a picture of a plastic bottle. How do I dispose of this plastic bottle?"

[0654] This allows users to easily obtain accurate sorting instructions in accordance with local sorting rules.

[0655] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0656] Step 1:

[0657] Users take pictures of the waste using their smartphone camera and upload them to a server via the app.

[0658] Input: Image of waste taken with a smartphone

[0659] Output: Image data sent to the server

[0660] Specific operation: The user launches the camera app on their smartphone, takes a photo of the waste, and then presses the upload button to send the image to the server.

[0661] Step 2:

[0662] The server receives the image data sent by the user and temporarily stores it.

[0663] Input: Image data sent by the user

[0664] Output: Image data temporarily stored on the server

[0665] Specific operation: The server receives the image data, temporarily stores it in a database or file system, and records a receipt confirmation in a log.

[0666] Step 3:

[0667] The server passes the temporarily stored image data to an image recognition module to identify the type of waste.

[0668] Input: Temporarily saved image data

[0669] Output: Analysis results (type of waste)

[0670] How it works: The server sends image data to the image recognition module, which analyzes the image using a generative AI model. The image recognition module identifies the type of waste and returns that information to the server.

[0671] Step 4:

[0672] The server searches the database based on the type of waste obtained from the image recognition module to obtain the corresponding sorting method.

[0673] Input: Analysis results (type of waste)

[0674] Output: Sorting method

[0675] Specific operation: The server searches the database to obtain the classification method corresponding to the analysis result, and then extracts the appropriate classification method by referring to the rules and instructions in the database.

[0676] Step 5:

[0677] Based on the sorting method obtained by the server, the AI ​​inference engine generates specific instructions for the user in natural language.

[0678] Input: Sorting method

[0679] Output: Separation instructions for the user

[0680] How it works: The AI ​​inference engine analyzes the sorting method and uses a generative AI model to generate instructions in natural language that are easy for the user to understand, such as "Remove the cap and label, and separate the bottle part as plastic."

[0681] Step 6:

[0682] The server sends the generated instructions to the user's smartphone and displays them in real time.

[0683] Input: Separation instructions for the user

[0684] Output: Instructions displayed on the smartphone

[0685] Specific operation: The server sends the generated instruction text to the user's smartphone via the notification system. The smartphone app receives the instruction text and displays it on the screen in real time.

[0686] This processing flow allows users to learn in real time how to properly separate the waste they have photographed, enabling them to dispose of it correctly in accordance with their local rules.

[0687] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0688] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. By using the emotion engine, it becomes possible to provide more flexible and appropriate instructions according to the user's emotional state.

[0689] System configuration

[0690] The system of the present invention mainly includes the following components:

[0691] 1. User device: A device where a user takes a photo of the garbage and uploads it to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[0692] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[0693] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0694] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[0695] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[0696] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[0697] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0698] Program processing

[0699] 1. User uploads a photo

[0700] Using a smartphone app, users take or select an image of the trash, which also captures facial expressions and audio data.

[0701] The terminal transmits the image file and emotion data to the server.

[0702] 2. The server receives the photo

[0703] The server receives the image file and emotion data sent from the terminal and temporarily stores them.

[0704] 3. The image recognition module analyzes the photo

[0705] The server passes the temporarily stored image data to the image recognition module.

[0706] An image recognition module analyzes the image and identifies the type of waste.

[0707] 4. Emotion engine analyzes emotions

[0708] The server passes the temporarily stored emotional data (facial expressions and voice) to the emotion engine.

[0709] The emotion engine analyzes this data to determine the user's emotional state.

[0710] 5. Check against the database

[0711] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[0712] 6. AI generates instructions

[0713] The server generates instructions for the user based on the image recognition results, the emotion engine results, and classification methods from the database. These instructions take into account the user's emotional state.

[0714] 7. Notify users

[0715] The server transmits the generated instruction text to the user terminal.

[0716] The device will then display the received instructions to the user within the app.

[0717] Specific examples

[0718] Scenario: A user wants to dispose of a plastic bottle.

[0719] 1. User uploads a photo

[0720] Users take a photo of the bottle using their smartphone camera and send it to the server via the app, which also captures their facial expressions and voice.

[0721] 2. The server receives the photo

[0722] The server receives the photo and emotion data and temporarily stores them.

[0723] 3. The image recognition module analyzes the photo

[0724] The server sends the photo to an image recognition module for analysis.

[0725] The module analyzes the photo and identifies it as a plastic bottle.

[0726] 4. Emotion engine analyzes emotions

[0727] The server sends the emotion data to the emotion engine for analysis.

[0728] The emotion engine identifies that the user is stressed.

[0729] 5. Check against the database

[0730] The server searches the database based on the results identified as a plastic bottle.

[0731] The database returns the data: "Remove the cap and label, and separate the bottle part as plastic."

[0732] 6. AI generates instructions

[0733] The AI ​​inference engine uses these results to generate instructions for the user. Based on the emotion engine's results, the instructions are adjusted to "Recycle in a few easy steps. Remove the cap and label and separate the bottle part as plastic."

[0734] 7. Notify users

[0735] The server sends this instruction to the user terminal.

[0736] The device will display these instructions within the app, making it easy for users to follow the proper sorting method.

[0737] The processing flow will be explained below.

[0738] Step 1:

[0739] A user launches the smartphone app and taps the "Upload Photo" button. They can then take a photo of the litter or select one from their existing photo gallery. The app also simultaneously captures the user's facial expressions and voice.

[0740] Step 2:

[0741] The device acquires the selected or taken photo data and the captured facial expression and voice data, generates an API request to send this data to the server, and sends it to the server via the Internet.

[0742] Step 3:

[0743] The server receives the API request, analyzes the uploaded photo data and emotion data, and temporarily stores this data.

[0744] Step 4:

[0745] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[0746] Step 5:

[0747] The server receives the analysis results from the image recognition module and identifies the garbage in the photo as a plastic bottle.

[0748] Step 6:

[0749] The server sends the temporarily stored facial expression and voice data to the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., stress, comfort).

[0750] Step 7:

[0751] The server generates an SQL query to retrieve an appropriate sorting method from the database based on the recognition result (plastic bottle) and emotion data (e.g., stress state). The server then sends this query to the database.

[0752] Step 8:

[0753] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[0754] Step 9:

[0755] The server generates instructions for the user based on the sorting method data returned from the database. At this time, the server uses natural language generation technology to adjust the instructions to take into account the user's emotional state. For example, "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic."

[0756] Step 10:

[0757] The server creates an API response containing the generated instruction and sends it to the user's device.

[0758] Step 11:

[0759] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app interface displays the message, "Recycling is easy. Remove the cap and label, and separate the bottle part as plastic."

[0760] Step 12:

[0761] Users follow the instructions provided to them to separate their garbage, making it easy for them to separate their garbage accurately.

[0762] Example 2

[0763] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0764] Conventional garbage sorting systems not only do not provide sufficient instructions for users to accurately sort waste types, but also lack the flexibility to take into account the user's emotional state, making it difficult for users to sort their waste efficiently and without stress.

[0765] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and temporarily storing images and emotional data taken or selected by the user, means for analyzing the temporarily stored images to identify the type of waste, means for analyzing the temporarily stored emotional data to identify the user's emotional state, means for acquiring a sorting method from a database based on the identified type of waste and the user's emotional state, means for using an AI inference engine to generate appropriate instructions that take into account the user's emotional state based on the acquired sorting method, and means for notifying the user terminal of the generated instructions. This allows the user to receive flexible and appropriate sorting instructions that take into account their own emotional state, allowing for stress-free and efficient sorting work.

[0766] "User" refers to a person who uses the system to take or select images of waste and receive instructions on how to sort it.

[0767] "Device" refers to a device used by a User, such as a smartphone or tablet, that is capable of taking and sending images, capturing emotional data, and receiving notifications.

[0768] "Server" refers to a computer system that receives image data and emotion data, temporarily stores them, and analyzes them.

[0769] "Image recognition module" refers to a software module that runs on a server and has the function of analyzing received image data using a machine learning model and identifying the type of waste.

[0770] An "emotion engine" refers to a deep learning model that runs on a server and has the ability to analyze received facial expressions and voice data to identify the user's emotional state.

[0771] "Database" refers to an information repository that systematically stores waste sorting methods according to type and other related information.

[0772] An "AI inference engine" refers to an algorithm or program that generates instructions for users based on the analysis results of the image recognition module and emotion engine and information from a database.

[0773] "Notification system" refers to a module that has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0774] "Emotional Data" refers to a user's facial expressions and voice data, and means data used to analyze a user's emotional state.

[0775] A "machine learning model" refers to statistical methods and algorithms that learn patterns from data and perform image recognition and emotion analysis.

[0776] A "deep learning model" refers to an algorithm that uses artificial neuron networks to perform more advanced data analysis.

[0777] "Temporary storage" means that the image data and emotion data received by the server are temporarily stored until analysis is complete.

[0778] "Sorting method" refers to the appropriate disposal method for each type of waste, and is information obtained from the database.

[0779] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. This system makes it possible to provide flexible and appropriate instructions according to the user's emotional state.

[0780] System configuration

[0781] The system of the present invention mainly includes the following components:

[0782] 1. User device: This is a device that users use to take photos of garbage and upload them to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[0783] 2. Server: Receives and temporarily stores image data and emotion data, and performs the necessary processing for the image recognition module, emotion engine, database, and AI inference engine.

[0784] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0785] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[0786] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[0787] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[0788] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0789] System operation explanation

[0790] Users take photos of waste using their smartphone camera and send them to the server via the application. During the transmission, facial expressions and voice data are also captured. The server receives and temporarily stores this data.

[0791] The server then passes the temporarily stored image data to an image recognition module, which uses a machine learning model to identify the type of waste. At the same time, it also passes the emotion data to an emotion engine, which uses a deep learning model to analyze the user's emotional state.

[0792] The analysis results are compared with the database, and the server uses an AI inference engine to generate instructions for the user. These instructions are based on the analyzed emotional data and take into account the user's emotional state, such as "Recycle in a few easy steps." Finally, the generated instructions are sent to the user's device and displayed within the app.

[0793] Specific examples

[0794] For example, if a user wants to throw away a plastic bottle, the following actions occur:

[0795] 1. The user takes a photo of a plastic bottle using their smartphone camera and sends the photo and emotional data to the server via the app.

[0796] 2. The server receives the photo and emotion data and temporarily stores them.

[0797] 3. The server sends the image data to the image recognition module, which identifies it as a plastic bottle.

[0798] 4. The server sends the emotion data to the emotion engine, which determines that the user is stressed.

[0799] 5. The server searches the database to find out how to separate plastic bottles. The database returns the information that says, "Remove the cap and label, and separate the bottle as plastic."

[0800] 6. The AI ​​inference engine generates the following instruction: "Recycle in a few simple steps. Remove the cap and label, and separate the bottle part as plastic."

[0801] 7. The server sends this instruction to the user's device, which displays the instruction within the app, allowing the user to view the notification and take appropriate action.

[0802] Prompt Sentence Examples

[0803] Please explain the process by which a user takes a photo of the garbage they want to dispose of, sends emotional data to the server, and receives sorting instructions based on their emotional state.

[0804] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0805] Step 1:

[0806] The user uploads a photo.

[0807] How it works: The user takes a photo of the waste using their smartphone camera, which also captures facial expressions and voice data. The user then presses the upload button within the app, which sends this data to the server.

[0808] Input: Garbage images, facial expression data, and audio data.

[0809] Output: Image files and emotion data sent to the server.

[0810] Step 2:

[0811] The server receives the photos and temporarily stores them.

[0812] Specific operation: The server receives the image file and emotion data sent from the device and temporarily stores them. Upon receiving the data, it checks for consistency and whether there are any missing data.

[0813] Input: Image file and emotion data sent from the user device.

[0814] Output: Image files and emotion data saved in temporary storage.

[0815] Step 3:

[0816] The image recognition module analyzes the photo.

[0817] How it works: The server passes the image data stored in the temporary storage area to the image recognition module, which uses a pre-trained machine learning model to analyze the image and identify the type of waste.

[0818] Input: Image data retrieved from temporary storage.

[0819] Output: Identified waste type (e.g., PET bottles, paper, plastic).

[0820] Step 4:

[0821] The emotion engine analyzes emotions.

[0822] How it works: The server passes the emotional data (facial expressions and voice data) stored in the temporary storage area to the emotion engine, which then uses a deep learning model to analyze the data and identify the user's emotional state.

[0823] Input: Emotion data retrieved from temporary storage.

[0824] Output: Identified user emotional state (e.g., stressed, relaxed, angry).

[0825] Step 5:

[0826] The server checks the database.

[0827] Specific operation: The server searches the database based on the type of waste obtained from the image recognition module. The database stores appropriate sorting methods and related information for each type of waste. The server retrieves the corresponding sorting method.

[0828] Input: Identified waste type.

[0829] Output: The corresponding sorting method (e.g. remove cap and label, and sort the bottle as plastic).

[0830] Step 6:

[0831] An AI inference engine generates instructions.

[0832] Specific operation: The server uses an AI inference engine to generate instructions for the user based on the image recognition results, the emotion engine results, and the classification method retrieved from the database. These instructions take into account the user's emotional state.

[0833] Input: Image recognition results, emotion engine results, classification methods from the database.

[0834] Output: Instructions for the user (e.g., "Recycle in a few easy steps. Remove the cap and label and separate the bottle for plastic.").

[0835] Step 7:

[0836] Notify the user.

[0837] Specific operation: The server sends the generated instruction text to the user's device, which then displays the instruction within the app and suggests appropriate sorting methods to the user.

[0838] Input: Instructions sent by the server.

[0839] Output: Instructions displayed on the user's terminal.

[0840] (Application example 2)

[0841] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0842] In recent years, with growing interest in environmental issues, accurate waste separation is becoming increasingly important. However, many users do not properly understand the types of waste and how to separate them, leading to incorrect separation. Furthermore, users' emotional state can sometimes affect their waste separation behavior. This situation hinders effective waste management. The present invention aims to solve these problems and provide a system that supports users in easily separating waste appropriately.

[0843] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken or selected by the user, means for acquiring and analyzing the user's emotional data, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for acquiring a sorting method from a database based on the identified type of waste, and means for providing the user with appropriate instructions based on the acquired sorting method and the user's emotional state. This makes it possible to provide flexible and appropriate sorting instructions based on the type of waste and the user's emotional state.

[0844] A "user" is an individual who uses the system to take or select images of waste and learn how to separate it.

[0845] "Waste" means items that are no longer needed or useful and that need to be disposed of in an environmentally friendly manner.

[0846] "Means for uploading images" refers to a function or device for transmitting image data taken or selected by a user to a server.

[0847] "Emotion data" is information that indicates the emotional state of the user, and is acquired based on facial expressions and voice data.

[0848] The "means for receiving and temporarily storing images" refers to a function or device that temporarily stores uploaded image data.

[0849] The "means for analyzing images and identifying the type of waste" refers to a function or device for identifying the type of waste from temporarily stored image data using a machine learning model.

[0850] The "database" is an information repository that systematically stores waste sorting methods according to type and other related information.

[0851] The "means for obtaining a sorting method" is a function or device that searches for and obtains an appropriate sorting method from a database based on the type of waste identified.

[0852] The "means for providing appropriate instructions" is a function or device that generates and conveys easy-to-understand instructions to the user based on the acquired discernment method and the user's emotional state.

[0853] This invention relates to an application that helps users properly separate waste generated during food delivery. Specifically, it provides a system that analyzes images of waste and provides guidance on the appropriate separation method based on the user's emotional state.

[0854] System configuration

[0855] The system of the present invention includes the following components:

[0856] 1. User device: This is the device on which the user takes a picture of the trash and acquires emotion data. This device can be a smartphone or tablet.

[0857] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[0858] 3. Image Recognition Module: This module analyzes uploaded images and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0859] 4. Emotion Engine: This engine has the ability to analyze the user's emotional state. It uses a deep learning model to perform emotion analysis.

[0860] 5. Database: Contains information on how to separate waste by type and related information.

[0861] 6. AI Inference Engine: Based on the results of the image recognition module and emotion engine, it generates specific and emotion-sensitive instructions for the user.

[0862] 7. Notification system: The generated instructions are sent to the user's terminal, and the user is informed of how to sort the waste.

[0863] System Operation

[0864] 1. User device operation:

[0865] Users use their smartphone camera to take pictures of the waste and capture facial expressions and voice as emotional data.

[0866] The data is then uploaded to the server through the application.

[0867] 2. Receiving and temporary storage on the server:

[0868] The server temporarily stores the uploaded image data and emotion data.

[0869] 3. Image Recognition Module Processing:

[0870] The temporarily stored image data is analyzed by an image recognition module to identify the type of waste.

[0871] 4. Emotion Engine Processing:

[0872] The emotion data is analyzed by an emotion engine to determine the user's emotional state.

[0873] 5. Database verification:

[0874] The server searches a database based on the identified type of waste and obtains the appropriate sorting method.

[0875] 6. Generate instructions:

[0876] The AI ​​inference engine generates easy-to-understand instructions based on the type of waste and the user's emotional state.

[0877] 7. Notice to Users:

[0878] The server then sends the generated instructions to the user's device and displays them within the app, allowing the user to learn how to properly separate their trash.

[0879] Specific examples

[0880] For example, consider the case where a user is about to dispose of a plastic bottle. The user takes a picture of the bottle, captures facial expressions and voice as emotional data, and uploads it to the server. The server uses an image recognition module to recognize the bottle, and an emotion engine to identify that the user is feeling stressed. The database is then retrieved to provide instructions such as "Remove the cap and label, and separate the bottle part as plastic," and the AI ​​inference engine generates instructions such as "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic." These instructions are displayed on the user's device, allowing the user to properly separate their waste.

[0881] Prompt Sentence Examples

[0882] 1. "waste_image.jpg" is an image of a plastic bottle taken by a user.

[0883] 2. User's face photo "user_image.jpg".

[0884] 3. User's audio file "user_audio.wav".

[0885] Using this data, the system suggests appropriate ways to dispose of plastic bottles, taking into account the user's emotional state.

[0886] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0887] Step 1:

[0888] Users take images of waste using their smartphone cameras and capture facial expressions and voice as emotional data. Users then upload these data to a server via an application. The input is waste images, facial expressions, and voice data, and the output is the uploaded data.

[0889] Step 2:

[0890] The server receives and temporarily stores image data and emotion data uploaded by the user. The input of this step is the uploaded image data and emotion data, and the output is the temporarily stored data.

[0891] Step 3:

[0892] The server passes the temporarily stored image data to an image recognition module, which uses machine learning models to analyze these images and identify the waste type. The input is the temporarily stored image data, and the output is the identified waste type.

[0893] Step 4:

[0894] The server passes the stored emotional data to the emotion engine, which uses a deep learning model to analyze the data and identify the user's emotional state. The input is the stored emotional data, and the output is the user's emotional state.

[0895] Step 5:

[0896] The server searches the database based on the identified waste type to obtain the corresponding sorting method. The input is the identified waste type, and the output is the corresponding sorting method.

[0897] Step 6:

[0898] The server uses an AI inference engine to generate specific and emotion-sensitive instructions for users based on the image recognition results, the emotion engine results, and the sorting method from the database. The input is the type of waste, the user's emotional state, and the sorting method, and the output is the generated instructions.

[0899] Step 7:

[0900] The server sends the generated instructions to the user's device, which then displays the received instructions to the user within the app. The input is the generated instructions, and the output is the sorting instructions displayed to the user.

[0901] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0902] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0903] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0904] [Third embodiment]

[0905] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0906] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0907] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0908] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0909] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0910] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0911] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0912] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0913] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0914] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0915] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0916] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0917] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate method for sorting waste.

[0918] System configuration

[0919] The system of the present invention mainly includes the following components:

[0920] 1. User device: This is the device that users use to take photos of trash and upload them to the server via the application. This can be a smartphone or tablet.

[0921] 2. Server: This is the central processing unit that receives and temporarily stores image data, and also performs the necessary processing for the image recognition module and database.

[0922] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[0923] 4. Database: An information repository that stores the methods for separating waste according to type, providing information on how to separate identified waste.

[0924] 5. AI inference engine: Generates specific instructions for the user based on the results obtained from the image recognition module and information retrieved from the database.

[0925] 6. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[0926] Program processing

[0927] 1. User uploads a photo

[0928] Users use a smartphone app to take or select an image of the trash.

[0929] The terminal transmits this image file to the server.

[0930] 2. The server receives the photo

[0931] The server receives the image file sent from the terminal and temporarily stores it.

[0932] 3. The image recognition module analyzes the photo

[0933] The server passes the temporarily stored image to the image recognition module.

[0934] An image recognition module analyzes the image and identifies the type of waste.

[0935] 4. Check against the database

[0936] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[0937] 5. AI generates instructions

[0938] Based on the classification method obtained, the server uses natural language generation technology to generate appropriate instructions for the user.

[0939] 6. Notify users

[0940] The server transmits the generated instructions to the user terminal.

[0941] The device will then display the received instructions to the user within the app.

[0942] Specific examples

[0943] Scenario: A user wants to dispose of a plastic bottle.

[0944] 1. User uploads a photo

[0945] Users take a photo of the plastic bottle using their smartphone camera and send it to the server via the app.

[0946] 2. The server receives the photo

[0947] The server receives this photo and temporarily stores it.

[0948] 3. The image recognition module analyzes the photo

[0949] The server sends the photo to an image recognition module for analysis.

[0950] The module analyzes the photo and identifies it as a plastic bottle.

[0951] 4. Check against the database

[0952] The server searches the database based on the results identified as a plastic bottle.

[0953] The database sends the result to the server: "Remove the cap and label and separate the bottle part as plastic."

[0954] 5. AI generates instructions

[0955] The AI ​​inference engine uses the results to generate instructions for the user.

[0956] The generated instructions are: "Remove the cap and label and separate the bottle as plastic."

[0957] 6. Notify users

[0958] The server sends this instruction to the user terminal.

[0959] The device will display these instructions to the user within the app, so they know how to properly separate their waste.

[0960] The processing flow will be explained below.

[0961] Step 1:

[0962] A user launches the smartphone app and taps the "Upload Photo" button. The user then takes a photo of the trash or selects one from an existing photo gallery.

[0963] Step 2:

[0964] The device acquires the selected or taken photo data and generates an API request to send it to the server. The device then sends this request to the server via the Internet.

[0965] Step 3:

[0966] The server receives the API request, analyzes the uploaded photo data, and stores the photo data in temporary storage.

[0967] Step 4:

[0968] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[0969] Step 5:

[0970] The server receives the analysis results from the image recognition module, which identifies the trash in the photo as a plastic bottle.

[0971] Step 6:

[0972] Based on the recognition results, the server generates an SQL query to retrieve the appropriate classification method from the database, and sends this query to the database.

[0973] Step 7:

[0974] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[0975] Step 8:

[0976] The server generates instructions for the user based on the sorting method data returned from the database, using natural language generation technology.

[0977] Step 9:

[0978] The server creates an API response containing the generated instruction text and sends it to the device.

[0979] Step 10:

[0980] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app displays the message "Please remove the cap and label and separate the bottle part as plastic." on the app interface.

[0981] Step 11:

[0982] The user follows the instructions provided to separate the garbage, making it easy for the user to separate the garbage accurately.

[0983] Example 1

[0984] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0985] Current waste sorting systems have problems, such as users finding it difficult to understand how to sort their waste, making it difficult to dispose of waste efficiently. Furthermore, manually identifying the type of waste and identifying the appropriate sorting method based on that is time-consuming and inconvenient for users.

[0986] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0987] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for generating appropriate instructions for the user based on the retrieved sorting method, and means for notifying the generated instructions to a user terminal. This allows the user to easily learn the appropriate method for sorting waste, enabling efficient and accurate waste disposal.

[0988] "User" means an individual or organization that uses the system to separate waste.

[0989] "Image" refers to photographic data of waste taken or selected by the user.

[0990] "Uploading means" refers to a function or device for transmitting image data from a user terminal to a server.

[0991] "Means for receiving and temporarily storing" refers to a function or device that allows the server to receive image data sent from a user terminal and temporarily store it.

[0992] "Means for analyzing" refers to a system or function that allows the server to analyze temporarily stored images and automatically identify the type of waste.

[0993] A "machine learning model" is an algorithm or statistical model that the server uses to identify types of waste, and has knowledge learned from image data.

[0994] "Database" refers to an information repository that stores sorting methods for different types of waste.

[0995] "Means for obtaining a sorting method" refers to the function of the server to search and obtain an appropriate sorting method from the database based on the type of waste identified.

[0996] "Natural language generation technology" refers to technology that generates easy-to-understand instructions for users based on the classification method acquired by the server.

[0997] "Means for generating" refers to the function of generating specific instructions for the user based on the sorting method obtained by the server.

[0998] "Means for notifying" refers to the function of sending instructions generated by the server to the user terminal and notifying the user.

[0999] "User terminal" refers to the device that a user uses to take or select images of waste and upload them to the server. This could be a smartphone or tablet.

[1000] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate waste sorting method. Specific embodiments for implementing this system will be described below.

[1001] The system mainly includes the following components:

[1002] 1. User Device:

[1003] This is a device that allows users to take photos of waste and upload them to a server via an application. Specifically, it is a smartphone or tablet. The user device has a camera function and an internet connection, so the captured images can be quickly sent to the server.

[1004] 2. Server:

[1005] The server receives and temporarily stores image data. It is also a central processing unit that performs the necessary processing for the image recognition module and database. The server runs on a web server and efficiently manages the reception and processing of data.

[1006] 3. Image Recognition Module:

[1007] This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models (e.g., TensorFlow or PyTorch) to perform image analysis. The image recognition module uses a trained model to detect the characteristics of the waste and classify it into a specific category.

[1008] 4. Database:

[1009] The database is an information repository that stores waste separation methods according to the type of waste. It provides information on the separation method for the identified waste. The database contains the appropriate separation method for each type of waste and has an index for searching them.

[1010] 5. AI inference engine:

[1011] The AI ​​inference engine generates specific instructions for the user based on the results from the image recognition module and information retrieved from the database, using natural language generation techniques (e.g., generative AI models such as GPT-3) to generate instructions in a form that is easy for humans to understand.

[1012] 6. Notification System:

[1013] The notification system sends instructions generated by the server to the user's device and provides instructions on how to sort the waste. The notification system uses in-app notifications and push notifications to quickly and reliably inform users.

[1014] Specific examples

[1015] Scenario: A user wants to dispose of a plastic bottle.

[1016] 1. User uploads a photo:

[1017] Users take a photo of the plastic bottle with their smartphone camera and send it to the server via the app.

[1018] 2. The server receives and temporarily stores the photos:

[1019] The server receives this photo and temporarily stores it in the "uploads" folder.

[1020] 3. The image recognition module analyzes the photo:

[1021] The server sends the photo to an image recognition module, which uses TensorFlow to identify it as a plastic bottle.

[1022] 4. The server checks the database and obtains the classification method:

[1023] The server queries the database and obtains the sorting method: "Remove the cap and label, and separate the bottle part as plastic."

[1024] 5. The AI ​​inference engine generates instructions:

[1025] Based on the results obtained, the AI ​​inference engine generates instructions such as, "Remove the cap and label and separate the bottle part as plastic."

[1026] 6. The server notifies the user:

[1027] The server sends the generated instructions to the user's device, and the user receives the instructions within the app.

[1028] Example prompt statement

[1029] Input: A user-taken image of a plastic bottle

[1030] Output: Instructions based on image analysis results: "Remove the cap and label and separate the bottle as plastic."

[1031] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1032] Step 1:

[1033] User uploads a photo

[1034] Input: Images of waste taken by the user with a smartphone camera

[1035] Specific operation: The user launches the app on their smartphone and takes a photo of the waste or selects an existing image. The taken or selected image is displayed in the app. The user presses the "Upload" button, and the image file is sent to the server.

[1036] Output: Image data sent from the device to the server

[1037] Step 2:

[1038] The server receives the photo and temporarily stores it.

[1039] Input: Image data sent from the device

[1040] Specific operation: The server receives the image data sent from the device and temporarily stores it in memory or storage. Once the server has received the image data, it registers the image metadata in a database. This metadata includes the image storage location, the date and time of transmission, etc.

[1041] Output: Temporarily saved image data and its metadata

[1042] Step 3:

[1043] Image recognition module analyzes the photo

[1044] Input: Temporarily saved image data

[1045] Specific operation: The server sends the temporarily stored image data to the image recognition module. The image recognition module analyzes the image using a machine learning model (e.g., TensorFlow or PyTorch). The model extracts waste characteristics from the image and identifies the type of waste. An analysis result (e.g., "plastic bottle") is generated.

[1046] Output: Analysis results identifying the type of waste

[1047] Step 4:

[1048] The server checks the database and obtains the sorting method.

[1049] Input: Analysis results from the image recognition module (type of waste)

[1050] Specific operation: The server searches a database based on the analysis results obtained from the image recognition module. The database stores sorting methods for each type of waste. The server retrieves the sorting method corresponding to the type of waste from the database.

[1051] Output: The obtained sorting method (e.g., "Remove the cap and label, and sort the bottle part as plastic")

[1052] Step 5:

[1053] AI inference engine generates instructions

[1054] Input: Obtained sorting method

[1055] Specific operation: The AI ​​inference engine uses natural language generation technology (for example, a generative AI model such as GPT-3) to generate instructions for the user based on the acquired sorting method. The generated instructions are expressed in a form that is easy for humans to understand. For example, the generated instruction might be, "Remove the cap and label, and separate the bottle part as plastic."

[1056] Output: Generated directives

[1057] Step 6:

[1058] The server notifies the user

[1059] Input: Generated instructions

[1060] Specific operation: The server sends the generated instruction text to the user's device. Using the notification system, the in-app notification function or push notification function is used to quickly and reliably convey information to the user.

[1061] Output: Instructions displayed on the user's terminal

[1062] This allows users to easily understand and practice proper waste separation methods.

[1063] (Application example 1)

[1064] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1065] Existing systems that provide users with proper waste sorting methods often use fixed devices and lack real-time operability. This creates a problem when users want to know how to sort their waste immediately, especially in physical stores and public places. Furthermore, there are only a limited number of systems that support waste sorting rules that vary from region to region, making it difficult for users to accurately sort their waste in accordance with their local rules.

[1066] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1067] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for providing appropriate instructions to the user based on the retrieved sorting method, means for displaying the sorting method on a device held by the user in real time, and means for customizing the sorting method based on the sorting rules of the area to which the user belongs. This allows the user to obtain an appropriate sorting method in real time, enabling accurate waste sorting suited to their area.

[1068] "Means for uploading images taken or selected by the user" refers to a function that allows a user to capture images of waste using a device such as a smartphone or smart glasses and send the images to the server.

[1069] "Means for receiving and temporarily storing uploaded images" refers to a function by which the server receives image data sent by a user and temporarily stores it for analysis.

[1070] "Means for analyzing temporarily stored images and identifying the type of waste" refers to a function that analyzes temporarily stored image data using machine learning models and image recognition algorithms to determine the type of waste indicated by the image.

[1071] "Means for retrieving sorting methods from a database based on the type of waste identified" is a function that searches for and retrieves appropriate sorting methods from a database based on the type of waste identified through analysis.

[1072] "Means for providing appropriate instructions to the user based on the obtained sorting method" is a function for generating and providing specific instructions to the user based on the sorting method obtained from the database.

[1073] "Means for displaying the sorting method in real time on a device held by the user" refers to a function for instantly displaying the acquired sorting method on a user device such as a smartphone or smart glasses.

[1074] "Means for customizing sorting methods based on the sorting rules of the area to which the user belongs" is a function for customizing appropriate sorting methods in accordance with the specific sorting rules of the area in which the user resides and providing that information.

[1075] This invention is a system that analyzes images of waste taken or selected by a user and provides sorting methods in real time. The system includes a user terminal, a server, an image recognition module, a database, an AI inference engine, and a notification system.

[1076] The system functions as follows:

[1077] User Device

[1078] The user device refers to a smartphone or smart glasses, and is responsible for allowing users to take images of waste and upload them to the server. When users take images of waste using the device's camera, the image data is saved on the device.

[1079] server

[1080] The server receives image data sent from the user terminal and temporarily stores it. It functions as a central processing unit for the image recognition module, database, AI inference engine, etc. After receiving the image data, the server sends the image data to the image recognition module.

[1081] Image Recognition Module

[1082] The image recognition module uses machine learning models to analyze image data submitted by users and identify the type of waste. This module uses generative AI models that have been trained on a wide range of waste data to perform highly accurate image analysis.

[1083] Database

[1084] The database stores the correct method for separating waste based on the analysis results. The server searches the database based on the results obtained from the image recognition module and obtains information on the corresponding separation method.

[1085] AI inference engine

[1086] Based on the classification methods retrieved from the database, the AI ​​inference engine uses natural language generation technology to generate specific instructions for the user, which are presented in a format that is easy for the user to understand.

[1087] Notification System

[1088] The notification system sends instructions generated by the server to the user's device, where the instructions are displayed in real time and users can receive guidance on how to properly separate their waste.

[1089] Specific examples

[1090] Here is a specific example using a smartphone.

[1091] When a user disposes of a plastic bottle purchased at a supermarket, they take a photo of the bottle with their smartphone camera. The photo is then sent to a server via the app. The server receives the photo and temporarily stores it. Next, an image recognition module analyzes the photo and identifies it as a plastic bottle. Based on this identification, the server searches a database for the corresponding sorting method, and uses an AI inference engine to generate instructions such as "Remove the cap and label, and separate the bottle portion as plastic." These instructions are sent to the smartphone, allowing the user to learn the correct sorting method in real time.

[1092] Prompt Sentence Examples

[1093] An example prompt for uploading an image of a plastic bottle is as follows:

[1094] "I'll upload a picture of a plastic bottle. How do I dispose of this plastic bottle?"

[1095] This allows users to easily obtain accurate sorting instructions in accordance with local sorting rules.

[1096] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1097] Step 1:

[1098] Users take pictures of the waste using their smartphone camera and upload them to a server via the app.

[1099] Input: Image of waste taken with a smartphone

[1100] Output: Image data sent to the server

[1101] Specific operation: The user launches the camera app on their smartphone, takes a photo of the waste, and then presses the upload button to send the image to the server.

[1102] Step 2:

[1103] The server receives the image data sent by the user and temporarily stores it.

[1104] Input: Image data sent by the user

[1105] Output: Image data temporarily stored on the server

[1106] Specific operation: The server receives the image data, temporarily stores it in a database or file system, and records a receipt confirmation in a log.

[1107] Step 3:

[1108] The server passes the temporarily stored image data to an image recognition module to identify the type of waste.

[1109] Input: Temporarily saved image data

[1110] Output: Analysis results (type of waste)

[1111] How it works: The server sends image data to the image recognition module, which analyzes the image using a generative AI model. The image recognition module identifies the type of waste and returns that information to the server.

[1112] Step 4:

[1113] The server searches the database based on the type of waste obtained from the image recognition module to obtain the corresponding sorting method.

[1114] Input: Analysis results (type of waste)

[1115] Output: Sorting method

[1116] Specific operation: The server searches the database to obtain the classification method corresponding to the analysis result, and then extracts the appropriate classification method by referring to the rules and instructions in the database.

[1117] Step 5:

[1118] Based on the sorting method obtained by the server, the AI ​​inference engine generates specific instructions for the user in natural language.

[1119] Input: Sorting method

[1120] Output: Separation instructions for the user

[1121] How it works: The AI ​​inference engine analyzes the sorting method and uses a generative AI model to generate instructions in natural language that are easy for the user to understand, such as "Remove the cap and label, and separate the bottle part as plastic."

[1122] Step 6:

[1123] The server sends the generated instructions to the user's smartphone and displays them in real time.

[1124] Input: Separation instructions for the user

[1125] Output: Instructions displayed on the smartphone

[1126] Specific operation: The server sends the generated instruction text to the user's smartphone via the notification system. The smartphone app receives the instruction text and displays it on the screen in real time.

[1127] This processing flow allows users to learn in real time how to properly separate the waste they have photographed, enabling them to dispose of it correctly in accordance with their local rules.

[1128] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1129] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. By using the emotion engine, it becomes possible to provide more flexible and appropriate instructions according to the user's emotional state.

[1130] System configuration

[1131] The system of the present invention mainly includes the following components:

[1132] 1. User device: A device where a user takes a photo of the garbage and uploads it to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[1133] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[1134] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[1135] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[1136] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[1137] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[1138] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[1139] Program processing

[1140] 1. User uploads a photo

[1141] Using a smartphone app, users take or select an image of the trash, which also captures facial expressions and audio data.

[1142] The terminal transmits the image file and emotion data to the server.

[1143] 2. The server receives the photo

[1144] The server receives the image file and emotion data sent from the terminal and temporarily stores them.

[1145] 3. The image recognition module analyzes the photo

[1146] The server passes the temporarily stored image data to the image recognition module.

[1147] An image recognition module analyzes the image and identifies the type of waste.

[1148] 4. Emotion engine analyzes emotions

[1149] The server passes the temporarily stored emotional data (facial expressions and voice) to the emotion engine.

[1150] The emotion engine analyzes this data to determine the user's emotional state.

[1151] 5. Check against the database

[1152] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[1153] 6. AI generates instructions

[1154] The server generates instructions for the user based on the image recognition results, the emotion engine results, and classification methods from the database. These instructions take into account the user's emotional state.

[1155] 7. Notify users

[1156] The server transmits the generated instruction text to the user terminal.

[1157] The device will then display the received instructions to the user within the app.

[1158] Specific examples

[1159] Scenario: A user wants to dispose of a plastic bottle.

[1160] 1. User uploads a photo

[1161] Users take a photo of the bottle using their smartphone camera and send it to the server via the app, which also captures their facial expressions and voice.

[1162] 2. The server receives the photo

[1163] The server receives the photo and emotion data and temporarily stores them.

[1164] 3. The image recognition module analyzes the photo

[1165] The server sends the photo to an image recognition module for analysis.

[1166] The module analyzes the photo and identifies it as a plastic bottle.

[1167] 4. Emotion engine analyzes emotions

[1168] The server sends the emotion data to the emotion engine for analysis.

[1169] The emotion engine identifies that the user is stressed.

[1170] 5. Check against the database

[1171] The server searches the database based on the results identified as a plastic bottle.

[1172] The database returns the data: "Remove the cap and label, and separate the bottle part as plastic."

[1173] 6. AI generates instructions

[1174] The AI ​​inference engine uses these results to generate instructions for the user. Based on the emotion engine's results, the instructions are adjusted to "Recycle in a few easy steps. Remove the cap and label and separate the bottle part as plastic."

[1175] 7. Notify users

[1176] The server sends this instruction to the user terminal.

[1177] The device will display these instructions within the app, making it easy for users to follow the proper sorting method.

[1178] The processing flow will be explained below.

[1179] Step 1:

[1180] A user launches the smartphone app and taps the "Upload Photo" button. They can then take a photo of the litter or select one from their existing photo gallery. The app also simultaneously captures the user's facial expressions and voice.

[1181] Step 2:

[1182] The device acquires the selected or taken photo data and the captured facial expression and voice data, generates an API request to send this data to the server, and sends it to the server via the Internet.

[1183] Step 3:

[1184] The server receives the API request, analyzes the uploaded photo data and emotion data, and temporarily stores this data.

[1185] Step 4:

[1186] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[1187] Step 5:

[1188] The server receives the analysis results from the image recognition module and identifies the garbage in the photo as a plastic bottle.

[1189] Step 6:

[1190] The server sends the temporarily stored facial expression and voice data to the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., stress, comfort).

[1191] Step 7:

[1192] The server generates an SQL query to retrieve an appropriate sorting method from the database based on the recognition result (plastic bottle) and emotion data (e.g., stress state). The server then sends this query to the database.

[1193] Step 8:

[1194] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[1195] Step 9:

[1196] The server generates instructions for the user based on the sorting method data returned from the database. At this time, the server uses natural language generation technology to adjust the instructions to take into account the user's emotional state. For example, "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic."

[1197] Step 10:

[1198] The server creates an API response containing the generated instruction and sends it to the user's device.

[1199] Step 11:

[1200] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app interface displays the message, "Recycling is easy. Remove the cap and label, and separate the bottle part as plastic."

[1201] Step 12:

[1202] Users follow the instructions provided to them to separate their garbage, making it easy for them to separate their garbage accurately.

[1203] Example 2

[1204] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1205] Conventional garbage sorting systems not only do not provide sufficient instructions for users to accurately sort waste types, but also lack the flexibility to take into account the user's emotional state, making it difficult for users to sort their waste efficiently and without stress.

[1206] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and temporarily storing images and emotional data taken or selected by the user, means for analyzing the temporarily stored images to identify the type of waste, means for analyzing the temporarily stored emotional data to identify the user's emotional state, means for acquiring a sorting method from a database based on the identified type of waste and the user's emotional state, means for using an AI inference engine to generate appropriate instructions that take into account the user's emotional state based on the acquired sorting method, and means for notifying the user terminal of the generated instructions. This allows the user to receive flexible and appropriate sorting instructions that take into account their own emotional state, allowing for stress-free and efficient sorting work.

[1207] "User" refers to a person who uses the system to take or select images of waste and receive instructions on how to sort it.

[1208] "Device" refers to a device used by a User, such as a smartphone or tablet, that is capable of taking and sending images, capturing emotional data, and receiving notifications.

[1209] "Server" refers to a computer system that receives image data and emotion data, temporarily stores them, and analyzes them.

[1210] "Image recognition module" refers to a software module that runs on a server and has the function of analyzing received image data using a machine learning model and identifying the type of waste.

[1211] An "emotion engine" refers to a deep learning model that runs on a server and has the ability to analyze received facial expressions and voice data to identify the user's emotional state.

[1212] "Database" refers to an information repository that systematically stores waste sorting methods according to type and other related information.

[1213] An "AI inference engine" refers to an algorithm or program that generates instructions for users based on the analysis results of the image recognition module and emotion engine and information from a database.

[1214] "Notification system" refers to a module that has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[1215] "Emotional Data" refers to a user's facial expressions and voice data, and means data used to analyze a user's emotional state.

[1216] A "machine learning model" refers to statistical methods and algorithms that learn patterns from data and perform image recognition and emotion analysis.

[1217] A "deep learning model" refers to an algorithm that uses artificial neuron networks to perform more advanced data analysis.

[1218] "Temporary storage" means that the image data and emotion data received by the server are temporarily stored until analysis is complete.

[1219] "Sorting method" refers to the appropriate disposal method for each type of waste, and is information obtained from the database.

[1220] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. This system makes it possible to provide flexible and appropriate instructions according to the user's emotional state.

[1221] System configuration

[1222] The system of the present invention mainly includes the following components:

[1223] 1. User device: This is a device that users use to take photos of garbage and upload them to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[1224] 2. Server: Receives and temporarily stores image data and emotion data, and performs the necessary processing for the image recognition module, emotion engine, database, and AI inference engine.

[1225] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[1226] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[1227] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[1228] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[1229] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[1230] System operation explanation

[1231] Users take photos of waste using their smartphone camera and send them to the server via the application. During the transmission, facial expressions and voice data are also captured. The server receives and temporarily stores this data.

[1232] The server then passes the temporarily stored image data to an image recognition module, which uses a machine learning model to identify the type of waste. At the same time, it also passes the emotion data to an emotion engine, which uses a deep learning model to analyze the user's emotional state.

[1233] The analysis results are compared with the database, and the server uses an AI inference engine to generate instructions for the user. These instructions are based on the analyzed emotional data and take into account the user's emotional state, such as "Recycle in a few easy steps." Finally, the generated instructions are sent to the user's device and displayed within the app.

[1234] Specific examples

[1235] For example, if a user wants to throw away a plastic bottle, the following actions occur:

[1236] 1. The user takes a photo of a plastic bottle using their smartphone camera and sends the photo and emotional data to the server via the app.

[1237] 2. The server receives the photo and emotion data and temporarily stores them.

[1238] 3. The server sends the image data to the image recognition module, which identifies it as a plastic bottle.

[1239] 4. The server sends the emotion data to the emotion engine, which determines that the user is stressed.

[1240] 5. The server searches the database to find out how to separate plastic bottles. The database returns the information that says, "Remove the cap and label, and separate the bottle as plastic."

[1241] 6. The AI ​​inference engine generates the following instruction: "Recycle in a few simple steps. Remove the cap and label, and separate the bottle part as plastic."

[1242] 7. The server sends this instruction to the user's device, which displays the instruction within the app, allowing the user to view the notification and take appropriate action.

[1243] Prompt Sentence Examples

[1244] Please explain the process by which a user takes a photo of the garbage they want to dispose of, sends emotional data to the server, and receives sorting instructions based on their emotional state.

[1245] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1246] Step 1:

[1247] The user uploads a photo.

[1248] How it works: The user takes a photo of the waste using their smartphone camera, which also captures facial expressions and voice data. The user then presses the upload button within the app, which sends this data to the server.

[1249] Input: Garbage images, facial expression data, and audio data.

[1250] Output: Image files and emotion data sent to the server.

[1251] Step 2:

[1252] The server receives the photos and temporarily stores them.

[1253] Specific operation: The server receives the image file and emotion data sent from the device and temporarily stores them. Upon receiving the data, it checks for consistency and whether there are any missing data.

[1254] Input: Image file and emotion data sent from the user device.

[1255] Output: Image files and emotion data saved in temporary storage.

[1256] Step 3:

[1257] The image recognition module analyzes the photo.

[1258] How it works: The server passes the image data stored in the temporary storage area to the image recognition module, which uses a pre-trained machine learning model to analyze the image and identify the type of waste.

[1259] Input: Image data retrieved from temporary storage.

[1260] Output: Identified waste type (e.g., PET bottles, paper, plastic).

[1261] Step 4:

[1262] The emotion engine analyzes emotions.

[1263] How it works: The server passes the emotional data (facial expressions and voice data) stored in the temporary storage area to the emotion engine, which then uses a deep learning model to analyze the data and identify the user's emotional state.

[1264] Input: Emotion data retrieved from temporary storage.

[1265] Output: Identified user emotional state (e.g., stressed, relaxed, angry).

[1266] Step 5:

[1267] The server checks the database.

[1268] Specific operation: The server searches the database based on the type of waste obtained from the image recognition module. The database stores appropriate sorting methods and related information for each type of waste. The server retrieves the corresponding sorting method.

[1269] Input: Identified waste type.

[1270] Output: The corresponding sorting method (e.g. remove cap and label, and sort the bottle as plastic).

[1271] Step 6:

[1272] An AI inference engine generates instructions.

[1273] Specific operation: The server uses an AI inference engine to generate instructions for the user based on the image recognition results, the emotion engine results, and the classification method retrieved from the database. These instructions take into account the user's emotional state.

[1274] Input: Image recognition results, emotion engine results, classification methods from the database.

[1275] Output: Instructions for the user (e.g., "Recycle in a few easy steps. Remove the cap and label and separate the bottle for plastic.").

[1276] Step 7:

[1277] Notify the user.

[1278] Specific operation: The server sends the generated instruction text to the user's device, which then displays the instruction within the app and suggests appropriate sorting methods to the user.

[1279] Input: Instructions sent by the server.

[1280] Output: Instructions displayed on the user's terminal.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1283] In recent years, with growing interest in environmental issues, accurate waste separation is becoming increasingly important. However, many users do not properly understand the types of waste and how to separate them, leading to incorrect separation. Furthermore, users' emotional state can sometimes affect their waste separation behavior. This situation hinders effective waste management. The present invention aims to solve these problems and provide a system that supports users in easily separating waste appropriately.

[1284] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken or selected by the user, means for acquiring and analyzing the user's emotional data, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for acquiring a sorting method from a database based on the identified type of waste, and means for providing the user with appropriate instructions based on the acquired sorting method and the user's emotional state. This makes it possible to provide flexible and appropriate sorting instructions based on the type of waste and the user's emotional state.

[1285] A "user" is an individual who uses the system to take or select images of waste and learn how to separate it.

[1286] "Waste" means items that are no longer needed or useful and that need to be disposed of in an environmentally friendly manner.

[1287] "Means for uploading images" refers to a function or device for transmitting image data taken or selected by a user to a server.

[1288] "Emotion data" is information that indicates the emotional state of the user, and is acquired based on facial expressions and voice data.

[1289] The "means for receiving and temporarily storing images" refers to a function or device that temporarily stores uploaded image data.

[1290] The "means for analyzing images and identifying the type of waste" refers to a function or device for identifying the type of waste from temporarily stored image data using a machine learning model.

[1291] The "database" is an information repository that systematically stores waste sorting methods according to type and other related information.

[1292] The "means for obtaining a sorting method" is a function or device that searches for and obtains an appropriate sorting method from a database based on the type of waste identified.

[1293] The "means for providing appropriate instructions" is a function or device that generates and conveys easy-to-understand instructions to the user based on the acquired discernment method and the user's emotional state.

[1294] This invention relates to an application that helps users properly separate waste generated during food delivery. Specifically, it provides a system that analyzes images of waste and provides guidance on the appropriate separation method based on the user's emotional state.

[1295] System configuration

[1296] The system of the present invention includes the following components:

[1297] 1. User device: This is the device on which the user takes a picture of the trash and acquires emotion data. This device can be a smartphone or tablet.

[1298] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[1299] 3. Image Recognition Module: This module analyzes uploaded images and identifies the type of waste. This module uses machine learning models to perform image analysis.

[1300] 4. Emotion Engine: This engine has the ability to analyze the user's emotional state. It uses a deep learning model to perform emotion analysis.

[1301] 5. Database: Contains information on how to separate waste by type and related information.

[1302] 6. AI Inference Engine: Based on the results of the image recognition module and emotion engine, it generates specific and emotion-sensitive instructions for the user.

[1303] 7. Notification system: The generated instructions are sent to the user's terminal, and the user is informed of how to sort the waste.

[1304] System Operation

[1305] 1. User device operation:

[1306] Users use their smartphone camera to take pictures of the waste and capture facial expressions and voice as emotional data.

[1307] The data is then uploaded to the server through the application.

[1308] 2. Receiving and temporary storage on the server:

[1309] The server temporarily stores the uploaded image data and emotion data.

[1310] 3. Image Recognition Module Processing:

[1311] The temporarily stored image data is analyzed by an image recognition module to identify the type of waste.

[1312] 4. Emotion Engine Processing:

[1313] The emotion data is analyzed by an emotion engine to determine the user's emotional state.

[1314] 5. Database verification:

[1315] The server searches a database based on the identified type of waste and obtains the appropriate sorting method.

[1316] 6. Generate instructions:

[1317] The AI ​​inference engine generates easy-to-understand instructions based on the type of waste and the user's emotional state.

[1318] 7. Notice to Users:

[1319] The server then sends the generated instructions to the user's device and displays them within the app, allowing the user to learn how to properly separate their trash.

[1320] Specific examples

[1321] For example, consider the case where a user is about to dispose of a plastic bottle. The user takes a picture of the bottle, captures facial expressions and voice as emotional data, and uploads it to the server. The server uses an image recognition module to recognize the bottle, and an emotion engine to identify that the user is feeling stressed. The database is then retrieved to provide instructions such as "Remove the cap and label, and separate the bottle part as plastic," and the AI ​​inference engine generates instructions such as "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic." These instructions are displayed on the user's device, allowing the user to properly separate their waste.

[1322] Prompt Sentence Examples

[1323] 1. "waste_image.jpg" is an image of a plastic bottle taken by a user.

[1324] 2. User's face photo "user_image.jpg".

[1325] 3. User's audio file "user_audio.wav".

[1326] Using this data, the system suggests appropriate ways to dispose of plastic bottles, taking into account the user's emotional state.

[1327] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1328] Step 1:

[1329] Users take images of waste using their smartphone cameras and capture facial expressions and voice as emotional data. Users then upload these data to a server via an application. The input is waste images, facial expressions, and voice data, and the output is the uploaded data.

[1330] Step 2:

[1331] The server receives and temporarily stores image data and emotion data uploaded by the user. The input of this step is the uploaded image data and emotion data, and the output is the temporarily stored data.

[1332] Step 3:

[1333] The server passes the temporarily stored image data to an image recognition module, which uses machine learning models to analyze these images and identify the waste type. The input is the temporarily stored image data, and the output is the identified waste type.

[1334] Step 4:

[1335] The server passes the stored emotional data to the emotion engine, which uses a deep learning model to analyze the data and identify the user's emotional state. The input is the stored emotional data, and the output is the user's emotional state.

[1336] Step 5:

[1337] The server searches the database based on the identified waste type to obtain the corresponding sorting method. The input is the identified waste type, and the output is the corresponding sorting method.

[1338] Step 6:

[1339] The server uses an AI inference engine to generate specific and emotion-sensitive instructions for users based on the image recognition results, the emotion engine results, and the sorting method from the database. The input is the type of waste, the user's emotional state, and the sorting method, and the output is the generated instructions.

[1340] Step 7:

[1341] The server sends the generated instructions to the user's device, which then displays the received instructions to the user within the app. The input is the generated instructions, and the output is the sorting instructions displayed to the user.

[1342] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1343] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1344] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1345] [Fourth embodiment]

[1346] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1347] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1348] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1349] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1350] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1351] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1352] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1353] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1354] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1355] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1356] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1357] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1358] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1359] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate method for sorting waste.

[1360] System configuration

[1361] The system of the present invention mainly includes the following components:

[1362] 1. User device: This is the device that users use to take photos of trash and upload them to the server via the application. This can be a smartphone or tablet.

[1363] 2. Server: This is the central processing unit that receives and temporarily stores image data, and also performs the necessary processing for the image recognition module and database.

[1364] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[1365] 4. Database: An information repository that stores the methods for separating waste according to type, providing information on how to separate identified waste.

[1366] 5. AI inference engine: Generates specific instructions for the user based on the results obtained from the image recognition module and information retrieved from the database.

[1367] 6. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[1368] Program processing

[1369] 1. User uploads a photo

[1370] Users use a smartphone app to take or select an image of the trash.

[1371] The terminal transmits this image file to the server.

[1372] 2. The server receives the photo

[1373] The server receives the image file sent from the terminal and temporarily stores it.

[1374] 3. The image recognition module analyzes the photo

[1375] The server passes the temporarily stored image to the image recognition module.

[1376] An image recognition module analyzes the image and identifies the type of waste.

[1377] 4. Check against the database

[1378] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[1379] 5. AI generates instructions

[1380] Based on the classification method obtained, the server uses natural language generation technology to generate appropriate instructions for the user.

[1381] 6. Notify users

[1382] The server transmits the generated instructions to the user terminal.

[1383] The device will then display the received instructions to the user within the app.

[1384] Specific examples

[1385] Scenario: A user wants to dispose of a plastic bottle.

[1386] 1. User uploads a photo

[1387] Users take a photo of the plastic bottle using their smartphone camera and send it to the server via the app.

[1388] 2. The server receives the photo

[1389] The server receives this photo and temporarily stores it.

[1390] 3. The image recognition module analyzes the photo

[1391] The server sends the photo to an image recognition module for analysis.

[1392] The module analyzes the photo and identifies it as a plastic bottle.

[1393] 4. Check against the database

[1394] The server searches the database based on the results identified as a plastic bottle.

[1395] The database sends the result to the server: "Remove the cap and label and separate the bottle part as plastic."

[1396] 5. AI generates instructions

[1397] The AI ​​inference engine uses the results to generate instructions for the user.

[1398] The generated instructions are: "Remove the cap and label and separate the bottle as plastic."

[1399] 6. Notify users

[1400] The server sends this instruction to the user terminal.

[1401] The device will display these instructions to the user within the app, so they know how to properly separate their waste.

[1402] The processing flow will be explained below.

[1403] Step 1:

[1404] A user launches the smartphone app and taps the "Upload Photo" button. The user then takes a photo of the trash or selects one from an existing photo gallery.

[1405] Step 2:

[1406] The device acquires the selected or taken photo data and generates an API request to send it to the server. The device then sends this request to the server via the Internet.

[1407] Step 3:

[1408] The server receives the API request, analyzes the uploaded photo data, and stores the photo data in temporary storage.

[1409] Step 4:

[1410] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[1411] Step 5:

[1412] The server receives the analysis results from the image recognition module, which identifies the trash in the photo as a plastic bottle.

[1413] Step 6:

[1414] Based on the recognition results, the server generates an SQL query to retrieve the appropriate classification method from the database, and sends this query to the database.

[1415] Step 7:

[1416] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[1417] Step 8:

[1418] The server generates instructions for the user based on the sorting method data returned from the database, using natural language generation technology.

[1419] Step 9:

[1420] The server creates an API response containing the generated instruction text and sends it to the device.

[1421] Step 10:

[1422] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app displays the message "Please remove the cap and label and separate the bottle part as plastic." on the app interface.

[1423] Step 11:

[1424] The user follows the instructions provided to separate the garbage, making it easy for the user to separate the garbage accurately.

[1425] Example 1

[1426] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] Current waste sorting systems have problems, such as users finding it difficult to understand how to sort their waste, making it difficult to dispose of waste efficiently. Furthermore, manually identifying the type of waste and identifying the appropriate sorting method based on that is time-consuming and inconvenient for users.

[1428] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1429] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for generating appropriate instructions for the user based on the retrieved sorting method, and means for notifying the generated instructions to a user terminal. This allows the user to easily learn the appropriate method for sorting waste, enabling efficient and accurate waste disposal.

[1430] "User" means an individual or organization that uses the system to separate waste.

[1431] "Image" refers to photographic data of waste taken or selected by the user.

[1432] "Uploading means" refers to a function or device for transmitting image data from a user terminal to a server.

[1433] "Means for receiving and temporarily storing" refers to a function or device that allows the server to receive image data sent from a user terminal and temporarily store it.

[1434] "Means for analyzing" refers to a system or function that allows the server to analyze temporarily stored images and automatically identify the type of waste.

[1435] A "machine learning model" is an algorithm or statistical model that the server uses to identify types of waste, and has knowledge learned from image data.

[1436] "Database" refers to an information repository that stores sorting methods for different types of waste.

[1437] "Means for obtaining a sorting method" refers to the function of the server to search and obtain an appropriate sorting method from the database based on the type of waste identified.

[1438] "Natural language generation technology" refers to technology that generates easy-to-understand instructions for users based on the classification method acquired by the server.

[1439] "Means for generating" refers to the function of generating specific instructions for the user based on the sorting method obtained by the server.

[1440] "Means for notifying" refers to the function of sending instructions generated by the server to the user terminal and notifying the user.

[1441] "User terminal" refers to the device that a user uses to take or select images of waste and upload them to the server. This could be a smartphone or tablet.

[1442] This invention is a system that allows a user to upload images of waste taken or selected by the user to a server, analyzes the images, and provides the user with an appropriate waste sorting method. Specific embodiments for implementing this system will be described below.

[1443] The system mainly includes the following components:

[1444] 1. User Device:

[1445] This is a device that allows users to take photos of waste and upload them to a server via an application. Specifically, it is a smartphone or tablet. The user device has a camera function and an internet connection, so the captured images can be quickly sent to the server.

[1446] 2. Server:

[1447] The server receives and temporarily stores image data. It is also a central processing unit that performs the necessary processing for the image recognition module and database. The server runs on a web server and efficiently manages the reception and processing of data.

[1448] 3. Image Recognition Module:

[1449] This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models (e.g., TensorFlow or PyTorch) to perform image analysis. The image recognition module uses a trained model to detect the characteristics of the waste and classify it into a specific category.

[1450] 4. Database:

[1451] The database is an information repository that stores waste separation methods according to the type of waste. It provides information on the separation method for the identified waste. The database contains the appropriate separation method for each type of waste and has an index for searching them.

[1452] 5. AI inference engine:

[1453] The AI ​​inference engine generates specific instructions for the user based on the results from the image recognition module and information retrieved from the database, using natural language generation techniques (e.g., generative AI models such as GPT-3) to generate instructions in a form that is easy for humans to understand.

[1454] 6. Notification System:

[1455] The notification system sends instructions generated by the server to the user's device and provides instructions on how to sort the waste. The notification system uses in-app notifications and push notifications to quickly and reliably inform users.

[1456] Specific examples

[1457] Scenario: A user wants to dispose of a plastic bottle.

[1458] 1. User uploads a photo:

[1459] Users take a photo of the plastic bottle with their smartphone camera and send it to the server via the app.

[1460] 2. The server receives and temporarily stores the photos:

[1461] The server receives this photo and temporarily stores it in the "uploads" folder.

[1462] 3. The image recognition module analyzes the photo:

[1463] The server sends the photo to an image recognition module, which uses TensorFlow to identify it as a plastic bottle.

[1464] 4. The server checks the database and obtains the classification method:

[1465] The server queries the database and obtains the sorting method: "Remove the cap and label, and separate the bottle part as plastic."

[1466] 5. The AI ​​inference engine generates instructions:

[1467] Based on the results obtained, the AI ​​inference engine generates instructions such as, "Remove the cap and label and separate the bottle part as plastic."

[1468] 6. The server notifies the user:

[1469] The server sends the generated instructions to the user's device, and the user receives the instructions within the app.

[1470] Example prompt statement

[1471] Input: A user-taken image of a plastic bottle

[1472] Output: Instructions based on image analysis results: "Remove the cap and label and separate the bottle as plastic."

[1473] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1474] Step 1:

[1475] User uploads a photo

[1476] Input: Images of waste taken by the user with a smartphone camera

[1477] Specific operation: The user launches the app on their smartphone and takes a photo of the waste or selects an existing image. The taken or selected image is displayed in the app. The user presses the "Upload" button, and the image file is sent to the server.

[1478] Output: Image data sent from the device to the server

[1479] Step 2:

[1480] The server receives the photo and temporarily stores it.

[1481] Input: Image data sent from the device

[1482] Specific operation: The server receives the image data sent from the device and temporarily stores it in memory or storage. Once the server has received the image data, it registers the image metadata in a database. This metadata includes the image storage location, the date and time of transmission, etc.

[1483] Output: Temporarily saved image data and its metadata

[1484] Step 3:

[1485] Image recognition module analyzes the photo

[1486] Input: Temporarily saved image data

[1487] Specific operation: The server sends the temporarily stored image data to the image recognition module. The image recognition module analyzes the image using a machine learning model (e.g., TensorFlow or PyTorch). The model extracts waste characteristics from the image and identifies the type of waste. An analysis result (e.g., "plastic bottle") is generated.

[1488] Output: Analysis results identifying the type of waste

[1489] Step 4:

[1490] The server checks the database and obtains the sorting method.

[1491] Input: Analysis results from the image recognition module (type of waste)

[1492] Specific operation: The server searches a database based on the analysis results obtained from the image recognition module. The database stores sorting methods for each type of waste. The server retrieves the sorting method corresponding to the type of waste from the database.

[1493] Output: The obtained sorting method (e.g., "Remove the cap and label, and sort the bottle part as plastic")

[1494] Step 5:

[1495] AI inference engine generates instructions

[1496] Input: Obtained sorting method

[1497] Specific operation: The AI ​​inference engine uses natural language generation technology (for example, a generative AI model such as GPT-3) to generate instructions for the user based on the acquired sorting method. The generated instructions are expressed in a form that is easy for humans to understand. For example, the generated instruction might be, "Remove the cap and label, and separate the bottle part as plastic."

[1498] Output: Generated directives

[1499] Step 6:

[1500] The server notifies the user

[1501] Input: Generated instructions

[1502] Specific operation: The server sends the generated instruction text to the user's device. Using the notification system, the in-app notification function or push notification function is used to quickly and reliably convey information to the user.

[1503] Output: Instructions displayed on the user's terminal

[1504] This allows users to easily understand and practice proper waste separation methods.

[1505] (Application example 1)

[1506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1507] Existing systems that provide users with proper waste sorting methods often use fixed devices and lack real-time operability. This creates a problem when users want to know how to sort their waste immediately, especially in physical stores and public places. Furthermore, there are only a limited number of systems that support waste sorting rules that vary from region to region, making it difficult for users to accurately sort their waste in accordance with their local rules.

[1508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1509] In this invention, the server includes means for uploading images taken or selected by the user, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for retrieving a sorting method from a database based on the identified type of waste, means for providing appropriate instructions to the user based on the retrieved sorting method, means for displaying the sorting method on a device held by the user in real time, and means for customizing the sorting method based on the sorting rules of the area to which the user belongs. This allows the user to obtain an appropriate sorting method in real time, enabling accurate waste sorting suited to their area.

[1510] "Means for uploading images taken or selected by the user" refers to a function that allows a user to capture images of waste using a device such as a smartphone or smart glasses and send the images to the server.

[1511] "Means for receiving and temporarily storing uploaded images" refers to a function by which the server receives image data sent by a user and temporarily stores it for analysis.

[1512] "Means for analyzing temporarily stored images and identifying the type of waste" refers to a function that analyzes temporarily stored image data using machine learning models and image recognition algorithms to determine the type of waste indicated by the image.

[1513] "Means for retrieving sorting methods from a database based on the type of waste identified" is a function that searches for and retrieves appropriate sorting methods from a database based on the type of waste identified through analysis.

[1514] "Means for providing appropriate instructions to the user based on the obtained sorting method" is a function for generating and providing specific instructions to the user based on the sorting method obtained from the database.

[1515] "Means for displaying the sorting method in real time on a device held by the user" refers to a function for instantly displaying the acquired sorting method on a user device such as a smartphone or smart glasses.

[1516] "Means for customizing sorting methods based on the sorting rules of the area to which the user belongs" is a function for customizing appropriate sorting methods in accordance with the specific sorting rules of the area in which the user resides and providing that information.

[1517] This invention is a system that analyzes images of waste taken or selected by a user and provides sorting methods in real time. The system includes a user terminal, a server, an image recognition module, a database, an AI inference engine, and a notification system.

[1518] The system functions as follows:

[1519] User Device

[1520] The user device refers to a smartphone or smart glasses, and is responsible for allowing users to take images of waste and upload them to the server. When users take images of waste using the device's camera, the image data is saved on the device.

[1521] server

[1522] The server receives image data sent from the user terminal and temporarily stores it. It functions as a central processing unit for the image recognition module, database, AI inference engine, etc. After receiving the image data, the server sends the image data to the image recognition module.

[1523] Image Recognition Module

[1524] The image recognition module uses machine learning models to analyze image data submitted by users and identify the type of waste. This module uses generative AI models that have been trained on a wide range of waste data to perform highly accurate image analysis.

[1525] Database

[1526] The database stores the correct method for separating waste based on the analysis results. The server searches the database based on the results obtained from the image recognition module and obtains information on the corresponding separation method.

[1527] AI inference engine

[1528] Based on the classification methods retrieved from the database, the AI ​​inference engine uses natural language generation technology to generate specific instructions for the user, which are presented in a format that is easy for the user to understand.

[1529] Notification System

[1530] The notification system sends instructions generated by the server to the user's device, where the instructions are displayed in real time and users can receive guidance on how to properly separate their waste.

[1531] Specific examples

[1532] Here is a specific example using a smartphone.

[1533] When a user disposes of a plastic bottle purchased at a supermarket, they take a photo of the bottle with their smartphone camera. The photo is then sent to a server via the app. The server receives the photo and temporarily stores it. Next, an image recognition module analyzes the photo and identifies it as a plastic bottle. Based on this identification, the server searches a database for the corresponding sorting method, and uses an AI inference engine to generate instructions such as "Remove the cap and label, and separate the bottle portion as plastic." These instructions are sent to the smartphone, allowing the user to learn the correct sorting method in real time.

[1534] Prompt Sentence Examples

[1535] An example prompt for uploading an image of a plastic bottle is as follows:

[1536] "I'll upload a picture of a plastic bottle. How do I dispose of this plastic bottle?"

[1537] This allows users to easily obtain accurate sorting instructions in accordance with local sorting rules.

[1538] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1539] Step 1:

[1540] Users take pictures of the waste using their smartphone camera and upload them to a server via the app.

[1541] Input: Image of waste taken with a smartphone

[1542] Output: Image data sent to the server

[1543] Specific operation: The user launches the camera app on their smartphone, takes a photo of the waste, and then presses the upload button to send the image to the server.

[1544] Step 2:

[1545] The server receives the image data sent by the user and temporarily stores it.

[1546] Input: Image data sent by the user

[1547] Output: Image data temporarily stored on the server

[1548] Specific operation: The server receives the image data, temporarily stores it in a database or file system, and records a receipt confirmation in a log.

[1549] Step 3:

[1550] The server passes the temporarily stored image data to an image recognition module to identify the type of waste.

[1551] Input: Temporarily saved image data

[1552] Output: Analysis results (type of waste)

[1553] How it works: The server sends image data to the image recognition module, which analyzes the image using a generative AI model. The image recognition module identifies the type of waste and returns that information to the server.

[1554] Step 4:

[1555] The server searches the database based on the type of waste obtained from the image recognition module to obtain the corresponding sorting method.

[1556] Input: Analysis results (type of waste)

[1557] Output: Sorting method

[1558] Specific operation: The server searches the database to obtain the classification method corresponding to the analysis result, and then extracts the appropriate classification method by referring to the rules and instructions in the database.

[1559] Step 5:

[1560] Based on the sorting method obtained by the server, the AI ​​inference engine generates specific instructions for the user in natural language.

[1561] Input: Sorting method

[1562] Output: Separation instructions for the user

[1563] How it works: The AI ​​inference engine analyzes the sorting method and uses a generative AI model to generate instructions in natural language that are easy for the user to understand, such as "Remove the cap and label, and separate the bottle part as plastic."

[1564] Step 6:

[1565] The server sends the generated instructions to the user's smartphone and displays them in real time.

[1566] Input: Separation instructions for the user

[1567] Output: Instructions displayed on the smartphone

[1568] Specific operation: The server sends the generated instruction text to the user's smartphone via the notification system. The smartphone app receives the instruction text and displays it on the screen in real time.

[1569] This processing flow allows users to learn in real time how to properly separate the waste they have photographed, enabling them to dispose of it correctly in accordance with their local rules.

[1570] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1571] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. By using the emotion engine, it becomes possible to provide more flexible and appropriate instructions according to the user's emotional state.

[1572] System configuration

[1573] The system of the present invention mainly includes the following components:

[1574] 1. User device: A device where a user takes a photo of the garbage and uploads it to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[1575] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[1576] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[1577] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[1578] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[1579] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[1580] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[1581] Program processing

[1582] 1. User uploads a photo

[1583] Using a smartphone app, users take or select an image of the trash, which also captures facial expressions and audio data.

[1584] The terminal transmits the image file and emotion data to the server.

[1585] 2. The server receives the photo

[1586] The server receives the image file and emotion data sent from the terminal and temporarily stores them.

[1587] 3. The image recognition module analyzes the photo

[1588] The server passes the temporarily stored image data to the image recognition module.

[1589] An image recognition module analyzes the image and identifies the type of waste.

[1590] 4. Emotion engine analyzes emotions

[1591] The server passes the temporarily stored emotional data (facial expressions and voice) to the emotion engine.

[1592] The emotion engine analyzes this data to determine the user's emotional state.

[1593] 5. Check against the database

[1594] The server searches the database based on the type of waste obtained from the image recognition module and obtains the corresponding sorting method.

[1595] 6. AI generates instructions

[1596] The server generates instructions for the user based on the image recognition results, the emotion engine results, and classification methods from the database. These instructions take into account the user's emotional state.

[1597] 7. Notify users

[1598] The server transmits the generated instruction text to the user terminal.

[1599] The device will then display the received instructions to the user within the app.

[1600] Specific examples

[1601] Scenario: A user wants to dispose of a plastic bottle.

[1602] 1. User uploads a photo

[1603] Users take a photo of the bottle using their smartphone camera and send it to the server via the app, which also captures their facial expressions and voice.

[1604] 2. The server receives the photo

[1605] The server receives the photo and emotion data and temporarily stores them.

[1606] 3. The image recognition module analyzes the photo

[1607] The server sends the photo to an image recognition module for analysis.

[1608] The module analyzes the photo and identifies it as a plastic bottle.

[1609] 4. Emotion engine analyzes emotions

[1610] The server sends the emotion data to the emotion engine for analysis.

[1611] The emotion engine identifies that the user is stressed.

[1612] 5. Check against the database

[1613] The server searches the database based on the results identified as a plastic bottle.

[1614] The database returns the data: "Remove the cap and label, and separate the bottle part as plastic."

[1615] 6. AI generates instructions

[1616] The AI ​​inference engine uses these results to generate instructions for the user. Based on the emotion engine's results, the instructions are adjusted to "Recycle in a few easy steps. Remove the cap and label and separate the bottle part as plastic."

[1617] 7. Notify users

[1618] The server sends this instruction to the user terminal.

[1619] The device will display these instructions within the app, making it easy for users to follow the proper sorting method.

[1620] The processing flow will be explained below.

[1621] Step 1:

[1622] A user launches the smartphone app and taps the "Upload Photo" button. They can then take a photo of the litter or select one from their existing photo gallery. The app also simultaneously captures the user's facial expressions and voice.

[1623] Step 2:

[1624] The device acquires the selected or taken photo data and the captured facial expression and voice data, generates an API request to send this data to the server, and sends it to the server via the Internet.

[1625] Step 3:

[1626] The server receives the API request, analyzes the uploaded photo data and emotion data, and temporarily stores this data.

[1627] Step 4:

[1628] The server then sends the temporarily stored photo data to an image recognition module, which uses machine learning models to analyze the objects in the photo and identify the type of waste.

[1629] Step 5:

[1630] The server receives the analysis results from the image recognition module and identifies the garbage in the photo as a plastic bottle.

[1631] Step 6:

[1632] The server sends the temporarily stored facial expression and voice data to the emotion engine, which analyzes this data and identifies the user's emotional state (e.g., stress, comfort).

[1633] Step 7:

[1634] The server generates an SQL query to retrieve an appropriate sorting method from the database based on the recognition result (plastic bottle) and emotion data (e.g., stress state). The server then sends this query to the database.

[1635] Step 8:

[1636] The database processes the query and returns the corresponding sorting instructions, for example, "Remove the cap and label, and sort the bottle as plastic."

[1637] Step 9:

[1638] The server generates instructions for the user based on the sorting method data returned from the database. At this time, the server uses natural language generation technology to adjust the instructions to take into account the user's emotional state. For example, "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic."

[1639] Step 10:

[1640] The server creates an API response containing the generated instruction and sends it to the user's device.

[1641] Step 11:

[1642] The device analyzes the API response received from the server and displays instructions to the user. Specifically, the app interface displays the message, "Recycling is easy. Remove the cap and label, and separate the bottle part as plastic."

[1643] Step 12:

[1644] Users follow the instructions provided to them to separate their garbage, making it easy for them to separate their garbage accurately.

[1645] Example 2

[1646] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1647] Conventional garbage sorting systems not only do not provide sufficient instructions for users to accurately sort waste types, but also lack the flexibility to take into account the user's emotional state, making it difficult for users to sort their waste efficiently and without stress.

[1648] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and temporarily storing images and emotional data taken or selected by the user, means for analyzing the temporarily stored images to identify the type of waste, means for analyzing the temporarily stored emotional data to identify the user's emotional state, means for acquiring a sorting method from a database based on the identified type of waste and the user's emotional state, means for using an AI inference engine to generate appropriate instructions that take into account the user's emotional state based on the acquired sorting method, and means for notifying the user terminal of the generated instructions. This allows the user to receive flexible and appropriate sorting instructions that take into account their own emotional state, allowing for stress-free and efficient sorting work.

[1649] "User" refers to a person who uses the system to take or select images of waste and receive instructions on how to sort it.

[1650] "Device" refers to a device used by a User, such as a smartphone or tablet, that is capable of taking and sending images, capturing emotional data, and receiving notifications.

[1651] "Server" refers to a computer system that receives image data and emotion data, temporarily stores them, and analyzes them.

[1652] "Image recognition module" refers to a software module that runs on a server and has the function of analyzing received image data using a machine learning model and identifying the type of waste.

[1653] An "emotion engine" refers to a deep learning model that runs on a server and has the ability to analyze received facial expressions and voice data to identify the user's emotional state.

[1654] "Database" refers to an information repository that systematically stores waste sorting methods according to type and other related information.

[1655] An "AI inference engine" refers to an algorithm or program that generates instructions for users based on the analysis results of the image recognition module and emotion engine and information from a database.

[1656] "Notification system" refers to a module that has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[1657] "Emotional Data" refers to a user's facial expressions and voice data, and means data used to analyze a user's emotional state.

[1658] A "machine learning model" refers to statistical methods and algorithms that learn patterns from data and perform image recognition and emotion analysis.

[1659] A "deep learning model" refers to an algorithm that uses artificial neuron networks to perform more advanced data analysis.

[1660] "Temporary storage" means that the image data and emotion data received by the server are temporarily stored until analysis is complete.

[1661] "Sorting method" refers to the appropriate disposal method for each type of waste, and is information obtained from the database.

[1662] This invention combines an emotion engine with a system that allows users to upload images of waste they have taken or selected to a server, analyzes the images, and provides users with appropriate waste sorting methods. This system makes it possible to provide flexible and appropriate instructions according to the user's emotional state.

[1663] System configuration

[1664] The system of the present invention mainly includes the following components:

[1665] 1. User device: This is a device that users use to take photos of garbage and upload them to the server via an application. This device is typically a smartphone or tablet. It also has a camera and microphone to capture the user's facial expressions and voice.

[1666] 2. Server: Receives and temporarily stores image data and emotion data, and performs the necessary processing for the image recognition module, emotion engine, database, and AI inference engine.

[1667] 3. Image Recognition Module: This module analyzes the image data sent by the server and identifies the type of waste. This module uses machine learning models to perform image analysis.

[1668] 4. Emotion Engine: This engine analyzes the user's facial expressions and voice data to identify their emotional state. This engine uses deep learning models to perform emotion analysis.

[1669] 5. Database: An information repository that systematically stores information on waste sorting methods according to type and other related information.

[1670] 6. AI Inference Engine: Based on the results obtained from the image recognition module, the emotional data from the emotion engine, and the information retrieved from the database, it generates specific and emotionally sensitive instructions for the user.

[1671] 7. Notification system: This system has the function of sending instructions generated by the server to the user's terminal and presenting the user with sorting methods.

[1672] System operation explanation

[1673] Users take photos of waste using their smartphone camera and send them to the server via the application. During the transmission, facial expressions and voice data are also captured. The server receives and temporarily stores this data.

[1674] The server then passes the temporarily stored image data to an image recognition module, which uses a machine learning model to identify the type of waste. At the same time, it also passes the emotion data to an emotion engine, which uses a deep learning model to analyze the user's emotional state.

[1675] The analysis results are compared with the database, and the server uses an AI inference engine to generate instructions for the user. These instructions are based on the analyzed emotional data and take into account the user's emotional state, such as "Recycle in a few easy steps." Finally, the generated instructions are sent to the user's device and displayed within the app.

[1676] Specific examples

[1677] For example, if a user wants to throw away a plastic bottle, the following actions occur:

[1678] 1. The user takes a photo of a plastic bottle using their smartphone camera and sends the photo and emotional data to the server via the app.

[1679] 2. The server receives the photo and emotion data and temporarily stores them.

[1680] 3. The server sends the image data to the image recognition module, which identifies it as a plastic bottle.

[1681] 4. The server sends the emotion data to the emotion engine, which determines that the user is stressed.

[1682] 5. The server searches the database to find out how to separate plastic bottles. The database returns the information that says, "Remove the cap and label, and separate the bottle as plastic."

[1683] 6. The AI ​​inference engine generates the following instruction: "Recycle in a few simple steps. Remove the cap and label, and separate the bottle part as plastic."

[1684] 7. The server sends this instruction to the user's device, which displays the instruction within the app, allowing the user to view the notification and take appropriate action.

[1685] Prompt Sentence Examples

[1686] Please explain the process by which a user takes a photo of the garbage they want to dispose of, sends emotional data to the server, and receives sorting instructions based on their emotional state.

[1687] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1688] Step 1:

[1689] The user uploads a photo.

[1690] How it works: The user takes a photo of the waste using their smartphone camera, which also captures facial expressions and voice data. The user then presses the upload button within the app, which sends this data to the server.

[1691] Input: Garbage images, facial expression data, and audio data.

[1692] Output: Image files and emotion data sent to the server.

[1693] Step 2:

[1694] The server receives the photos and temporarily stores them.

[1695] Specific operation: The server receives the image file and emotion data sent from the device and temporarily stores them. Upon receiving the data, it checks for consistency and whether there are any missing data.

[1696] Input: Image file and emotion data sent from the user device.

[1697] Output: Image files and emotion data saved in temporary storage.

[1698] Step 3:

[1699] The image recognition module analyzes the photo.

[1700] How it works: The server passes the image data stored in the temporary storage area to the image recognition module, which uses a pre-trained machine learning model to analyze the image and identify the type of waste.

[1701] Input: Image data retrieved from temporary storage.

[1702] Output: Identified waste type (e.g., PET bottles, paper, plastic).

[1703] Step 4:

[1704] The emotion engine analyzes emotions.

[1705] How it works: The server passes the emotional data (facial expressions and voice data) stored in the temporary storage area to the emotion engine, which then uses a deep learning model to analyze the data and identify the user's emotional state.

[1706] Input: Emotion data retrieved from temporary storage.

[1707] Output: Identified user emotional state (e.g., stressed, relaxed, angry).

[1708] Step 5:

[1709] The server checks the database.

[1710] Specific operation: The server searches the database based on the type of waste obtained from the image recognition module. The database stores appropriate sorting methods and related information for each type of waste. The server retrieves the corresponding sorting method.

[1711] Input: Identified waste type.

[1712] Output: The corresponding sorting method (e.g. remove cap and label, and sort the bottle as plastic).

[1713] Step 6:

[1714] An AI inference engine generates instructions.

[1715] Specific operation: The server uses an AI inference engine to generate instructions for the user based on the image recognition results, the emotion engine results, and the classification method retrieved from the database. These instructions take into account the user's emotional state.

[1716] Input: Image recognition results, emotion engine results, classification methods from the database.

[1717] Output: Instructions for the user (e.g., "Recycle in a few easy steps. Remove the cap and label and separate the bottle for plastic.").

[1718] Step 7:

[1719] Notify the user.

[1720] Specific operation: The server sends the generated instruction text to the user's device, which then displays the instruction within the app and suggests appropriate sorting methods to the user.

[1721] Input: Instructions sent by the server.

[1722] Output: Instructions displayed on the user's terminal.

[1723] (Application example 2)

[1724] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1725] In recent years, with growing interest in environmental issues, accurate waste separation is becoming increasingly important. However, many users do not properly understand the types of waste and how to separate them, leading to incorrect separation. Furthermore, users' emotional state can sometimes affect their waste separation behavior. This situation hinders effective waste management. The present invention aims to solve these problems and provide a system that supports users in easily separating waste appropriately.

[1726] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading images taken or selected by the user, means for acquiring and analyzing the user's emotional data, means for receiving and temporarily storing the uploaded images, means for analyzing the temporarily stored images and identifying the type of waste, means for acquiring a sorting method from a database based on the identified type of waste, and means for providing the user with appropriate instructions based on the acquired sorting method and the user's emotional state. This makes it possible to provide flexible and appropriate sorting instructions based on the type of waste and the user's emotional state.

[1727] A "user" is an individual who uses the system to take or select images of waste and learn how to separate it.

[1728] "Waste" means items that are no longer needed or useful and that need to be disposed of in an environmentally friendly manner.

[1729] "Means for uploading images" refers to a function or device for transmitting image data taken or selected by a user to a server.

[1730] "Emotion data" is information that indicates the emotional state of the user, and is acquired based on facial expressions and voice data.

[1731] The "means for receiving and temporarily storing images" refers to a function or device that temporarily stores uploaded image data.

[1732] The "means for analyzing images and identifying the type of waste" refers to a function or device for identifying the type of waste from temporarily stored image data using a machine learning model.

[1733] The "database" is an information repository that systematically stores waste sorting methods according to type and other related information.

[1734] The "means for obtaining a sorting method" is a function or device that searches for and obtains an appropriate sorting method from a database based on the type of waste identified.

[1735] The "means for providing appropriate instructions" is a function or device that generates and conveys easy-to-understand instructions to the user based on the acquired discernment method and the user's emotional state.

[1736] This invention relates to an application that helps users properly separate waste generated during food delivery. Specifically, it provides a system that analyzes images of waste and provides guidance on the appropriate separation method based on the user's emotional state.

[1737] System configuration

[1738] The system of the present invention includes the following components:

[1739] 1. User device: This is the device on which the user takes a picture of the trash and acquires emotion data. This device can be a smartphone or tablet.

[1740] 2. Server: This is the central processing unit that receives and temporarily stores image data and emotion data, and also performs the necessary processing for the image recognition module, emotion engine, and database.

[1741] 3. Image Recognition Module: This module analyzes uploaded images and identifies the type of waste. This module uses machine learning models to perform image analysis.

[1742] 4. Emotion Engine: This engine has the ability to analyze the user's emotional state. It uses a deep learning model to perform emotion analysis.

[1743] 5. Database: Contains information on how to separate waste by type and related information.

[1744] 6. AI Inference Engine: Based on the results of the image recognition module and emotion engine, it generates specific and emotion-sensitive instructions for the user.

[1745] 7. Notification system: The generated instructions are sent to the user's terminal, and the user is informed of how to sort the waste.

[1746] System Operation

[1747] 1. User device operation:

[1748] Users use their smartphone camera to take pictures of the waste and capture facial expressions and voice as emotional data.

[1749] The data is then uploaded to the server through the application.

[1750] 2. Receiving and temporary storage on the server:

[1751] The server temporarily stores the uploaded image data and emotion data.

[1752] 3. Image Recognition Module Processing:

[1753] The temporarily stored image data is analyzed by an image recognition module to identify the type of waste.

[1754] 4. Emotion Engine Processing:

[1755] The emotion data is analyzed by an emotion engine to determine the user's emotional state.

[1756] 5. Database verification:

[1757] The server searches a database based on the identified type of waste and obtains the appropriate sorting method.

[1758] 6. Generate instructions:

[1759] The AI ​​inference engine generates easy-to-understand instructions based on the type of waste and the user's emotional state.

[1760] 7. Notice to Users:

[1761] The server then sends the generated instructions to the user's device and displays them within the app, allowing the user to learn how to properly separate their trash.

[1762] Specific examples

[1763] For example, consider the case where a user is about to dispose of a plastic bottle. The user takes a picture of the bottle, captures facial expressions and voice as emotional data, and uploads it to the server. The server uses an image recognition module to recognize the bottle, and an emotion engine to identify that the user is feeling stressed. The database is then retrieved to provide instructions such as "Remove the cap and label, and separate the bottle part as plastic," and the AI ​​inference engine generates instructions such as "Recycling can be done in simple steps. Remove the cap and label, and separate the bottle part as plastic." These instructions are displayed on the user's device, allowing the user to properly separate their waste.

[1764] Prompt Sentence Examples

[1765] 1. "waste_image.jpg" is an image of a plastic bottle taken by a user.

[1766] 2. User's face photo "user_image.jpg".

[1767] 3. User's audio file "user_audio.wav".

[1768] Using this data, the system suggests appropriate ways to dispose of plastic bottles, taking into account the user's emotional state.

[1769] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1770] Step 1:

[1771] Users take images of waste using their smartphone cameras and capture facial expressions and voice as emotional data. Users then upload these data to a server via an application. The input is waste images, facial expressions, and voice data, and the output is the uploaded data.

[1772] Step 2:

[1773] The server receives and temporarily stores image data and emotion data uploaded by the user. The input of this step is the uploaded image data and emotion data, and the output is the temporarily stored data.

[1774] Step 3:

[1775] The server passes the temporarily stored image data to an image recognition module, which uses machine learning models to analyze these images and identify the waste type. The input is the temporarily stored image data, and the output is the identified waste type.

[1776] Step 4:

[1777] The server passes the stored emotional data to the emotion engine, which uses a deep learning model to analyze the data and identify the user's emotional state. The input is the stored emotional data, and the output is the user's emotional state.

[1778] Step 5:

[1779] The server searches the database based on the identified waste type to obtain the corresponding sorting method. The input is the identified waste type, and the output is the corresponding sorting method.

[1780] Step 6:

[1781] The server uses an AI inference engine to generate specific and emotion-sensitive instructions for users based on the image recognition results, the emotion engine results, and the sorting method from the database. The input is the type of waste, the user's emotional state, and the sorting method, and the output is the generated instructions.

[1782] Step 7:

[1783] The server sends the generated instructions to the user's device, which then displays the received instructions to the user within the app. The input is the generated instructions, and the output is the sorting instructions displayed to the user.

[1784] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1785] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1786] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1787] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1788] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1789] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1790] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1791] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1792] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1793] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1794] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1795] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1796] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1797] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1798] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1799] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1800] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1801] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1802] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1803] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1804] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1805] The following is further disclosed regarding the above embodiment.

[1806] (Claim 1)

[1807] A means for users to upload images they have taken or selected;

[1808] means for receiving and temporarily storing uploaded images;

[1809] A means for analyzing the temporarily stored images and identifying the type of waste;

[1810] A means for retrieving a separation method from a database based on the identified waste type;

[1811] a means for providing appropriate instructions to the user based on the obtained separation method;

[1812] A system including:

[1813] (Claim 2)

[1814] 10. The system of claim 1, wherein the system uses a machine learning model to identify the type of waste.

[1815] (Claim 3)

[1816] 10. The system of claim 1, wherein natural language generation techniques are used to generate appropriate instructions to provide to the user.

[1817] "Example 1"

[1818] (Claim 1)

[1819] A means for users to upload images they have taken or selected;

[1820] means for receiving and temporarily storing uploaded images;

[1821] A means for analyzing the temporarily stored images and identifying the type of waste;

[1822] A means for retrieving a separation method from a database based on the identified waste type;

[1823] means for generating appropriate instructions to a user based on the obtained classification method;

[1824] means for notifying a user terminal of the generated instructions;

[1825] A system including:

[1826] (Claim 2)

[1827] 10. The system of claim 1, wherein the system uses a machine learning model to identify the type of waste.

[1828] (Claim 3)

[1829] 10. The system of claim 1, wherein natural language generation techniques are used to generate appropriate instructions to provide to the user.

[1830] "Application Example 1"

[1831] (Claim 1)

[1832] A means for users to upload images they have taken or selected;

[1833] means for receiving and temporarily storing uploaded images;

[1834] A means for analyzing the temporarily stored images and identifying the type of waste;

[1835] A means for retrieving a separation method from a database based on the identified waste type;

[1836] a means for providing appropriate instructions to the user based on the obtained separation method;

[1837] A means for displaying the sorting method in real time on the user's device;

[1838] A means to customize the sorting method based on the sorting rules of the area where the user resides;

[1839] A system including:

[1840] (Claim 2)

[1841] 10. The system of claim 1, wherein the system uses a machine learning model to identify the type of waste.

[1842] (Claim 3)

[1843] 10. The system of claim 1, wherein natural language generation techniques are used to generate appropriate instructions to provide to the user.

[1844] "Example 2: Combining Emotion Engines"

[1845] (Claim 1)

[1846] A means for users to upload images they have taken or selected;

[1847] means for receiving and temporarily storing the uploaded image and emotion data;

[1848] A means for analyzing the temporarily stored images and identifying the type of waste;

[1849] means for analyzing the temporarily stored emotional data to identify the emotional state of the user;

[1850] a means for retrieving a sorting method from a database based on the identified waste type and the user's emotional state;

[1851] A means for generating appropriate instructions that take into consideration the user's emotional state using an AI inference engine based on the acquired classification method;

[1852] means for notifying a user terminal of the generated instructions;

[1853] A system including:

[1854] (Claim 2)

[1855] 10. The system of claim 1, wherein the system uses a machine learning model to identify the type of waste.

[1856] (Claim 3)

[1857] 10. The system of claim 1, wherein the system uses a generative AI model to generate appropriate instructions to provide to the user.

[1858] "Application example 2 when combining emotion engines"

[1859] (Claim 1)

[1860] A means for users to upload images they have taken or selected;

[1861] means for receiving and temporarily storing uploaded images;

[1862] A means for analyzing the temporarily stored images and identifying the type of waste;

[1863] A means for acquiring and analyzing user emotion data;

[1864] A means for retrieving a separation method from a database based on the identified waste type;

[1865] means for providing appropriate instructions to the user based on the obtained classification method and the user's emotional state;

[1866] A system including:

[1867] (Claim 2)

[1868] 10. The system of claim 1, wherein the system uses a machine learning model to identify the type of waste.

[1869] (Claim 3)

[1870] 10. The system of claim 1, wherein natural language generation techniques are used to generate appropriate instructions to provide to the user. [Explanation of symbols]

[1871] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to upload images they have taken or selected; means for receiving and temporarily storing uploaded images; A means for analyzing the temporarily stored images and identifying the type of waste; A means for retrieving a separation method from a database based on the identified waste type; a means for providing appropriate instructions to the user based on the obtained separation method; A system including:

2. The system of claim 1 , wherein the system uses a machine learning model to identify the type of waste.

3. The system of claim 1 , wherein natural language generation techniques are used to generate appropriate instructions to provide to the user.

Images