system

Abstract

Provide a system. 【Solution means】 Means for taking pictures of articles in space using acquisition means, Means for analyzing the taken pictures to identify articles, Means for observing and recording the usage status and usage frequency of articles, Means for generating a disposal proposal for articles based on the usage frequency, Means for performing procedures for automatically putting articles on the market, Means for arranging donations and recycling as disposal methods, Means for providing feedback to users and improving the system, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern households, there are numerous items that are not truly needed. These items occupy physical space and cause economic losses. Such unused items are difficult to grasp and efficiently manage and dispose of, so their utilization as household assets is insufficient. The present invention aims to maximize the asset value within a household and reduce economic losses by efficiently managing and disposing of such unnecessary items.

Means for Solving the Problems

[0005] This invention provides a system that uses acquisition means to photograph objects in a space, analyzes them, and identifies them. It has a function to observe and record the usage status and frequency of use of objects, and generates optimal decluttering suggestions based on this data. Furthermore, it automatically lists items on the market and arranges donation or recycling as needed. Through these means, it is possible to easily organize unwanted items in the home and improve the system by incorporating user feedback.

[0006] "Acquisition means" refers to devices or methods for photographing objects in a space and acquiring image data of those objects.

[0007] "Analysis" refers to the data processing procedure used to identify and record specific items by processing acquired image data.

[0008] "Items" refers to specific goods or items that exist within a household.

[0009] "Usage status" refers to the condition of when, how, and how often a particular item is used.

[0010] "Frequency of use" refers to an indicator that quantitatively shows the number of times or duration an item is used over a certain period of time.

[0011] "Disposal suggestions" refer to proposals that show users how to dispose of unwanted items, based on data on the usage status and frequency of use of those items.

[0012] "Automatic listing on the marketplace" refers to the process of automatically registering items deemed unwanted on online marketplaces in an attempt to sell them.

[0013] "Arranging donations or recycling" refers to the process of donating unwanted items to appropriate organizations or arranging for recycling companies.

[0014] "Feedback" refers to the process of obtaining feedback and evaluations from users regarding their use of the system and the suggestions they provide.

Brief Description of Drawings

[0015] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined. [[ID=I47]]

Modes for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system for supporting the efficient management and disposal of items within the home. The system includes acquisition means, analysis functions, and a process that supports decluttering through interaction with the user.

[0037] First, the terminal activates a camera installed in the space to capture images of objects. This device photographs various objects in the home at regular intervals and sends the latest information to a server. This information serves as basic data for object recognition and management.

[0038] Next, the server analyzes the received images and automatically identifies the type, location, and characteristics of the items. For example, it uses AI-based image recognition technology to identify furniture, home appliances, and personal belongings, and registers them in a database.

[0039] The server then monitors and efficiently records the usage and frequency of each item. It integrates multiple camera inputs to analyze how each item is being used. This allows it to quantify how often an item is used over a certain period and profile its usage frequency.

[0040] Furthermore, the server utilizes usage frequency data to generate optimal decluttering suggestions for the user. For example, regarding unused electronic devices, it might notify the user via their device with a specific suggestion such as, "This item hasn't been used in the last six months. Would you like to consider selling or donating it?" This notification includes information on marketplaces where the item can be sold and potential recipients for donation, supporting quick decision-making.

[0041] If the user accepts the proposal, the server accesses the online marketplace's API and automatically processes the listing of the item. If the user chooses to donate or recycle, the server can also coordinate with relevant businesses to smoothly arrange collection and delivery.

[0042] For example, if a user decides to sell an oven toaster they haven't used for six months, the server sends the necessary information to the marketplace to complete the listing and then notifies the user of the status via their device.

[0043] This invention allows for the efficient management of unused items in the home, facilitating the disposal of unwanted items and maximizing their economic value.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The device activates its AI camera at set intervals to capture images of objects in the home. The captured images are immediately processed and prepared to extract the contours and features of the objects.

[0047] Step 2:

[0048] The terminal compresses the acquired image data and sends it to the server. The server analyzes the received image data and applies an object recognition algorithm to identify individual items. At this time, the identified items are registered in the database, and the metadata corresponding to each item is updated.

[0049] Step 3:

[0050] The server observes the location and usage of each specific item, recording usage frequency data. For example, if an item is lifted or its position is changed, it is considered used, and the usage frequency is updated.

[0051] Step 4:

[0052] The server extracts items with low usage frequency at regular intervals and generates disposal suggestions for these items. The server uses a generation AI to create disposal suggestion messages based on usage frequency and market information.

[0053] Step 5:

[0054] The user receives disposal suggestions via the terminal and reviews the suggestions. The user considers the options of selling, donating, or recycling the suggested items and enters their selection via the terminal.

[0055] Step 6:

[0056] Based on the user's selection, the server automatically lists the items for sale via the online marketplace's API. If the user chooses donation or recycling, the server handles the necessary procedures with the relevant businesses or organizations.

[0057] Step 7:

[0058] The server notifies the user of the results of the disposal procedure and records that all disposals have been completed. A feedback function collects user opinions and evaluations, which are used to propose future disposals and improve the system.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In modern households, items tend to accumulate, and their management and disposal are often inefficient. As a result, limited living space is unnecessarily occupied, and their economic value is overlooked. Furthermore, an increase in unused items leads to wasted resources and increased environmental burden. This invention aims to solve these problems by supporting the efficient management of household items, analysis of usage frequency, and appropriate disposal of unwanted items.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for photographing objects in a space using acquisition means, means for analyzing the captured images and identifying the objects, and means for observing and recording the usage status and frequency of use of the objects. This enables detailed analysis of objects in the home and appropriate decluttering suggestions to the user.

[0064] "Acquisition means" refers to a device that has the function of taking an image of an item and transmitting that data to a server.

[0065] A "photography device" is a hardware device installed in a home environment to capture images of objects at specific intervals.

[0066] A "server" is a computer system that analyzes captured image data to identify items, record their usage, and generate disposal suggestions.

[0067] "Item identification" is the process of identifying and classifying each item within a captured image using image analysis technology.

[0068] "Usage status" refers to information about how and how often an item is used.

[0069] "Frequency of use" refers to data that shows how many times a particular item was used within a certain period of time.

[0070] "Disposal suggestions" refer to generating suggestions to encourage users to sell or donate items, based on data such as usage frequency.

[0071] "Donation and reuse" refers to the act of transferring unused items to individuals or organizations in need, or to utilizing them in a way that makes them usable again.

[0072] "Feedback" is the process of collecting and analyzing information based on the results of suggestions and user actions in order to improve the performance and usability of the system.

[0073] This system is designed to efficiently manage household items and assist in the disposal of unwanted items. Specifically, it involves installing network-connected cameras in each space to periodically acquire images of items. For example, in the living room, the camera takes a picture of the entire room at a set time each day, recording past changes.

[0074] The terminal controls the shooting process and sends the captured images to the server in real time. The server receives these images and performs image analysis using a generative AI model. This analysis automatically identifies objects, classifies them into specific categories, and registers them in a database. Software used includes image analysis tools such as TENSORFLOW® and OpenCV.

[0075] The server also monitors the usage status and frequency of items in real time and stores this data. Based on this usage frequency data, it generates decluttering suggestions for unused items and notifies the user of specific suggestions via their device. For example, a suggestion might be, "This mixer hasn't been used in the last six months. Would you like to consider selling or donating it?"

[0076] If a user accepts a proposal, the server automatically calls the online marketplace API to list the item for sale. If the user chooses to donate or reuse the item, the server coordinates with relevant services to smoothly arrange collection and delivery. This entire process streamlines the management and disposal of items without requiring any user intervention.

[0077] Prompt statements as concrete examples:

[0078] "List items that haven't been used in the past six months and suggest decluttering options to users. Example: This guitar hasn't been used for six months, so please suggest where to sell or donate it."

[0079] This system provides a practical solution for effectively utilizing space within the home and maximizing its economic value.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The terminal controls a camera installed in the home and sets a schedule for periodically photographing items in each room. Inputs are the current date and time, and a pre-set shooting schedule. Output is image data of the photographed items, which is transmitted to the server in real time.

[0083] Step 2:

[0084] The server analyzes the received images using AI-based image recognition software. The input is image data sent from the terminal. The server utilizes a generative AI model to identify individual items through image analysis and classify them based on their characteristics. The output obtained from this process is category information for the items.

[0085] Step 3:

[0086] The server registers the image analysis results in a database and observes the usage status and frequency of items. The input consists of item category information obtained from image analysis and past usage data. Based on this information, the server calculates the usage frequency of each item and records it as usage history data. This output data is later used to generate decluttering suggestions.

[0087] Step 4:

[0088] The server analyzes usage frequency data for items and generates decluttering suggestions for items deemed unnecessary. The input is usage and frequency data. The server uses a generative AI model to refine the suggestions presented to the user. The output is a specific suggestion statement, such as, "This item hasn't been used in the last six months. Would you consider selling or donating it?"

[0089] Step 5:

[0090] The terminal receives a proposal notification from the server and displays it to the user. The input is the proposal text sent from the server. The user reviews the proposal and makes a decision from the options of selling, donating, or recycling. The output is the option selected by the user.

[0091] Step 6:

[0092] The server performs the necessary procedures based on the user's selection. The input is the user's selection information. The server calls the marketplace API to either automatically list the item or contact the charity to arrange delivery. After this procedure is complete, the status is notified to the terminal, and the user is provided with final feedback. The output is the result of the procedures performed.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] In modern households, the number of items continues to increase, making management and proper disposal difficult. In particular, the process of regularly reviewing belongings and efficiently disposing of unnecessary items is time-consuming and laborious, often leading to it being postponed in many households. Furthermore, environmental considerations are necessary when disposing of items; options such as reuse and donation should be considered, rather than simply throwing things away. A system is needed to address these challenges and support the efficient management and disposal of household items.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes means for an autonomous mobile device to patrol the environment and collect item data, means for generating disposal suggestions for items based on usage frequency, and means for performing procedures for automatically listing items on the market. This makes it possible to keep household items constantly up-to-date, propose effective disposal methods for unused items, and automate the procedures.

[0098] "Acquisition means" refers to devices or methods for acquiring images of objects present in a space.

[0099] "Photography equipment" refers to devices such as cameras and sensors used to capture images of objects.

[0100] "Image analysis" refers to the process of identifying objects from acquired image data and determining their characteristics and type.

[0101] "Item usage status" refers to information indicating how a particular item is being used or not being used.

[0102] "Frequency of use" refers to numerical data indicating how many times an item was used within a specific period.

[0103] A "disposal suggestion" refers to a proposal to the user on how to dispose of a specific item based on its frequency of use or other conditions.

[0104] "Automatic listing" refers to the process of automatically listing products on the marketplace.

[0105] "Donation and reuse" refers to providing items to new users or allowing them to be reused, rather than simply discarding them.

[0106] An "autonomous mobile device" refers to a robot or machine that has the function of automatically patrolling a home and collecting necessary data.

[0107] "Feedback" refers to information that reflects the data and results obtained by the system or users, and is used to help with the next steps and improvements.

[0108] The system implementing this invention is designed to effectively support the management and disposal of items within the home. The system operates by integrating multiple hardware and software components.

[0109] First, an autonomous mobile device patrols the home, acquiring images of all objects. The autonomous mobile device is equipped with high-performance cameras and sensors connected to a communication network, accurately capturing the location and characteristics of objects within the space. The captured image data is transmitted via the communication network to a server in the cloud.

[0110] The server processes the received image data using AI-based image analysis technology to identify the type, location, and characteristics of items. It uses image analysis APIs such as Google® Cloud Vision API to accurately identify items. The identified information is registered in a database, and the frequency of use of each item is recorded.

[0111] Furthermore, the server generates disposal suggestions based on usage frequency. These suggestions include recommendations for selling or donating unused items, and also provide disposal methods and market information. Notifications are sent to the user's smartphone or device in real time using a notification system like Amazon SNS. If the user accepts a suggestion, the server automatically executes the listing process by integrating with market APIs or initiates the donation process.

[0112] As a concrete example, the system detects an oven toaster that hasn't been used for six months and sends a notification to the user saying, "This oven toaster hasn't been used for six months. Would you like to consider selling or donating it?" Upon receiving this notification, the user chooses to sell it via their device, and that information is instantly sent to an online marketplace.

[0113] By utilizing generative AI models, it is possible to derive more sophisticated suggestion models and solutions from prompts such as, "Please tell me how to design an AI assistant that identifies items infrequently used in the home and suggests selling or donating them."

[0114] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0115] Step 1:

[0116] An autonomous mobile device patrols the home, acquiring images of objects with its onboard high-performance camera. This image data, including the location and characteristics of the objects, is transmitted to a server via a communication network. The input is the raw image data acquired by the camera, and the output is the image data transmitted to the server.

[0117] Step 2:

[0118] The server receives image data and performs AI-based image analysis. The input here is image data, and the type and characteristics of the objects are identified using the Google Cloud Vision API. The output is information data about the identified objects, which is registered in the database.

[0119] Step 3:

[0120] The server observes and records usage status and frequency based on the information of registered items. This data analysis takes the past usage history of each item as input and outputs data that profiles the frequency of use of each item.

[0121] Step 4:

[0122] The server generates disposal suggestions for items based on usage frequency data. The input is numerical data regarding usage frequency, and the output is a specific disposal suggestion that is communicated to the user. For example, it might generate recommendations to sell or donate an unused toaster oven.

[0123] Step 5:

[0124] The terminal notifies the user in real time of disposal suggestions from the server. The input is the content of the suggestion sent from the server, and the output is the information delivered to the user via the terminal's screen or audio.

[0125] Step 6:

[0126] If the user accepts the proposal, the server uses the marketplace API to initiate the automated listing process for the item. The input here is the user's selection information, and the output is the status of the item's listing on the marketplace.

[0127] Step 7:

[0128] The server also provides a means to arrange donations and reuse, and facilitates appropriate coordination according to the user's wishes. The input is the user's choice of disposal method, and the output is the status of the procedures with the donation or reuse recipient.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] This invention adds an emotion engine to a system designed to improve the efficiency of item management, thereby providing optimal suggestions and interfaces that take into account the user's emotional state. This system includes a process for effectively disposing of household items through acquisition means, analysis functions, emotion recognition, and user interaction.

[0131] First, the terminal uses a network-connected camera to capture images of items in the room. This image data is sent to a server to continuously monitor the condition of the items. The server uses image analysis technology to identify each item and register it in a database.

[0132] The server monitors the frequency and condition of item usage, identifying items that are not being used. If an item is used infrequently, the server generates a suggestion and prepares a notification for the user. This suggestion takes into account the item's market value and recyclability.

[0133] The key here is the emotion engine built into the server. When a user receives a suggestion using their device, the emotion engine analyzes the user's facial expressions, voice, and interaction patterns to estimate the user's current emotional state. The results of the emotion recognition are used to adjust how the suggestion is presented. For example, if a user shows anxiety about a disposal suggestion, the emotion engine detects this, and the server provides the user with additional information or options to alleviate the anxiety.

[0134] For example, if a user expresses an emotion such as "I'm wondering whether to get rid of this old camera," the emotion engine detects this through the device. The server then provides additional reassuring information to help the user make a decision, such as ways to preserve cherished photos from the camera or examples of successful sales of similar items.

[0135] Thus, by introducing an emotion engine, the present invention can go beyond mere item management and provide an experience that is attuned to the user's emotions. As a result, users can efficiently manage and dispose of items without emotional burden.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The terminal periodically activates a network-connected camera to capture images of objects in the space. The captured images are optimized to extract the features of the objects and then sent to a server.

[0139] Step 2:

[0140] The server analyzes the received image data and uses machine learning algorithms to identify items. During this process, it recognizes the unique characteristics of each item and registers them in a database. Item classifications and historical data are also associated with these items.

[0141] Step 3:

[0142] The server continuously monitors the acquired data to record the frequency and usage status of each item. It also tracks the movement and actions of items and updates usage information accordingly.

[0143] Step 4:

[0144] The server, having identified infrequently used items, automatically generates disposal suggestions. These suggestions include information such as the item's market value, recyclability, and where it can be donated. The suggestions are then prepared to be notified to the user at an appropriate time.

[0145] Step 5:

[0146] The emotion engine integrated into the server activates when the user receives a disposal suggestion. The terminal collects emotional data from the user's facial expressions and voice and sends it to the server in real time. The server uses the emotion engine to analyze the user's emotional state and make any necessary adjustments.

[0147] Step 6:

[0148] Users can review disposal suggestions on their devices and make selections based on their own emotions. The suggestions are customized based on the emotions engine's judgment, and additional information is added to alleviate the user's anxiety and hesitation.

[0149] Step 7:

[0150] After the user selects a disposal method for their items, the server accesses online marketplaces and automatically lists the items. If donation or recycling is selected, the server notifies the relevant businesses and initiates the necessary procedures. The server then verifies that the disposal is complete and provides feedback to the user.

[0151] (Example 2)

[0152] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0153] Managing belongings in modern homes and offices is becoming increasingly complex due to the sheer number and diversity of items. In particular, traditional systems are insufficient to efficiently identify unwanted items, select appropriate disposal methods, and support decision-making that considers user emotions. Therefore, there is a need for a system that not only improves the efficiency of belongings management but also reduces the psychological burden on users.

[0154] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0155] In this invention, the server includes means for photographing objects in a space using an acquisition device, means for analyzing the captured images and identifying the objects, and means for analyzing the user's emotional state and adjusting the suggested content. This enables efficient management of objects and improved accuracy of disposal suggestions, and further reduces psychological burden by providing suggestions that are sensitive to the user's emotions.

[0156] "Acquisition device" is a general term for equipment and systems used to photograph objects in a space, and mainly refers to cameras, smartphones, etc.

[0157] A "communication network" is an infrastructure for transmitting digital data, and includes the internet and wireless LANs.

[0158] "Image analysis" is a technology that processes captured image data to identify objects, and includes processing using artificial intelligence technology.

[0159] "Artificial intelligence technology" refers to technologies that enable computer systems to perform perceptions and judgments similar to those of humans, and includes machine learning and deep learning.

[0160] "Emotional state" refers to the psychological or emotional responses exhibited by the user, and is information analyzed from facial expressions and tone of voice.

[0161] "Adjusting the proposal" refers to the process of appropriately changing the details and presentation method of the proposal regarding the disposal of items, taking into account the user's emotional state.

[0162] This invention provides a system for streamlining inventory management and making suggestions that take user emotions into consideration. The system operates using acquisition devices, communication networks, image analysis technology, artificial intelligence technology, and the like.

[0163] The terminal utilizes a network-connected camera or smartphone as an acquisition device to photograph objects in a room. This device periodically or based on user instructions acquires images of objects and transmits the data to a server. Wi-Fi or Bluetooth are commonly used as the communication network.

[0164] The server analyzes the received image data using artificial intelligence technology. For image analysis, software such as OpenCV is used to identify objects based on their shape, color, and size. The analysis results are registered in a database, and the attribute information of the objects is updated.

[0165] Furthermore, the server analyzes the user's emotional state and makes appropriate suggestions. When the user receives a suggestion, the terminal monitors facial expressions, voice, and interaction patterns through an emotion engine and sends this data to the server. Generally, emotion recognition software such as IBM Watson® or Affectiva is used. Based on the analysis results, the server provides the user with information and choices that take their emotions into consideration.

[0166] For example, if a user says, "I'm wondering whether to get rid of this old camera," the server will suggest ways to preserve memories related to the camera and provide examples of effective selling methods to alleviate their concerns. By presenting the user with appropriate information, the server helps them make a decision about disposing of their items with confidence.

[0167] An example of a prompt for a generative AI model is, "When designing a system to manage household items through image analysis, please tell me how to make suggestions that take user emotions into consideration."

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The terminal uses a network-connected camera to capture images of objects in a room. This process automatically acquires images based on user instructions or a regular schedule. Input data consists of images of the objects, while output data is an image file transmitted to a server via the network. Specifically, the terminal has the functionality to start capturing images at predetermined times and immediately transfer the image data to the server.

[0171] Step 2:

[0172] The server receives images of items sent from the terminal. The input is image data, and image analysis techniques are used to identify the items. The output is the type and quantity of the items, and their associated attribute information, which is registered in the database. Specifically, the process involves extracting image features using software such as OpenCV, classifying the items using machine learning models, and then registering them.

[0173] Step 3:

[0174] The server monitors the frequency of item use based on information in the database. The input is the item's usage history in the database, and this data is analyzed to evaluate usage frequency. The output is a list of items that need to be disposed of based on usage frequency. The server tracks changes in usage history and has a periodic analysis function to identify items that have not been used for a certain period.

[0175] Step 4:

[0176] The server generates suggestions for disposing of items and prepares to notify the user. The input is a list of identified unwanted items and market information, and the output is a message containing the suggestions. The server gathers information on market value reassessments and recycling options, and proposes specific disposal methods to the user.

[0177] Step 5:

[0178] When a user receives a suggestion, the device monitors the user's emotions. The input consists of data from the user's facial expressions and voice, which is analyzed by an emotion recognition engine. The output is data indicating the user's emotional state, which is used to adjust the suggestion. The device utilizes its camera and microphone to feed back the user's reactions to the server in real time.

[0179] Step 6:

[0180] The server adjusts the suggestions based on the received sentiment analysis data and provides the final suggestions to the user. The input is the user's emotional state and the original suggestions, and the output is an optimized suggestion that takes emotions into account. Specifically, the server has the function to add additional options such as "Does this information alleviate anxiety?" and further explanations.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] Traditional inventory management systems could suggest disposal based on the frequency of use of items, but they struggled to provide an interface that considered the user's emotional state. Furthermore, system suggestions often didn't resonate with the user's emotions, resulting in a lack of significant improvement in the user experience. Additionally, there was insufficient support to alleviate anxiety about item disposal and enable users to make informed decisions.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes means for photographing items in a space using acquisition means, means for observing and recording the usage status and frequency of use of items, and means for analyzing the user's emotional state using an emotion engine. This enables the generation of optimal item disposal suggestions according to the user's emotional state, thereby improving the user experience and reducing anxiety.

[0186] "Acquisition means" refers to a device for photographing objects in a space, including a photographic device connected to a network.

[0187] "Means for analyzing images and identifying objects" refers to a system that processes captured image data to identify objects present within the image.

[0188] "Means for observing and recording the usage status and frequency of use of an item" refers to a system that has the function of monitoring how much an item is being used and storing that data.

[0189] A "means for generating disposal suggestions for items based on usage frequency" is a system that makes suggestions to users about the disposal of items based on recorded usage data.

[0190] "Methods for analyzing a user's emotional state using an emotion engine" refer to technologies that analyze information such as a user's facial expressions and voice to infer what emotions the user is experiencing.

[0191] "Means of adjusting suggested content based on the user's emotional state" refers to a system that changes suggested content and its presentation method based on the user's emotional information analyzed by an emotion engine.

[0192] "Means for carrying out procedures for automatically listing items on the market" refers to a system that automatically executes the process of selling items that have been decided to be disposed of on the market.

[0193] A "means of arranging donations and recycling" refers to a system that has the function of facilitating the disposal of items as donations or for recycling when they are deemed unnecessary.

[0194] "Means of providing user feedback and improving the system" refers to a mechanism for generating feedback based on system usage and user reactions, and for improving system performance.

[0195] The system for implementing this invention has a configuration comprising acquisition means, analysis means, emotion recognition means, and user interaction means. The system first uses a terminal to photograph objects in space. For this purpose, a smartphone or a network-connected imaging device is used as hardware.

[0196] The server receives the captured image data and uses image analysis software to identify each item. Machine learning algorithms are commonly used for item identification, specifically libraries such as TensorFlow and OpenCV. The server also observes the usage status and frequency of the identified items and records this information in a database.

[0197] Based on an analysis of the frequency of use and condition of items, the server generates disposal suggestions. These suggestions are designed to take into account the market value and recyclability of the items being considered for disposal.

[0198] When user interaction occurs, the server uses an emotion engine to analyze the user's facial expressions and voice to identify their emotional state. This allows the server to present information tailored to the user's current emotions. Emotion recognition may utilize libraries such as Python's `emotion_recognition` library.

[0199] For example, when a user is struggling with whether to dispose of an old item, the server uses an emotion engine to detect the user's anxiety and provides additional information to alleviate it. For instance, it might present examples of successful item sales or methods for preserving memories, thereby providing reassurance.

[0200] An example of a prompt that utilizes a generative AI model is, "When disposing of an item, analyze the emotions the user is experiencing and provide information that will give them peace of mind." In this way, the system can provide an item management experience that is sensitive to the user's emotions.

[0201] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0202] Step 1:

[0203] The terminal uses a network-connected camera to photograph objects in the space. It acquires visual information of the objects as input and generates image data as output. This image data is sent to a server to record the status of the objects.

[0204] Step 2:

[0205] The server analyzes image data received from the terminal. Using the acquired image data as input, it identifies objects using a machine learning model (e.g., TensorFlow) and stores the results in a database. The specific operations in this process are image feature extraction and the application of an identification algorithm.

[0206] Step 3:

[0207] The server monitors and analyzes the usage and frequency of items based on records in the database. It takes log data as input, performs data calculations to identify items with low usage frequency, and generates output that prepares disposal suggestions based on the results. Statistical analysis methods are used in this process.

[0208] Step 4:

[0209] Before notifying the user of the generated disposal suggestions, the server uses an emotion engine to analyze the user's emotional state. It receives user facial expressions and voice data as input, applies an emotion recognition algorithm to estimate the user's emotions, and obtains the estimation result as output.

[0210] Step 5:

[0211] The server adjusts the suggestions based on the emotion recognition results. It uses the emotion data received as input to process and modify the suggestions and information presentation methods, and outputs the results. Specific actions include prioritizing suggestions and selecting additional information.

[0212] Step 6:

[0213] When a user receives a proposal, they send a response back to the server. The server receives this response as input, provides feedback, and outputs it as data to improve system performance. Specific examples include whether to accept or reject a disposal proposal and the selection of additional options.

[0214] Step 7:

[0215] Ultimately, the system will be improved by using a generative AI model based on user feedback to create prompts and provide better suggestions. In this process, user feedback data is used as input, the model is retrained, and an optimized suggestion method is obtained as output.

[0216] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0217] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0218] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0219] [Second Embodiment]

[0220] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0221] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0222] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0223] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0224] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0225] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0226] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0227] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0228] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0229] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0230] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0231] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0232] This invention is a system for supporting the efficient management and disposal of items within the home. The system includes acquisition means, analysis functions, and a process that supports decluttering through interaction with the user.

[0233] First, the terminal activates a camera installed in the space to capture images of objects. This device photographs various objects in the home at regular intervals and sends the latest information to a server. This information serves as basic data for object recognition and management.

[0234] Next, the server analyzes the received images and automatically identifies the type, location, and characteristics of the items. For example, it uses AI-based image recognition technology to identify furniture, home appliances, and personal belongings, and registers them in a database.

[0235] The server then monitors and efficiently records the usage and frequency of each item. It integrates multiple camera inputs to analyze how each item is being used. This allows it to quantify how often an item is used over a certain period and profile its usage frequency.

[0236] Furthermore, the server utilizes usage frequency data to generate optimal decluttering suggestions for the user. For example, regarding unused electronic devices, it might notify the user via their device with a specific suggestion such as, "This item hasn't been used in the last six months. Would you like to consider selling or donating it?" This notification includes information on marketplaces where the item can be sold and potential recipients for donation, supporting quick decision-making.

[0237] If the user accepts the proposal, the server accesses the online marketplace's API and automatically processes the listing of the item. If the user chooses to donate or recycle, the server can also coordinate with relevant businesses to smoothly arrange collection and delivery.

[0238] For example, if a user decides to sell an oven toaster they haven't used for six months, the server sends the necessary information to the marketplace to complete the listing and then notifies the user of the status via their device.

[0239] This invention allows for the efficient management of unused items in the home, facilitating the disposal of unwanted items and maximizing their economic value.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] The device activates its AI camera at set intervals to capture images of objects in the home. The captured images are immediately processed and prepared to extract the contours and features of the objects.

[0243] Step 2:

[0244] The terminal compresses the acquired image data and sends it to the server. The server analyzes the received image data and applies an object recognition algorithm to identify individual items. At this time, the identified items are registered in the database, and the metadata corresponding to each item is updated.

[0245] Step 3:

[0246] The server observes the location and usage of each specific item, recording usage frequency data. For example, if an item is lifted or its position is changed, it is considered used, and the usage frequency is updated.

[0247] Step 4:

[0248] The server extracts items with low usage frequency at regular intervals and generates disposal suggestions for these items. The server uses a generation AI to create disposal suggestion messages based on usage frequency and market information.

[0249] Step 5:

[0250] The user receives disposal suggestions via the terminal and reviews the suggestions. The user considers the options of selling, donating, or recycling the suggested items and enters their selection via the terminal.

[0251] Step 6:

[0252] Based on the user's selection, the server automatically lists the items for sale via the online marketplace's API. If the user chooses donation or recycling, the server handles the necessary procedures with the relevant businesses or organizations.

[0253] Step 7:

[0254] The server notifies the user of the results of the disposal procedure and records that all disposals have been completed. A feedback function collects user opinions and evaluations, which are used to propose future disposals and improve the system.

[0255] (Example 1)

[0256] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0257] In modern households, items tend to accumulate, and their management and disposal are often inefficient. As a result, limited living space is unnecessarily occupied, and their economic value is overlooked. Furthermore, an increase in unused items leads to wasted resources and increased environmental burden. This invention aims to solve these problems by supporting the efficient management of household items, analysis of usage frequency, and appropriate disposal of unwanted items.

[0258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0259] In this invention, the server includes means for photographing objects in a space using acquisition means, means for analyzing the captured images and identifying the objects, and means for observing and recording the usage status and frequency of use of the objects. This enables detailed analysis of objects in the home and appropriate decluttering suggestions to the user.

[0260] "Acquisition means" refers to a device that has the function of taking an image of an item and transmitting that data to a server.

[0261] A "photography device" is a hardware device installed in a home environment to capture images of objects at specific intervals.

[0262] A "server" is a computer system that analyzes captured image data to identify items, record their usage, and generate disposal suggestions.

[0263] "Item identification" is the process of identifying and classifying each item within a captured image using image analysis technology.

[0264] "Usage status" refers to information about how and how often an item is used.

[0265] "Frequency of use" refers to data that shows how many times a particular item was used within a certain period of time.

[0266] "Disposal suggestions" refer to generating suggestions to encourage users to sell or donate items, based on data such as usage frequency.

[0267] "Donation and reuse" refers to the act of transferring unused items to individuals or organizations in need, or to utilizing them in a way that makes them usable again.

[0268] "Feedback" is the process of collecting and analyzing information based on the results of suggestions and user actions in order to improve the performance and usability of the system.

[0269] This system is designed to efficiently manage household items and assist in the disposal of unwanted items. Specifically, it involves installing network-connected cameras in each space to periodically acquire images of items. For example, in the living room, the camera takes a picture of the entire room at a set time each day, recording past changes.

[0270] The terminal controls the shooting process and sends the captured images to the server in real time. The server receives these images and performs image analysis using a generative AI model. This analysis automatically identifies objects, classifies them into specific categories, and registers them in a database. Software used for this includes image analysis tools such as TensorFlow and OpenCV.

[0271] The server also monitors the usage status and frequency of items in real time and stores this data. Based on this usage frequency data, it generates decluttering suggestions for unused items and notifies the user of specific suggestions via their device. For example, a suggestion might be, "This mixer hasn't been used in the last six months. Would you like to consider selling or donating it?"

[0272] If a user accepts a proposal, the server automatically calls the online marketplace API to list the item for sale. If the user chooses to donate or reuse the item, the server coordinates with relevant services to smoothly arrange collection and delivery. This entire process streamlines the management and disposal of items without requiring any user intervention.

[0273] Prompt statements as concrete examples:

[0274] "List items that haven't been used in the past six months and suggest decluttering options to users. Example: This guitar hasn't been used for six months, so please suggest where to sell or donate it."

[0275] This system provides a practical solution for effectively utilizing space within the home and maximizing its economic value.

[0276] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0277] Step 1:

[0278] The terminal controls the imaging device installed within the home and sets a schedule for periodically imaging the items in each room. The inputs are the current date and time, and a pre-set imaging schedule. The output is the image data of the imaged items, which is transmitted to the server in real time.

[0279] Step 2:

[0280] The server analyzes the received images using AI-based image recognition software. The input is the image data transmitted from the terminal. The server utilizes a generated AI model to individually identify the items through image analysis and classify them based on their characteristics. The output obtained from this process is the category information of the items.

[0281] Step 3:

[0282] The server registers the image analysis results in the database and observes the usage status and frequency of use of the items. The inputs are the category information of the items obtained from image analysis and past usage data. Based on this information, the server calculates the usage frequency of each item and records it as usage history data. This output data is used for generating decluttering proposals later.

[0283] Step 4:

[0284] The server analyzes the usage frequency data of the items and generates decluttering proposals for the items considered unnecessary. The input is the usage status and frequency data. The server uses the generated AI model to materialize the proposals to be presented to the user. The output is a specific proposal text, which includes content such as "This product has not been used for the past six months. Have you considered selling or donating it?"

[0285] [[ID=2十七]] Step 5:

[0286] The terminal receives the proposal notification from the server and displays it to the user. The input is the proposal text sent from the server. The user confirms the proposal and makes a decision from the options of selling, donating, or recycling. The output is the option selected by the user.

[0287] Step 6:

[0288] The server executes the necessary procedures according to the user's selection. The input is the user's selection information. The server calls the marketplace API to perform automatic listing or contact the donation organization to arrange for delivery. After this procedure is completed, the processing status is notified to the terminal, and the user is provided with the final feedback. The output is the result of the executed procedure.

[0289] (Application Example 1)

[0290] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0291] In modern households, the number of items continues to increase, making their management and proper disposal difficult. In particular, the process of regularly reviewing items and efficiently disposing of unnecessary ones requires time and effort, so it is easily postponed in many households. Furthermore, environmental considerations are required for item disposal, and not only simple disposal but also options such as reuse and donation need to be considered. There is a need for a system that solves such problems, efficiently manages household items, and supports their disposal.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0293] In this invention, the server includes means for the autonomous mobile device to patrol the environment and collect item data, means for generating a proposal for item disposal based on the usage frequency, and means for performing procedures for automatically listing items on the market. Thereby, it becomes possible to always manage household items in the latest state, propose effective disposal methods for unused items, and automate the procedures.

[0294] The "acquisition means" refers to a device or method for acquiring an item existing in the space as an image.

[0295] "Photography equipment" refers to devices such as cameras and sensors used to capture images of objects.

[0296] "Image analysis" refers to the process of identifying objects from acquired image data and determining their characteristics and type.

[0297] "Item usage status" refers to information indicating how a particular item is being used or not being used.

[0298] "Frequency of use" refers to numerical data indicating how many times an item was used within a specific period.

[0299] A "disposal suggestion" refers to a proposal to the user on how to dispose of a specific item based on its frequency of use or other conditions.

[0300] "Automatic listing" refers to the process of automatically listing products on the marketplace.

[0301] "Donation and reuse" refers to providing items to new users or allowing them to be reused, rather than simply discarding them.

[0302] An "autonomous mobile device" refers to a robot or machine that has the function of automatically patrolling a home and collecting necessary data.

[0303] "Feedback" refers to information that reflects the data and results obtained by the system or users, and is used to help with the next steps and improvements.

[0304] The system implementing this invention is designed to effectively support the management and disposal of items within the home. The system operates by integrating multiple hardware and software components.

[0305] First, the autonomous mobile device patrols within the home and acquires images of all items. The autonomous mobile device is equipped with a high-performance camera and sensors connected to a communication network, and accurately captures the positions and features of items within the space. The captured image data is transmitted to a server on the cloud through the communication network.

[0306] The server processes the received image data using AI-based image analysis technology to identify the types, positions, and features of the items. At this time, an image analysis API such as the Google Cloud Vision API is used to accurately identify the items. The identified information is registered in the database, and the usage frequency of the items is recorded.

[0307] Furthermore, the server generates a disposal proposal based on the usage frequency. This proposal includes recommendations for selling or donating unused items, and also provides disposal methods and market information. Using a notification system such as Amazon SNS, the user's smartphone or terminal is notified in real time. If the user accepts the proposal, the server cooperates with the market's API to automatically execute the listing process or conduct procedures for the donation destination.

[0308] As a specific example, the system detects a toaster oven that has not been used for half a year and sends a notification to the user saying, "This toaster oven has not been used for half a year. Are you considering selling or donating it?" The user who receives this notification selects to sell via the terminal, and that information is immediately transmitted to the online marketplace.

[0309] By leveraging the generated AI model, it is possible to derive more advanced proposal models and solutions from a prompt sentence such as, "Please teach me the design method of an AI assistant that discriminates items with low usage frequency in the home and makes proposals for selling or donating them."

[0310] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0311] Step 1:

[0312] An autonomous mobile device patrols the home, acquiring images of objects with its onboard high-performance camera. This image data, including the location and characteristics of the objects, is transmitted to a server via a communication network. The input is the raw image data acquired by the camera, and the output is the image data transmitted to the server.

[0313] Step 2:

[0314] The server receives image data and performs AI-based image analysis. The input here is image data, and the type and characteristics of the objects are identified using the Google Cloud Vision API. The output is information data about the identified objects, which is registered in the database.

[0315] Step 3:

[0316] The server observes and records usage status and frequency based on the information of registered items. This data analysis takes the past usage history of each item as input and outputs data that profiles the frequency of use of each item.

[0317] Step 4:

[0318] The server generates disposal suggestions for items based on usage frequency data. The input is numerical data regarding usage frequency, and the output is a specific disposal suggestion that is communicated to the user. For example, it might generate recommendations to sell or donate an unused toaster oven.

[0319] Step 5:

[0320] The terminal notifies the user in real time of disposal suggestions from the server. The input is the content of the suggestion sent from the server, and the output is the information delivered to the user via the terminal's screen or audio.

[0321] Step 6:

[0322] If the user accepts the proposal, the server uses the marketplace API to initiate the automated listing process for the item. The input here is the user's selection information, and the output is the status of the item's listing on the marketplace.

[0323] Step 7:

[0324] The server also provides a means to arrange donations and reuse, and facilitates appropriate coordination according to the user's wishes. The input is the user's choice of disposal method, and the output is the status of the procedures with the donation or reuse recipient.

[0325] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0326] This invention adds an emotion engine to a system designed to improve the efficiency of item management, thereby providing optimal suggestions and interfaces that take into account the user's emotional state. This system includes a process for effectively disposing of household items through acquisition means, analysis functions, emotion recognition, and user interaction.

[0327] First, the terminal uses a network-connected camera to capture images of items in the room. This image data is sent to a server to continuously monitor the condition of the items. The server uses image analysis technology to identify each item and register it in a database.

[0328] The server monitors the frequency and condition of item usage, identifying items that are not being used. If an item is used infrequently, the server generates a suggestion and prepares a notification for the user. This suggestion takes into account the item's market value and recyclability.

[0329] The key here is the emotion engine built into the server. When a user receives a suggestion using their device, the emotion engine analyzes the user's facial expressions, voice, and interaction patterns to estimate the user's current emotional state. The results of the emotion recognition are used to adjust how the suggestion is presented. For example, if a user shows anxiety about a disposal suggestion, the emotion engine detects this, and the server provides the user with additional information or options to alleviate the anxiety.

[0330] For example, if a user expresses an emotion such as "I'm wondering whether to get rid of this old camera," the emotion engine detects this through the device. The server then provides additional reassuring information to help the user make a decision, such as ways to preserve cherished photos from the camera or examples of successful sales of similar items.

[0331] Thus, by introducing an emotion engine, the present invention can go beyond mere item management and provide an experience that is attuned to the user's emotions. As a result, users can efficiently manage and dispose of items without emotional burden.

[0332] The following describes the processing flow.

[0333] Step 1:

[0334] The terminal periodically activates a network-connected camera to capture images of objects in the space. The captured images are optimized to extract the features of the objects and then sent to a server.

[0335] Step 2:

[0336] The server analyzes the received image data and uses machine learning algorithms to identify items. During this process, it recognizes the unique characteristics of each item and registers them in a database. Item classifications and historical data are also associated with these items.

[0337] Step 3:

[0338] The server continuously monitors the acquired data to record the frequency and usage status of each item. It also tracks the movement and actions of items and updates usage information accordingly.

[0339] Step 4:

[0340] The server, having identified infrequently used items, automatically generates disposal suggestions. These suggestions include information such as the item's market value, recyclability, and where it can be donated. The suggestions are then prepared to be notified to the user at an appropriate time.

[0341] Step 5:

[0342] The emotion engine integrated into the server activates when the user receives a disposal suggestion. The terminal collects emotional data from the user's facial expressions and voice and sends it to the server in real time. The server uses the emotion engine to analyze the user's emotional state and make any necessary adjustments.

[0343] Step 6:

[0344] Users can review disposal suggestions on their devices and make selections based on their own emotions. The suggestions are customized based on the emotions engine's judgment, and additional information is added to alleviate the user's anxiety and hesitation.

[0345] Step 7:

[0346] After the user selects a disposal method for their items, the server accesses online marketplaces and automatically lists the items. If donation or recycling is selected, the server notifies the relevant businesses and initiates the necessary procedures. The server then verifies that the disposal is complete and provides feedback to the user.

[0347] (Example 2)

[0348] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0349] Managing belongings in modern homes and offices is becoming increasingly complex due to the sheer number and diversity of items. In particular, traditional systems are insufficient to efficiently identify unwanted items, select appropriate disposal methods, and support decision-making that considers user emotions. Therefore, there is a need for a system that not only improves the efficiency of belongings management but also reduces the psychological burden on users.

[0350] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0351] In this invention, the server includes means for photographing objects in a space using an acquisition device, means for analyzing the captured images and identifying the objects, and means for analyzing the user's emotional state and adjusting the suggested content. This enables efficient management of objects and improved accuracy of disposal suggestions, and further reduces psychological burden by providing suggestions that are sensitive to the user's emotions.

[0352] "Acquisition device" is a general term for equipment and systems used to photograph objects in a space, and mainly refers to cameras, smartphones, etc.

[0353] A "communication network" is an infrastructure for transmitting digital data, and includes the internet and wireless LANs.

[0354] "Image analysis" is a technology that processes captured image data to identify objects, and includes processing using artificial intelligence technology.

[0355] "Artificial intelligence technology" refers to technologies that enable computer systems to perform perceptions and judgments similar to those of humans, and includes machine learning and deep learning.

[0356] "Emotional state" refers to the psychological or emotional responses exhibited by the user, and is information analyzed from facial expressions and tone of voice.

[0357] "Adjusting the proposal" refers to the process of appropriately changing the details and presentation method of the proposal regarding the disposal of items, taking into account the user's emotional state.

[0358] This invention provides a system for streamlining inventory management and making suggestions that take user emotions into consideration. The system operates using acquisition devices, communication networks, image analysis technology, artificial intelligence technology, and the like.

[0359] The terminal utilizes a network-connected camera or smartphone as an acquisition device to photograph objects in a room. This device periodically or based on user instructions acquires images of objects and transmits the data to a server. Wi-Fi or Bluetooth are commonly used as the communication network.

[0360] The server analyzes the received image data using artificial intelligence technology. For image analysis, software such as OpenCV is used to identify objects based on their shape, color, and size. The analysis results are registered in a database, and the attribute information of the objects is updated.

[0361] Furthermore, the server analyzes the user's emotional state and provides appropriate suggestions. When the user receives a suggestion, the terminal monitors facial expressions, voice, and interaction patterns through an emotion engine and sends this data to the server. Generally, emotion recognition software such as IBM Watson or Affectiva is used. Based on the analysis results, the server provides the user with information and choices that take their emotions into consideration.

[0362] For example, if a user says, "I'm wondering whether to get rid of this old camera," the server will suggest ways to preserve memories related to the camera and provide examples of effective selling methods to alleviate their concerns. By presenting the user with appropriate information, the server helps them make a decision about disposing of their items with confidence.

[0363] An example of a prompt for a generative AI model is, "When designing a system to manage household items through image analysis, please tell me how to make suggestions that take user emotions into consideration."

[0364] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0365] Step 1:

[0366] The terminal uses a network-connected camera to capture images of objects in a room. This process automatically acquires images based on user instructions or a regular schedule. Input data consists of images of the objects, while output data is an image file transmitted to a server via the network. Specifically, the terminal has the functionality to start capturing images at predetermined times and immediately transfer the image data to the server.

[0367] Step 2:

[0368] The server receives images of items sent from the terminal. The input is image data, and image analysis techniques are used to identify the items. The output is the type and quantity of the items, and their associated attribute information, which is registered in the database. Specifically, the process involves extracting image features using software such as OpenCV, classifying the items using machine learning models, and then registering them.

[0369] Step 3:

[0370] The server monitors the frequency of item use based on information in the database. The input is the item's usage history in the database, and this data is analyzed to evaluate usage frequency. The output is a list of items that need to be disposed of based on usage frequency. The server tracks changes in usage history and has a periodic analysis function to identify items that have not been used for a certain period.

[0371] Step 4:

[0372] The server generates suggestions for disposing of items and prepares to notify the user. The input is a list of identified unwanted items and market information, and the output is a message containing the suggestions. The server gathers information on market value reassessments and recycling options, and proposes specific disposal methods to the user.

[0373] Step 5:

[0374] When a user receives a suggestion, the device monitors the user's emotions. The input consists of data from the user's facial expressions and voice, which is analyzed by an emotion recognition engine. The output is data indicating the user's emotional state, which is used to adjust the suggestion. The device utilizes its camera and microphone to feed back the user's reactions to the server in real time.

[0375] Step 6:

[0376] The server adjusts the suggestions based on the received sentiment analysis data and provides the final suggestions to the user. The input is the user's emotional state and the original suggestions, and the output is an optimized suggestion that takes emotions into account. Specifically, the server has the function to add additional options such as "Does this information alleviate anxiety?" and further explanations.

[0377] (Application Example 2)

[0378] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0379] Traditional inventory management systems could suggest disposal based on the frequency of use of items, but they struggled to provide an interface that considered the user's emotional state. Furthermore, system suggestions often didn't resonate with the user's emotions, resulting in a lack of significant improvement in the user experience. Additionally, there was insufficient support to alleviate anxiety about item disposal and enable users to make informed decisions.

[0380] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0381] In this invention, the server includes means for photographing items in a space using acquisition means, means for observing and recording the usage status and frequency of use of items, and means for analyzing the user's emotional state using an emotion engine. This enables the generation of optimal item disposal suggestions according to the user's emotional state, thereby improving the user experience and reducing anxiety.

[0382] "Acquisition means" refers to a device for photographing objects in a space, including a photographic device connected to a network.

[0383] "Means for analyzing images and identifying objects" refers to a system that processes captured image data to identify objects present within the image.

[0384] "Means for observing and recording the usage status and frequency of use of an item" refers to a system that has the function of monitoring how much an item is being used and storing that data.

[0385] A "means for generating disposal suggestions for items based on usage frequency" is a system that makes suggestions to users about the disposal of items based on recorded usage data.

[0386] "Methods for analyzing a user's emotional state using an emotion engine" refer to technologies that analyze information such as a user's facial expressions and voice to infer what emotions the user is experiencing.

[0387] "Means of adjusting suggested content based on the user's emotional state" refers to a system that changes suggested content and its presentation method based on the user's emotional information analyzed by an emotion engine.

[0388] "Means for carrying out procedures for automatically listing items on the market" refers to a system that automatically executes the process of selling items that have been decided to be disposed of on the market.

[0389] A "means of arranging donations and recycling" refers to a system that has the function of facilitating the disposal of items as donations or for recycling when they are deemed unnecessary.

[0390] "Means of providing user feedback and improving the system" refers to a mechanism for generating feedback based on system usage and user reactions, and for improving system performance.

[0391] The system for implementing this invention has a configuration comprising acquisition means, analysis means, emotion recognition means, and user interaction means. The system first uses a terminal to photograph objects in space. For this purpose, a smartphone or a network-connected imaging device is used as hardware.

[0392] The server receives the captured image data and uses image analysis software to identify each item. Machine learning algorithms are commonly used for item identification, specifically libraries such as TensorFlow and OpenCV. The server also observes the usage status and frequency of the identified items and records this information in a database.

[0393] Based on an analysis of the frequency of use and condition of items, the server generates disposal suggestions. These suggestions are designed to take into account the market value and recyclability of the items being considered for disposal.

[0394] When user interaction occurs, the server uses an emotion engine to analyze the user's facial expressions and voice to identify their emotional state. This allows the server to present information tailored to the user's current emotions. Emotion recognition may utilize libraries such as Python's `emotion_recognition` library.

[0395] For example, when a user is struggling with whether to dispose of an old item, the server uses an emotion engine to detect the user's anxiety and provides additional information to alleviate it. For instance, it might present examples of successful item sales or methods for preserving memories, thereby providing reassurance.

[0396] An example of a prompt that utilizes a generative AI model is, "When disposing of an item, analyze the emotions the user is experiencing and provide information that will give them peace of mind." In this way, the system can provide an item management experience that is sensitive to the user's emotions.

[0397] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0398] Step 1:

[0399] The terminal uses a network-connected camera to photograph objects in the space. It acquires visual information of the objects as input and generates image data as output. This image data is sent to a server to record the status of the objects.

[0400] Step 2:

[0401] The server analyzes image data received from the terminal. Using the acquired image data as input, it identifies objects using a machine learning model (e.g., TensorFlow) and stores the results in a database. The specific operations in this process are image feature extraction and the application of an identification algorithm.

[0402] Step 3:

[0403] The server monitors and analyzes the usage and frequency of items based on records in the database. It takes log data as input, performs data calculations to identify items with low usage frequency, and generates output that prepares disposal suggestions based on the results. Statistical analysis methods are used in this process.

[0404] Step 4:

[0405] Before notifying the user of the generated disposal suggestions, the server uses an emotion engine to analyze the user's emotional state. It receives user facial expressions and voice data as input, applies an emotion recognition algorithm to estimate the user's emotions, and obtains the estimation result as output.

[0406] Step 5:

[0407] The server adjusts the suggestions based on the emotion recognition results. It uses the emotion data received as input to process and modify the suggestions and information presentation methods, and outputs the results. Specific actions include prioritizing suggestions and selecting additional information.

[0408] Step 6:

[0409] When a user receives a proposal, they send a response back to the server. The server receives this response as input, provides feedback, and outputs it as data to improve system performance. Specific examples include whether to accept or reject a disposal proposal and the selection of additional options.

[0410] Step 7:

[0411] Ultimately, the system will be improved by using a generative AI model based on user feedback to create prompts and provide better suggestions. In this process, user feedback data is used as input, the model is retrained, and an optimized suggestion method is obtained as output.

[0412] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0413] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0414] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0415] [Third Embodiment]

[0416] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0417] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0418] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0419] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0420] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0421] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0422] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0423] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0424] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0425] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0426] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0427] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0428] This invention is a system for supporting the efficient management and disposal of items within the home. The system includes acquisition means, analysis functions, and a process that supports decluttering through interaction with the user.

[0429] First, the terminal activates a camera installed in the space to capture images of objects. This device photographs various objects in the home at regular intervals and sends the latest information to a server. This information serves as basic data for object recognition and management.

[0430] Next, the server analyzes the received images and automatically identifies the type, location, and characteristics of the items. For example, it uses AI-based image recognition technology to identify furniture, home appliances, and personal belongings, and registers them in a database.

[0431] The server then monitors and efficiently records the usage and frequency of each item. It integrates multiple camera inputs to analyze how each item is being used. This allows it to quantify how often an item is used over a certain period and profile its usage frequency.

[0432] Furthermore, the server utilizes usage frequency data to generate optimal decluttering suggestions for the user. For example, regarding unused electronic devices, it might notify the user via their device with a specific suggestion such as, "This item hasn't been used in the last six months. Would you like to consider selling or donating it?" This notification includes information on marketplaces where the item can be sold and potential recipients for donation, supporting quick decision-making.

[0433] If the user accepts the proposal, the server accesses the online marketplace's API and automatically processes the listing of the item. If the user chooses to donate or recycle, the server can also coordinate with relevant businesses to smoothly arrange collection and delivery.

[0434] For example, if a user decides to sell an oven toaster they haven't used for six months, the server sends the necessary information to the marketplace to complete the listing and then notifies the user of the status via their device.

[0435] This invention allows for the efficient management of unused items in the home, facilitating the disposal of unwanted items and maximizing their economic value.

[0436] The following describes the processing flow.

[0437] Step 1:

[0438] The device activates its AI camera at set intervals to capture images of objects in the home. The captured images are immediately processed and prepared to extract the contours and features of the objects.

[0439] Step 2:

[0440] The terminal compresses the acquired image data and sends it to the server. The server analyzes the received image data and applies an object recognition algorithm to identify individual items. At this time, the identified items are registered in the database, and the metadata corresponding to each item is updated.

[0441] Step 3:

[0442] The server observes the location and usage of each specific item, recording usage frequency data. For example, if an item is lifted or its position is changed, it is considered used, and the usage frequency is updated.

[0443] Step 4:

[0444] The server extracts items with low usage frequency at regular intervals and generates disposal suggestions for these items. The server uses a generation AI to create disposal suggestion messages based on usage frequency and market information.

[0445] Step 5:

[0446] The user receives disposal suggestions via the terminal and reviews the suggestions. The user considers the options of selling, donating, or recycling the suggested items and enters their selection via the terminal.

[0447] Step 6:

[0448] Based on the user's selection, the server automatically lists the items for sale via the online marketplace's API. If the user chooses donation or recycling, the server handles the necessary procedures with the relevant businesses or organizations.

[0449] Step 7:

[0450] The server notifies the user of the results of the disposal procedure and records that all disposals have been completed. A feedback function collects user opinions and evaluations, which are used to propose future disposals and improve the system.

[0451] (Example 1)

[0452] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0453] In modern households, items tend to accumulate, and their management and disposal are often inefficient. As a result, limited living space is unnecessarily occupied, and their economic value is overlooked. Furthermore, an increase in unused items leads to wasted resources and increased environmental burden. This invention aims to solve these problems by supporting the efficient management of household items, analysis of usage frequency, and appropriate disposal of unwanted items.

[0454] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0455] In this invention, the server includes means for photographing objects in a space using acquisition means, means for analyzing the captured images and identifying the objects, and means for observing and recording the usage status and frequency of use of the objects. This enables detailed analysis of objects in the home and appropriate decluttering suggestions to the user.

[0456] "Acquisition means" refers to a device that has the function of taking an image of an item and transmitting that data to a server.

[0457] A "photography device" is a hardware device installed in a home environment to capture images of objects at specific intervals.

[0458] A "server" is a computer system that analyzes captured image data to identify items, record their usage, and generate disposal suggestions.

[0459] "Item identification" is the process of identifying and classifying each item within a captured image using image analysis technology.

[0460] "Usage status" refers to information about how and how often an item is used.

[0461] "Frequency of use" refers to data that shows how many times a particular item was used within a certain period of time.

[0462] "Disposal suggestions" refer to generating suggestions to encourage users to sell or donate items, based on data such as usage frequency.

[0463] "Donation and reuse" refers to the act of transferring unused items to individuals or organizations in need, or to utilizing them in a way that makes them usable again.

[0464] "Feedback" is the process of collecting and analyzing information based on the results of suggestions and user actions in order to improve the performance and usability of the system.

[0465] This system is designed to efficiently manage household items and assist in the disposal of unwanted items. Specifically, it involves installing network-connected cameras in each space to periodically acquire images of items. For example, in the living room, the camera takes a picture of the entire room at a set time each day, recording past changes.

[0466] The terminal controls the shooting process and sends the captured images to the server in real time. The server receives these images and performs image analysis using a generative AI model. This analysis automatically identifies objects, classifies them into specific categories, and registers them in a database. Software used for this includes image analysis tools such as TensorFlow and OpenCV.

[0467] The server also monitors the usage status and frequency of items in real time and stores this data. Based on this usage frequency data, it generates decluttering suggestions for unused items and notifies the user of specific suggestions via their device. For example, a suggestion might be, "This mixer hasn't been used in the last six months. Would you like to consider selling or donating it?"

[0468] If a user accepts a proposal, the server automatically calls the online marketplace API to list the item for sale. If the user chooses to donate or reuse the item, the server coordinates with relevant services to smoothly arrange collection and delivery. This entire process streamlines the management and disposal of items without requiring any user intervention.

[0469] Prompt statements as concrete examples:

[0470] "List items that haven't been used in the past six months and suggest decluttering options to users. Example: This guitar hasn't been used for six months, so please suggest where to sell or donate it."

[0471] This system provides a practical solution for effectively utilizing space within the home and maximizing its economic value.

[0472] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0473] Step 1:

[0474] The terminal controls a camera installed in the home and sets a schedule for periodically photographing items in each room. Inputs are the current date and time, and a pre-set shooting schedule. Output is image data of the photographed items, which is transmitted to the server in real time.

[0475] Step 2:

[0476] The server analyzes the received images using AI-based image recognition software. The input is image data sent from the terminal. The server utilizes a generative AI model to identify individual items through image analysis and classify them based on their characteristics. The output obtained from this process is category information for the items.

[0477] Step 3:

[0478] The server registers the image analysis results in a database and observes the usage status and frequency of items. The input consists of item category information obtained from image analysis and past usage data. Based on this information, the server calculates the usage frequency of each item and records it as usage history data. This output data is later used to generate decluttering suggestions.

[0479] Step 4:

[0480] The server analyzes usage frequency data for items and generates decluttering suggestions for items deemed unnecessary. The input is usage and frequency data. The server uses a generative AI model to refine the suggestions presented to the user. The output is a specific suggestion statement, such as, "This item hasn't been used in the last six months. Would you consider selling or donating it?"

[0481] Step 5:

[0482] The terminal receives a proposal notification from the server and displays it to the user. The input is the proposal text sent from the server. The user reviews the proposal and makes a decision from the options of selling, donating, or recycling. The output is the option selected by the user.

[0483] Step 6:

[0484] The server performs the necessary procedures based on the user's selection. The input is the user's selection information. The server calls the marketplace API to either automatically list the item or contact the charity to arrange delivery. After this procedure is complete, the status is notified to the terminal, and the user is provided with final feedback. The output is the result of the procedures performed.

[0485] (Application Example 1)

[0486] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0487] In modern households, the number of items continues to increase, making management and proper disposal difficult. In particular, the process of regularly reviewing belongings and efficiently disposing of unnecessary items is time-consuming and laborious, often leading to it being postponed in many households. Furthermore, environmental considerations are necessary when disposing of items; options such as reuse and donation should be considered, rather than simply throwing things away. A system is needed to address these challenges and support the efficient management and disposal of household items.

[0488] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0489] In this invention, the server includes means for an autonomous mobile device to patrol the environment and collect item data, means for generating disposal suggestions for items based on usage frequency, and means for performing procedures for automatically listing items on the market. This makes it possible to keep household items constantly up-to-date, propose effective disposal methods for unused items, and automate the procedures.

[0490] "Acquisition means" refers to devices or methods for acquiring images of objects present in a space.

[0491] "Photography equipment" refers to devices such as cameras and sensors used to capture images of objects.

[0492] "Image analysis" refers to the process of identifying objects from acquired image data and determining their characteristics and type.

[0493] "Item usage status" refers to information indicating how a particular item is being used or not being used.

[0494] "Frequency of use" refers to numerical data indicating how many times an item was used within a specific period.

[0495] A "disposal suggestion" refers to a proposal to the user on how to dispose of a specific item based on its frequency of use or other conditions.

[0496] "Automatic listing" refers to the process of automatically listing products on the marketplace.

[0497] "Donation and reuse" refers to providing items to new users or allowing them to be reused, rather than simply discarding them.

[0498] An "autonomous mobile device" refers to a robot or machine that has the function of automatically patrolling a home and collecting necessary data.

[0499] "Feedback" refers to information that reflects the data and results obtained by the system or users, and is used to help with the next steps and improvements.

[0500] The system implementing this invention is designed to effectively support the management and disposal of items within the home. The system operates by integrating multiple hardware and software components.

[0501] First, an autonomous mobile device patrols the home, acquiring images of all objects. The autonomous mobile device is equipped with high-performance cameras and sensors connected to a communication network, accurately capturing the location and characteristics of objects within the space. The captured image data is transmitted via the communication network to a server in the cloud.

[0502] The server processes the received image data using AI-based image analysis technology to identify the type, location, and characteristics of items. It uses image analysis APIs such as the Google Cloud Vision API to accurately identify items. The identified information is registered in a database, and the frequency of use of each item is recorded.

[0503] Furthermore, the server generates disposal suggestions based on usage frequency. These suggestions include recommendations for selling or donating unused items, and also provide disposal methods and market information. Notifications are sent to the user's smartphone or device in real time using a notification system like Amazon SNS. If the user accepts a suggestion, the server automatically executes the listing process by integrating with market APIs or initiates the donation process.

[0504] As a concrete example, the system detects an oven toaster that hasn't been used for six months and sends a notification to the user saying, "This oven toaster hasn't been used for six months. Would you like to consider selling or donating it?" Upon receiving this notification, the user chooses to sell it via their device, and that information is instantly sent to an online marketplace.

[0505] By utilizing generative AI models, it is possible to derive more sophisticated suggestion models and solutions from prompts such as, "Please tell me how to design an AI assistant that identifies items infrequently used in the home and suggests selling or donating them."

[0506] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0507] Step 1:

[0508] An autonomous mobile device patrols the home, acquiring images of objects with its onboard high-performance camera. This image data, including the location and characteristics of the objects, is transmitted to a server via a communication network. The input is the raw image data acquired by the camera, and the output is the image data transmitted to the server.

[0509] Step 2:

[0510] The server receives image data and performs AI-based image analysis. The input here is image data, and the type and characteristics of the objects are identified using the Google Cloud Vision API. The output is information data about the identified objects, which is registered in the database.

[0511] Step 3:

[0512] The server observes and records usage status and frequency based on the information of registered items. This data analysis takes the past usage history of each item as input and outputs data that profiles the frequency of use of each item.

[0513] Step 4:

[0514] The server generates disposal suggestions for items based on usage frequency data. The input is numerical data regarding usage frequency, and the output is a specific disposal suggestion that is communicated to the user. For example, it might generate recommendations to sell or donate an unused toaster oven.

[0515] Step 5:

[0516] The terminal notifies the user in real time of disposal suggestions from the server. The input is the content of the suggestion sent from the server, and the output is the information delivered to the user via the terminal's screen or audio.

[0517] Step 6:

[0518] If the user accepts the proposal, the server uses the marketplace API to initiate the automated listing process for the item. The input here is the user's selection information, and the output is the status of the item's listing on the marketplace.

[0519] Step 7:

[0520] The server also provides a means to arrange donations and reuse, and facilitates appropriate coordination according to the user's wishes. The input is the user's choice of disposal method, and the output is the status of the procedures with the donation or reuse recipient.

[0521] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0522] This invention adds an emotion engine to a system designed to improve the efficiency of item management, thereby providing optimal suggestions and interfaces that take into account the user's emotional state. This system includes a process for effectively disposing of household items through acquisition means, analysis functions, emotion recognition, and user interaction.

[0523] First, the terminal uses a network-connected camera to capture images of items in the room. This image data is sent to a server to continuously monitor the condition of the items. The server uses image analysis technology to identify each item and register it in a database.

[0524] The server monitors the frequency and condition of item usage, identifying items that are not being used. If an item is used infrequently, the server generates a suggestion and prepares a notification for the user. This suggestion takes into account the item's market value and recyclability.

[0525] The key here is the emotion engine built into the server. When a user receives a suggestion using their device, the emotion engine analyzes the user's facial expressions, voice, and interaction patterns to estimate the user's current emotional state. The results of the emotion recognition are used to adjust how the suggestion is presented. For example, if a user shows anxiety about a disposal suggestion, the emotion engine detects this, and the server provides the user with additional information or options to alleviate the anxiety.

[0526] For example, if a user expresses an emotion such as "I'm wondering whether to get rid of this old camera," the emotion engine detects this through the device. The server then provides additional reassuring information to help the user make a decision, such as ways to preserve cherished photos from the camera or examples of successful sales of similar items.

[0527] Thus, by introducing an emotion engine, the present invention can go beyond mere item management and provide an experience that is attuned to the user's emotions. As a result, users can efficiently manage and dispose of items without emotional burden.

[0528] The following describes the processing flow.

[0529] Step 1:

[0530] The terminal periodically activates a network-connected camera to capture images of objects in the space. The captured images are optimized to extract the features of the objects and then sent to a server.

[0531] Step 2:

[0532] The server analyzes the received image data and uses machine learning algorithms to identify items. During this process, it recognizes the unique characteristics of each item and registers them in a database. Item classifications and historical data are also associated with these items.

[0533] Step 3:

[0534] The server continuously monitors the acquired data to record the frequency and usage status of each item. It also tracks the movement and actions of items and updates usage information accordingly.

[0535] Step 4:

[0536] The server, having identified infrequently used items, automatically generates disposal suggestions. These suggestions include information such as the item's market value, recyclability, and where it can be donated. The suggestions are then prepared to be notified to the user at an appropriate time.

[0537] Step 5:

[0538] The emotion engine integrated into the server activates when the user receives a disposal suggestion. The terminal collects emotional data from the user's facial expressions and voice and sends it to the server in real time. The server uses the emotion engine to analyze the user's emotional state and make any necessary adjustments.

[0539] Step 6:

[0540] Users can review disposal suggestions on their devices and make selections based on their own emotions. The suggestions are customized based on the emotions engine's judgment, and additional information is added to alleviate the user's anxiety and hesitation.

[0541] Step 7:

[0542] After the user selects a disposal method for their items, the server accesses online marketplaces and automatically lists the items. If donation or recycling is selected, the server notifies the relevant businesses and initiates the necessary procedures. The server then verifies that the disposal is complete and provides feedback to the user.

[0543] (Example 2)

[0544] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0545] Managing belongings in modern homes and offices is becoming increasingly complex due to the sheer number and diversity of items. In particular, traditional systems are insufficient to efficiently identify unwanted items, select appropriate disposal methods, and support decision-making that considers user emotions. Therefore, there is a need for a system that not only improves the efficiency of belongings management but also reduces the psychological burden on users.

[0546] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0547] In this invention, the server includes means for photographing objects in a space using an acquisition device, means for analyzing the captured images and identifying the objects, and means for analyzing the user's emotional state and adjusting the suggested content. This enables efficient management of objects and improved accuracy of disposal suggestions, and further reduces psychological burden by providing suggestions that are sensitive to the user's emotions.

[0548] "Acquisition device" is a general term for equipment and systems used to photograph objects in a space, and mainly refers to cameras, smartphones, etc.

[0549] A "communication network" is an infrastructure for transmitting digital data, and includes the internet and wireless LANs.

[0550] "Image analysis" is a technology that processes captured image data to identify objects, and includes processing using artificial intelligence technology.

[0551] "Artificial intelligence technology" refers to technologies that enable computer systems to perform perceptions and judgments similar to those of humans, and includes machine learning and deep learning.

[0552] "Emotional state" refers to the psychological or emotional responses exhibited by the user, and is information analyzed from facial expressions and tone of voice.

[0553] "Adjusting the proposal" refers to the process of appropriately changing the details and presentation method of the proposal regarding the disposal of items, taking into account the user's emotional state.

[0554] This invention provides a system for streamlining inventory management and making suggestions that take user emotions into consideration. The system operates using acquisition devices, communication networks, image analysis technology, artificial intelligence technology, and the like.

[0555] The terminal utilizes a network-connected camera or smartphone as an acquisition device to photograph objects in a room. This device periodically or based on user instructions acquires images of objects and transmits the data to a server. Wi-Fi or Bluetooth are commonly used as the communication network.

[0556] The server analyzes the received image data using artificial intelligence technology. For image analysis, software such as OpenCV is used to identify objects based on their shape, color, and size. The analysis results are registered in a database, and the attribute information of the objects is updated.

[0557] Furthermore, the server analyzes the user's emotional state and provides appropriate suggestions. When the user receives a suggestion, the terminal monitors facial expressions, voice, and interaction patterns through an emotion engine and sends this data to the server. Generally, emotion recognition software such as IBM Watson or Affectiva is used. Based on the analysis results, the server provides the user with information and choices that take their emotions into consideration.

[0558] For example, if a user says, "I'm wondering whether to get rid of this old camera," the server will suggest ways to preserve memories related to the camera and provide examples of effective selling methods to alleviate their concerns. By presenting the user with appropriate information, the server helps them make a decision about disposing of their items with confidence.

[0559] An example of a prompt for a generative AI model is, "When designing a system to manage household items through image analysis, please tell me how to make suggestions that take user emotions into consideration."

[0560] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0561] Step 1:

[0562] The terminal uses a network-connected camera to capture images of objects in a room. This process automatically acquires images based on user instructions or a regular schedule. Input data consists of images of the objects, while output data is an image file transmitted to a server via the network. Specifically, the terminal has the functionality to start capturing images at predetermined times and immediately transfer the image data to the server.

[0563] Step 2:

[0564] The server receives images of items sent from the terminal. The input is image data, and image analysis techniques are used to identify the items. The output is the type and quantity of the items, and their associated attribute information, which is registered in the database. Specifically, the process involves extracting image features using software such as OpenCV, classifying the items using machine learning models, and then registering them.

[0565] Step 3:

[0566] The server monitors the frequency of item use based on information in the database. The input is the item's usage history in the database, and this data is analyzed to evaluate usage frequency. The output is a list of items that need to be disposed of based on usage frequency. The server tracks changes in usage history and has a periodic analysis function to identify items that have not been used for a certain period.

[0567] Step 4:

[0568] The server generates suggestions for disposing of items and prepares to notify the user. The input is a list of identified unwanted items and market information, and the output is a message containing the suggestions. The server gathers information on market value reassessments and recycling options, and proposes specific disposal methods to the user.

[0569] Step 5:

[0570] When a user receives a suggestion, the device monitors the user's emotions. The input consists of data from the user's facial expressions and voice, which is analyzed by an emotion recognition engine. The output is data indicating the user's emotional state, which is used to adjust the suggestion. The device utilizes its camera and microphone to feed back the user's reactions to the server in real time.

[0571] Step 6:

[0572] The server adjusts the suggestions based on the received sentiment analysis data and provides the final suggestions to the user. The input is the user's emotional state and the original suggestions, and the output is an optimized suggestion that takes emotions into account. Specifically, the server has the function to add additional options such as "Does this information alleviate anxiety?" and further explanations.

[0573] (Application Example 2)

[0574] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0575] Traditional inventory management systems could suggest disposal based on the frequency of use of items, but they struggled to provide an interface that considered the user's emotional state. Furthermore, system suggestions often didn't resonate with the user's emotions, resulting in a lack of significant improvement in the user experience. Additionally, there was insufficient support to alleviate anxiety about item disposal and enable users to make informed decisions.

[0576] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0577] In this invention, the server includes means for photographing items in a space using acquisition means, means for observing and recording the usage status and frequency of use of items, and means for analyzing the user's emotional state using an emotion engine. This enables the generation of optimal item disposal suggestions according to the user's emotional state, thereby improving the user experience and reducing anxiety.

[0578] "Acquisition means" refers to a device for photographing objects in a space, including a photographic device connected to a network.

[0579] "Means for analyzing images and identifying objects" refers to a system that processes captured image data to identify objects present within the image.

[0580] "Means for observing and recording the usage status and frequency of use of an item" refers to a system that has the function of monitoring how much an item is being used and storing that data.

[0581] A "means for generating disposal suggestions for items based on usage frequency" is a system that makes suggestions to users about the disposal of items based on recorded usage data.

[0582] "Methods for analyzing a user's emotional state using an emotion engine" refer to technologies that analyze information such as a user's facial expressions and voice to infer what emotions the user is experiencing.

[0583] "Means of adjusting suggested content based on the user's emotional state" refers to a system that changes suggested content and its presentation method based on the user's emotional information analyzed by an emotion engine.

[0584] "Means for carrying out procedures for automatically listing items on the market" refers to a system that automatically executes the process of selling items that have been decided to be disposed of on the market.

[0585] A "means of arranging donations and recycling" refers to a system that has the function of facilitating the disposal of items as donations or for recycling when they are deemed unnecessary.

[0586] "Means of providing user feedback and improving the system" refers to a mechanism for generating feedback based on system usage and user reactions, and for improving system performance.

[0587] The system for implementing this invention has a configuration comprising acquisition means, analysis means, emotion recognition means, and user interaction means. The system first uses a terminal to photograph objects in space. For this purpose, a smartphone or a network-connected imaging device is used as hardware.

[0588] The server receives the captured image data and uses image analysis software to identify each item. Machine learning algorithms are commonly used for item identification, specifically libraries such as TensorFlow and OpenCV. The server also observes the usage status and frequency of the identified items and records this information in a database.

[0589] Based on an analysis of the frequency of use and condition of items, the server generates disposal suggestions. These suggestions are designed to take into account the market value and recyclability of the items being considered for disposal.

[0590] When user interaction occurs, the server uses an emotion engine to analyze the user's facial expressions and voice to identify their emotional state. This allows the server to present information tailored to the user's current emotions. Emotion recognition may utilize libraries such as Python's `emotion_recognition` library.

[0591] For example, when a user is struggling with whether to dispose of an old item, the server uses an emotion engine to detect the user's anxiety and provides additional information to alleviate it. For instance, it might present examples of successful item sales or methods for preserving memories, thereby providing reassurance.

[0592] An example of a prompt that utilizes a generative AI model is, "When disposing of an item, analyze the emotions the user is experiencing and provide information that will give them peace of mind." In this way, the system can provide an item management experience that is sensitive to the user's emotions.

[0593] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0594] Step 1:

[0595] The terminal uses a network-connected camera to photograph objects in the space. It acquires visual information of the objects as input and generates image data as output. This image data is sent to a server to record the status of the objects.

[0596] Step 2:

[0597] The server analyzes image data received from the terminal. Using the acquired image data as input, it identifies objects using a machine learning model (e.g., TensorFlow) and stores the results in a database. The specific operations in this process are image feature extraction and the application of an identification algorithm.

[0598] Step 3:

[0599] The server monitors and analyzes the usage and frequency of items based on records in the database. It takes log data as input, performs data calculations to identify items with low usage frequency, and generates output that prepares disposal suggestions based on the results. Statistical analysis methods are used in this process.

[0600] Step 4:

[0601] Before notifying the user of the generated disposal suggestions, the server uses an emotion engine to analyze the user's emotional state. It receives user facial expressions and voice data as input, applies an emotion recognition algorithm to estimate the user's emotions, and obtains the estimation result as output.

[0602] Step 5:

[0603] The server adjusts the suggestions based on the emotion recognition results. It uses the emotion data received as input to process and modify the suggestions and information presentation methods, and outputs the results. Specific actions include prioritizing suggestions and selecting additional information.

[0604] Step 6:

[0605] When a user receives a proposal, they send a response back to the server. The server receives this response as input, provides feedback, and outputs it as data to improve system performance. Specific examples include whether to accept or reject a disposal proposal and the selection of additional options.

[0606] Step 7:

[0607] Ultimately, the system will be improved by using a generative AI model based on user feedback to create prompts and provide better suggestions. In this process, user feedback data is used as input, the model is retrained, and an optimized suggestion method is obtained as output.

[0608] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0609] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0610] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0611] [Fourth Embodiment]

[0612] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0613] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0614] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0615] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0616] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0617] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0618] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0619] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0620] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0621] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0622] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0623] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0624] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0625] This invention is a system for supporting the efficient management and disposal of items within the home. The system includes acquisition means, analysis functions, and a process that supports decluttering through interaction with the user.

[0626] First, the terminal activates a camera installed in the space to capture images of objects. This device photographs various objects in the home at regular intervals and sends the latest information to a server. This information serves as basic data for object recognition and management.

[0627] Next, the server analyzes the received images and automatically identifies the type, location, and characteristics of the items. For example, it uses AI-based image recognition technology to identify furniture, home appliances, and personal belongings, and registers them in a database.

[0628] The server then monitors and efficiently records the usage and frequency of each item. It integrates multiple camera inputs to analyze how each item is being used. This allows it to quantify how often an item is used over a certain period and profile its usage frequency.

[0629] Furthermore, the server utilizes usage frequency data to generate optimal decluttering suggestions for the user. For example, regarding unused electronic devices, it might notify the user via their device with a specific suggestion such as, "This item hasn't been used in the last six months. Would you like to consider selling or donating it?" This notification includes information on marketplaces where the item can be sold and potential recipients for donation, supporting quick decision-making.

[0630] If the user accepts the proposal, the server accesses the online marketplace's API and automatically processes the listing of the item. If the user chooses to donate or recycle, the server can also coordinate with relevant businesses to smoothly arrange collection and delivery.

[0631] For example, if a user decides to sell an oven toaster they haven't used for six months, the server sends the necessary information to the marketplace to complete the listing and then notifies the user of the status via their device.

[0632] This invention allows for the efficient management of unused items in the home, facilitating the disposal of unwanted items and maximizing their economic value.

[0633] The following describes the processing flow.

[0634] Step 1:

[0635] The device activates its AI camera at set intervals to capture images of objects in the home. The captured images are immediately processed and prepared to extract the contours and features of the objects.

[0636] Step 2:

[0637] The terminal compresses the acquired image data and sends it to the server. The server analyzes the received image data and applies an object recognition algorithm to identify individual items. At this time, the identified items are registered in the database, and the metadata corresponding to each item is updated.

[0638] Step 3:

[0639] The server observes the location and usage of each specific item, recording usage frequency data. For example, if an item is lifted or its position is changed, it is considered used, and the usage frequency is updated.

[0640] Step 4:

[0641] The server extracts items with low usage frequency at regular intervals and generates disposal suggestions for these items. The server uses a generation AI to create disposal suggestion messages based on usage frequency and market information.

[0642] Step 5:

[0643] The user receives disposal suggestions via the terminal and reviews the suggestions. The user considers the options of selling, donating, or recycling the suggested items and enters their selection via the terminal.

[0644] Step 6:

[0645] Based on the user's selection, the server automatically lists the items for sale via the online marketplace's API. If the user chooses donation or recycling, the server handles the necessary procedures with the relevant businesses or organizations.

[0646] Step 7:

[0647] The server notifies the user of the results of the disposal procedure and records that all disposals have been completed. A feedback function collects user opinions and evaluations, which are used to propose future disposals and improve the system.

[0648] (Example 1)

[0649] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0650] In modern households, items tend to accumulate, and their management and disposal are often inefficient. As a result, limited living space is unnecessarily occupied, and their economic value is overlooked. Furthermore, an increase in unused items leads to wasted resources and increased environmental burden. This invention aims to solve these problems by supporting the efficient management of household items, analysis of usage frequency, and appropriate disposal of unwanted items.

[0651] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0652] In this invention, the server includes means for photographing objects in a space using acquisition means, means for analyzing the captured images and identifying the objects, and means for observing and recording the usage status and frequency of use of the objects. This enables detailed analysis of objects in the home and appropriate decluttering suggestions to the user.

[0653] "Acquisition means" refers to a device that has the function of taking an image of an item and transmitting that data to a server.

[0654] A "photography device" is a hardware device installed in a home environment to capture images of objects at specific intervals.

[0655] A "server" is a computer system that analyzes captured image data to identify items, record their usage, and generate disposal suggestions.

[0656] "Item identification" is the process of identifying and classifying each item within a captured image using image analysis technology.

[0657] "Usage status" refers to information about how and how often an item is used.

[0658] "Frequency of use" refers to data that shows how many times a particular item was used within a certain period of time.

[0659] "Disposal suggestions" refer to generating suggestions to encourage users to sell or donate items, based on data such as usage frequency.

[0660] "Donation and reuse" refers to the act of transferring unused items to individuals or organizations in need, or to utilizing them in a way that makes them usable again.

[0661] "Feedback" is the process of collecting and analyzing information based on the results of suggestions and user actions in order to improve the performance and usability of the system.

[0662] This system is designed to efficiently manage household items and assist in the disposal of unwanted items. Specifically, it involves installing network-connected cameras in each space to periodically acquire images of items. For example, in the living room, the camera takes a picture of the entire room at a set time each day, recording past changes.

[0663] The terminal controls the shooting process and sends the captured images to the server in real time. The server receives these images and performs image analysis using a generative AI model. This analysis automatically identifies objects, classifies them into specific categories, and registers them in a database. Software used for this includes image analysis tools such as TensorFlow and OpenCV.

[0664] The server also monitors the usage status and frequency of items in real time and stores this data. Based on this usage frequency data, it generates decluttering suggestions for unused items and notifies the user of specific suggestions via their device. For example, a suggestion might be, "This mixer hasn't been used in the last six months. Would you like to consider selling or donating it?"

[0665] If a user accepts a proposal, the server automatically calls the online marketplace API to list the item for sale. If the user chooses to donate or reuse the item, the server coordinates with relevant services to smoothly arrange collection and delivery. This entire process streamlines the management and disposal of items without requiring any user intervention.

[0666] Prompt statements as concrete examples:

[0667] "List items that haven't been used in the past six months and suggest decluttering options to users. Example: This guitar hasn't been used for six months, so please suggest where to sell or donate it."

[0668] This system provides a practical solution for effectively utilizing space within the home and maximizing its economic value.

[0669] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0670] Step 1:

[0671] The terminal controls a camera installed in the home and sets a schedule for periodically photographing items in each room. Inputs are the current date and time, and a pre-set shooting schedule. Output is image data of the photographed items, which is transmitted to the server in real time.

[0672] Step 2:

[0673] The server analyzes the received images using AI-based image recognition software. The input is image data sent from the terminal. The server utilizes a generative AI model to identify individual items through image analysis and classify them based on their characteristics. The output obtained from this process is category information for the items.

[0674] Step 3:

[0675] The server registers the image analysis results in a database and observes the usage status and frequency of items. The input consists of item category information obtained from image analysis and past usage data. Based on this information, the server calculates the usage frequency of each item and records it as usage history data. This output data is later used to generate decluttering suggestions.

[0676] Step 4:

[0677] The server analyzes usage frequency data for items and generates decluttering suggestions for items deemed unnecessary. The input is usage and frequency data. The server uses a generative AI model to refine the suggestions presented to the user. The output is a specific suggestion statement, such as, "This item hasn't been used in the last six months. Would you consider selling or donating it?"

[0678] Step 5:

[0679] The terminal receives a proposal notification from the server and displays it to the user. The input is the proposal text sent from the server. The user reviews the proposal and makes a decision from the options of selling, donating, or recycling. The output is the option selected by the user.

[0680] Step 6:

[0681] The server performs the necessary procedures based on the user's selection. The input is the user's selection information. The server calls the marketplace API to either automatically list the item or contact the charity to arrange delivery. After this procedure is complete, the status is notified to the terminal, and the user is provided with final feedback. The output is the result of the procedures performed.

[0682] (Application Example 1)

[0683] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] In modern households, the number of items continues to increase, making management and proper disposal difficult. In particular, the process of regularly reviewing belongings and efficiently disposing of unnecessary items is time-consuming and laborious, often leading to it being postponed in many households. Furthermore, environmental considerations are necessary when disposing of items; options such as reuse and donation should be considered, rather than simply throwing things away. A system is needed to address these challenges and support the efficient management and disposal of household items.

[0685] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0686] In this invention, the server includes means for an autonomous mobile device to patrol the environment and collect item data, means for generating disposal suggestions for items based on usage frequency, and means for performing procedures for automatically listing items on the market. This makes it possible to keep household items constantly up-to-date, propose effective disposal methods for unused items, and automate the procedures.

[0687] "Acquisition means" refers to devices or methods for acquiring images of objects present in a space.

[0688] "Photography equipment" refers to devices such as cameras and sensors used to capture images of objects.

[0689] "Image analysis" refers to the process of identifying objects from acquired image data and determining their characteristics and type.

[0690] "Item usage status" refers to information indicating how a particular item is being used or not being used.

[0691] "Frequency of use" refers to numerical data indicating how many times an item was used within a specific period.

[0692] A "disposal suggestion" refers to a proposal to the user on how to dispose of a specific item based on its frequency of use or other conditions.

[0693] "Automatic listing" refers to the process of automatically listing products on the marketplace.

[0694] "Donation and reuse" refers to providing items to new users or allowing them to be reused, rather than simply discarding them.

[0695] An "autonomous mobile device" refers to a robot or machine that has the function of automatically patrolling a home and collecting necessary data.

[0696] "Feedback" refers to information that reflects the data and results obtained by the system or users, and is used to help with the next steps and improvements.

[0697] The system implementing this invention is designed to effectively support the management and disposal of items within the home. The system operates by integrating multiple hardware and software components.

[0698] First, an autonomous mobile device patrols the home, acquiring images of all objects. The autonomous mobile device is equipped with high-performance cameras and sensors connected to a communication network, accurately capturing the location and characteristics of objects within the space. The captured image data is transmitted via the communication network to a server in the cloud.

[0699] The server processes the received image data using AI-based image analysis technology to identify the type, location, and characteristics of items. It uses image analysis APIs such as the Google Cloud Vision API to accurately identify items. The identified information is registered in a database, and the frequency of use of each item is recorded.

[0700] Furthermore, the server generates disposal suggestions based on usage frequency. These suggestions include recommendations for selling or donating unused items, and also provide disposal methods and market information. Notifications are sent to the user's smartphone or device in real time using a notification system like Amazon SNS. If the user accepts a suggestion, the server automatically executes the listing process by integrating with market APIs or initiates the donation process.

[0701] As a concrete example, the system detects an oven toaster that hasn't been used for six months and sends a notification to the user saying, "This oven toaster hasn't been used for six months. Would you like to consider selling or donating it?" Upon receiving this notification, the user chooses to sell it via their device, and that information is instantly sent to an online marketplace.

[0702] By utilizing generative AI models, it is possible to derive more sophisticated suggestion models and solutions from prompts such as, "Please tell me how to design an AI assistant that identifies items infrequently used in the home and suggests selling or donating them."

[0703] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0704] Step 1:

[0705] An autonomous mobile device patrols the home, acquiring images of objects with its onboard high-performance camera. This image data, including the location and characteristics of the objects, is transmitted to a server via a communication network. The input is the raw image data acquired by the camera, and the output is the image data transmitted to the server.

[0706] Step 2:

[0707] The server receives image data and performs AI-based image analysis. The input here is image data, and the type and characteristics of the objects are identified using the Google Cloud Vision API. The output is information data about the identified objects, which is registered in the database.

[0708] Step 3:

[0709] The server observes and records usage status and frequency based on the information of registered items. This data analysis takes the past usage history of each item as input and outputs data that profiles the frequency of use of each item.

[0710] Step 4:

[0711] The server generates disposal suggestions for items based on usage frequency data. The input is numerical data regarding usage frequency, and the output is a specific disposal suggestion that is communicated to the user. For example, it might generate recommendations to sell or donate an unused toaster oven.

[0712] Step 5:

[0713] The terminal notifies the user in real time of disposal suggestions from the server. The input is the content of the suggestion sent from the server, and the output is the information delivered to the user via the terminal's screen or audio.

[0714] Step 6:

[0715] If the user accepts the proposal, the server uses the marketplace API to initiate the automated listing process for the item. The input here is the user's selection information, and the output is the status of the item's listing on the marketplace.

[0716] Step 7:

[0717] The server also provides a means to arrange donations and reuse, and facilitates appropriate coordination according to the user's wishes. The input is the user's choice of disposal method, and the output is the status of the procedures with the donation or reuse recipient.

[0718] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0719] This invention adds an emotion engine to a system designed to improve the efficiency of item management, thereby providing optimal suggestions and interfaces that take into account the user's emotional state. This system includes a process for effectively disposing of household items through acquisition means, analysis functions, emotion recognition, and user interaction.

[0720] First, the terminal uses a network-connected camera to capture images of items in the room. This image data is sent to a server to continuously monitor the condition of the items. The server uses image analysis technology to identify each item and register it in a database.

[0721] The server monitors the frequency and condition of item usage, identifying items that are not being used. If an item is used infrequently, the server generates a suggestion and prepares a notification for the user. This suggestion takes into account the item's market value and recyclability.

[0722] The key here is the emotion engine built into the server. When a user receives a suggestion using their device, the emotion engine analyzes the user's facial expressions, voice, and interaction patterns to estimate the user's current emotional state. The results of the emotion recognition are used to adjust how the suggestion is presented. For example, if a user shows anxiety about a disposal suggestion, the emotion engine detects this, and the server provides the user with additional information or options to alleviate the anxiety.

[0723] For example, if a user expresses an emotion such as "I'm wondering whether to get rid of this old camera," the emotion engine detects this through the device. The server then provides additional reassuring information to help the user make a decision, such as ways to preserve cherished photos from the camera or examples of successful sales of similar items.

[0724] Thus, by introducing an emotion engine, the present invention can go beyond mere item management and provide an experience that is attuned to the user's emotions. As a result, users can efficiently manage and dispose of items without emotional burden.

[0725] The following describes the processing flow.

[0726] Step 1:

[0727] The terminal periodically activates a network-connected camera to capture images of objects in the space. The captured images are optimized to extract the features of the objects and then sent to a server.

[0728] Step 2:

[0729] The server analyzes the received image data and uses machine learning algorithms to identify items. During this process, it recognizes the unique characteristics of each item and registers them in a database. Item classifications and historical data are also associated with these items.

[0730] Step 3:

[0731] The server continuously monitors the acquired data to record the frequency and usage status of each item. It also tracks the movement and actions of items and updates usage information accordingly.

[0732] Step 4:

[0733] The server, having identified infrequently used items, automatically generates disposal suggestions. These suggestions include information such as the item's market value, recyclability, and where it can be donated. The suggestions are then prepared to be notified to the user at an appropriate time.

[0734] Step 5:

[0735] The emotion engine integrated into the server activates when the user receives a disposal suggestion. The terminal collects emotional data from the user's facial expressions and voice and sends it to the server in real time. The server uses the emotion engine to analyze the user's emotional state and make any necessary adjustments.

[0736] Step 6:

[0737] Users can review disposal suggestions on their devices and make selections based on their own emotions. The suggestions are customized based on the emotions engine's judgment, and additional information is added to alleviate the user's anxiety and hesitation.

[0738] Step 7:

[0739] After the user selects a disposal method for their items, the server accesses online marketplaces and automatically lists the items. If donation or recycling is selected, the server notifies the relevant businesses and initiates the necessary procedures. The server then verifies that the disposal is complete and provides feedback to the user.

[0740] (Example 2)

[0741] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0742] Managing belongings in modern homes and offices is becoming increasingly complex due to the sheer number and diversity of items. In particular, traditional systems are insufficient to efficiently identify unwanted items, select appropriate disposal methods, and support decision-making that considers user emotions. Therefore, there is a need for a system that not only improves the efficiency of belongings management but also reduces the psychological burden on users.

[0743] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0744] In this invention, the server includes means for photographing objects in a space using an acquisition device, means for analyzing the captured images and identifying the objects, and means for analyzing the user's emotional state and adjusting the suggested content. This enables efficient management of objects and improved accuracy of disposal suggestions, and further reduces psychological burden by providing suggestions that are sensitive to the user's emotions.

[0745] "Acquisition device" is a general term for equipment and systems used to photograph objects in a space, and mainly refers to cameras, smartphones, etc.

[0746] A "communication network" is an infrastructure for transmitting digital data, and includes the internet and wireless LANs.

[0747] "Image analysis" is a technology that processes captured image data to identify objects, and includes processing using artificial intelligence technology.

[0748] "Artificial intelligence technology" refers to technologies that enable computer systems to perform perceptions and judgments similar to those of humans, and includes machine learning and deep learning.

[0749] "Emotional state" refers to the psychological or emotional responses exhibited by the user, and is information analyzed from facial expressions and tone of voice.

[0750] "Adjusting the proposal" refers to the process of appropriately changing the details and presentation method of the proposal regarding the disposal of items, taking into account the user's emotional state.

[0751] This invention provides a system for streamlining inventory management and making suggestions that take user emotions into consideration. The system operates using acquisition devices, communication networks, image analysis technology, artificial intelligence technology, and the like.

[0752] The terminal utilizes a network-connected camera or smartphone as an acquisition device to photograph objects in a room. This device periodically or based on user instructions acquires images of objects and transmits the data to a server. Wi-Fi or Bluetooth are commonly used as the communication network.

[0753] The server analyzes the received image data using artificial intelligence technology. For image analysis, software such as OpenCV is used to identify objects based on their shape, color, and size. The analysis results are registered in a database, and the attribute information of the objects is updated.

[0754] Furthermore, the server analyzes the user's emotional state and provides appropriate suggestions. When the user receives a suggestion, the terminal monitors facial expressions, voice, and interaction patterns through an emotion engine and sends this data to the server. Generally, emotion recognition software such as IBM Watson or Affectiva is used. Based on the analysis results, the server provides the user with information and choices that take their emotions into consideration.

[0755] For example, if a user says, "I'm wondering whether to get rid of this old camera," the server will suggest ways to preserve memories related to the camera and provide examples of effective selling methods to alleviate their concerns. By presenting the user with appropriate information, the server helps them make a decision about disposing of their items with confidence.

[0756] An example of a prompt for a generative AI model is, "When designing a system to manage household items through image analysis, please tell me how to make suggestions that take user emotions into consideration."

[0757] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0758] Step 1:

[0759] The terminal uses a network-connected camera to capture images of objects in a room. This process automatically acquires images based on user instructions or a regular schedule. Input data consists of images of the objects, while output data is an image file transmitted to a server via the network. Specifically, the terminal has the functionality to start capturing images at predetermined times and immediately transfer the image data to the server.

[0760] Step 2:

[0761] The server receives images of items sent from the terminal. The input is image data, and image analysis techniques are used to identify the items. The output is the type and quantity of the items, and their associated attribute information, which is registered in the database. Specifically, the process involves extracting image features using software such as OpenCV, classifying the items using machine learning models, and then registering them.

[0762] Step 3:

[0763] The server monitors the frequency of item use based on information in the database. The input is the item's usage history in the database, and this data is analyzed to evaluate usage frequency. The output is a list of items that need to be disposed of based on usage frequency. The server tracks changes in usage history and has a periodic analysis function to identify items that have not been used for a certain period.

[0764] Step 4:

[0765] The server generates suggestions for disposing of items and prepares to notify the user. The input is a list of identified unwanted items and market information, and the output is a message containing the suggestions. The server gathers information on market value reassessments and recycling options, and proposes specific disposal methods to the user.

[0766] Step 5:

[0767] When a user receives a suggestion, the device monitors the user's emotions. The input consists of data from the user's facial expressions and voice, which is analyzed by an emotion recognition engine. The output is data indicating the user's emotional state, which is used to adjust the suggestion. The device utilizes its camera and microphone to feed back the user's reactions to the server in real time.

[0768] Step 6:

[0769] The server adjusts the suggestions based on the received sentiment analysis data and provides the final suggestions to the user. The input is the user's emotional state and the original suggestions, and the output is an optimized suggestion that takes emotions into account. Specifically, the server has the function to add additional options such as "Does this information alleviate anxiety?" and further explanations.

[0770] (Application Example 2)

[0771] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0772] Traditional inventory management systems could suggest disposal based on the frequency of use of items, but they struggled to provide an interface that considered the user's emotional state. Furthermore, system suggestions often didn't resonate with the user's emotions, resulting in a lack of significant improvement in the user experience. Additionally, there was insufficient support to alleviate anxiety about item disposal and enable users to make informed decisions.

[0773] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0774] In this invention, the server includes means for photographing items in a space using acquisition means, means for observing and recording the usage status and frequency of use of items, and means for analyzing the user's emotional state using an emotion engine. This enables the generation of optimal item disposal suggestions according to the user's emotional state, thereby improving the user experience and reducing anxiety.

[0775] "Acquisition means" refers to a device for photographing objects in a space, including a photographic device connected to a network.

[0776] "Means for analyzing images and identifying objects" refers to a system that processes captured image data to identify objects present within the image.

[0777] "Means for observing and recording the usage status and frequency of use of an item" refers to a system that has the function of monitoring how much an item is being used and storing that data.

[0778] A "means for generating disposal suggestions for items based on usage frequency" is a system that makes suggestions to users about the disposal of items based on recorded usage data.

[0779] "Methods for analyzing a user's emotional state using an emotion engine" refer to technologies that analyze information such as a user's facial expressions and voice to infer what emotions the user is experiencing.

[0780] "Means of adjusting suggested content based on the user's emotional state" refers to a system that changes suggested content and its presentation method based on the user's emotional information analyzed by an emotion engine.

[0781] "Means for carrying out procedures for automatically listing items on the market" refers to a system that automatically executes the process of selling items that have been decided to be disposed of on the market.

[0782] A "means of arranging donations and recycling" refers to a system that has the function of facilitating the disposal of items as donations or for recycling when they are deemed unnecessary.

[0783] "Means of providing user feedback and improving the system" refers to a mechanism for generating feedback based on system usage and user reactions, and for improving system performance.

[0784] The system for implementing this invention has a configuration comprising acquisition means, analysis means, emotion recognition means, and user interaction means. The system first uses a terminal to photograph objects in space. For this purpose, a smartphone or a network-connected imaging device is used as hardware.

[0785] The server receives the captured image data and uses image analysis software to identify each item. Machine learning algorithms are commonly used for item identification, specifically libraries such as TensorFlow and OpenCV. The server also observes the usage status and frequency of the identified items and records this information in a database.

[0786] Based on an analysis of the frequency of use and condition of items, the server generates disposal suggestions. These suggestions are designed to take into account the market value and recyclability of the items being considered for disposal.

[0787] When user interaction occurs, the server uses an emotion engine to analyze the user's facial expressions and voice to identify their emotional state. This allows the server to present information tailored to the user's current emotions. Emotion recognition may utilize libraries such as Python's `emotion_recognition` library.

[0788] For example, when a user is struggling with whether to dispose of an old item, the server uses an emotion engine to detect the user's anxiety and provides additional information to alleviate it. For instance, it might present examples of successful item sales or methods for preserving memories, thereby providing reassurance.

[0789] An example of a prompt that utilizes a generative AI model is, "When disposing of an item, analyze the emotions the user is experiencing and provide information that will give them peace of mind." In this way, the system can provide an item management experience that is sensitive to the user's emotions.

[0790] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0791] Step 1:

[0792] The terminal uses a network-connected camera to photograph objects in the space. It acquires visual information of the objects as input and generates image data as output. This image data is sent to a server to record the status of the objects.

[0793] Step 2:

[0794] The server analyzes image data received from the terminal. Using the acquired image data as input, it identifies objects using a machine learning model (e.g., TensorFlow) and stores the results in a database. The specific operations in this process are image feature extraction and the application of an identification algorithm.

[0795] Step 3:

[0796] The server monitors and analyzes the usage and frequency of items based on records in the database. It takes log data as input, performs data calculations to identify items with low usage frequency, and generates output that prepares disposal suggestions based on the results. Statistical analysis methods are used in this process.

[0797] Step 4:

[0798] Before notifying the user of the generated disposal suggestions, the server uses an emotion engine to analyze the user's emotional state. It receives user facial expressions and voice data as input, applies an emotion recognition algorithm to estimate the user's emotions, and obtains the estimation result as output.

[0799] Step 5:

[0800] The server adjusts the suggestions based on the emotion recognition results. It uses the emotion data received as input to process and modify the suggestions and information presentation methods, and outputs the results. Specific actions include prioritizing suggestions and selecting additional information.

[0801] Step 6:

[0802] When a user receives a proposal, they send a response back to the server. The server receives this response as input, provides feedback, and outputs it as data to improve system performance. Specific examples include whether to accept or reject a disposal proposal and the selection of additional options.

[0803] Step 7:

[0804] Ultimately, the system will be improved by using a generative AI model based on user feedback to create prompts and provide better suggestions. In this process, user feedback data is used as input, the model is retrained, and an optimized suggestion method is obtained as output.

[0805] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0806] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0807] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0808] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0809] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0810] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0811] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0812] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0813] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0814] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0815] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0816] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0817] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0818] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0819] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0820] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0821] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0822] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0823] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0824] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0825] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0826] The following is further disclosed regarding the embodiments described above.

[0827] (Claim 1)

[0828] A means for photographing objects in a space using an acquisition means,

[0829] A means for analyzing captured images and identifying items,

[0830] Means for observing and recording the usage status and frequency of use of items,

[0831] A means for generating disposal suggestions for items based on frequency of use,

[0832] A means of carrying out procedures for automatically listing goods on the market,

[0833] Methods of disposal include arranging donation or recycling,

[0834] A means of providing user feedback and improving the system,

[0835] A system that includes this.

[0836] (Claim 2)

[0837] The acquisition means includes a network-connected imaging device, as described in claim 1.

[0838] (Claim 3)

[0839] The system according to claim 1, characterized in that machine learning is used for identifying items.

[0840] "Example 1"

[0841] (Claim 1)

[0842] A means for photographing objects in a space using an acquisition means,

[0843] A means for analyzing captured images and identifying items,

[0844] Means for observing and recording the usage status and frequency of use of items,

[0845] A means for generating disposal suggestions for items based on frequency of use,

[0846] A means of carrying out procedures for automatically listing goods on the market,

[0847] Methods of disposal include arranging donation or reuse,

[0848] A means of presenting users with options to help them make decisions,

[0849] A means of automatically executing the relevant procedures upon acceptance of the proposal,

[0850] A means of providing user feedback and improving the system,

[0851] A system that includes this.

[0852] (Claim 2)

[0853] The acquisition means includes a network-connected imaging device, as described in claim 1.

[0854] (Claim 3)

[0855] The system according to claim 1, characterized in that a generative AI model is used for identifying articles.

[0856] "Application Example 1"

[0857] (Claim 1)

[0858] A means for photographing objects in a space using an acquisition means,

[0859] A means for analyzing captured images and identifying items,

[0860] Means for observing and recording the usage status and frequency of use of items,

[0861] A means for generating disposal suggestions for items based on frequency of use,

[0862] A means of carrying out procedures for automatically listing goods on the market,

[0863] Methods of disposal include arranging donation or reuse,

[0864] A means by which an autonomous mobile device patrols the environment and collects item data,

[0865] A means of providing user feedback and improving the system,

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The acquisition means includes a photographic device connected to a communication network, according to claim 1.

[0869] (Claim 3)

[0870] The system according to claim 1, characterized in that machine learning is used for identifying items.

[0871] "Example 2 of combining an emotion engine"

[0872] (Claim 1)

[0873] A means of photographing objects in a space using an acquisition device,

[0874] A means for analyzing captured images and identifying items,

[0875] Means for observing and recording the usage status and frequency of use of items,

[0876] A means for generating disposal suggestions for items based on frequency of use,

[0877] A means of notifying users of proposed disposals,

[0878] A means of analyzing the user's emotional state and adjusting the suggested content,

[0879] A means of carrying out procedures for automatically listing goods on the market,

[0880] Methods of disposal include arranging donation or reuse,

[0881] A means of providing user feedback and improving the system,

[0882] A system that includes this.

[0883] (Claim 2)

[0884] The acquisition device includes a camera connected to a communication network, according to claim 1.

[0885] (Claim 3)

[0886] The system according to claim 1, characterized in that artificial intelligence technology is used for identifying articles.

[0887] "Application example 2 when combining with an emotional engine"

[0888] (Claim 1)

[0889] A means for photographing objects in a space using an acquisition means,

[0890] A means for analyzing captured images and identifying items,

[0891] Means for observing and recording the usage status and frequency of use of items,

[0892] A means for generating disposal suggestions for items based on frequency of use,

[0893] A means of analyzing a user's emotional state using an emotion engine,

[0894] A means of adjusting the content of suggestions based on the user's emotional state,

[0895] A means of carrying out procedures for automatically listing goods on the market,

[0896] Methods of disposal include arranging donation or recycling,

[0897] A means of providing user feedback and improving the system,

[0898] A system that includes this.

[0899] (Claim 2)

[0900] The acquisition means includes a network-connected imaging device, as described in claim 1.

[0901] (Claim 3)

[0902] The system according to claim 1, characterized in that machine learning is used for identifying items. [Explanation of Symbols]

[0903] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for photographing objects in a space using an acquisition means, A means for analyzing captured images and identifying items, Means for observing and recording the usage status and frequency of use of items, A means for generating disposal suggestions for items based on frequency of use, A means of carrying out procedures for automatically listing goods on the market, Methods of disposal include arranging donation or recycling, A means of providing user feedback and improving the system, A system that includes this.

2. The acquisition means includes a network-connected imaging device, as described in claim 1.

3. The system according to claim 1, characterized in that machine learning is used for identifying items.

Images