system

Abstract

We provide the system. [Solution] Information gathering means for acquiring images, Information analysis means for analyzing acquired images to recognize the target, A means for estimating the frequency of use of the target, A proposal generation means that proposes the organization or disposal of an object based on its frequency of use, A processing means for disposing of waste based on the proposal, The processing means is a presentation means that presents the proposed content using a household autonomous device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the home, many items are left unused, resulting in waste of space and financial losses. With conventional methods, it is difficult to grasp the usage frequency and efficiently organize or dispose of them, which is a great burden for users. Furthermore, no method has been provided for automatically managing and disposing of items while protecting privacy efficiently.

Means for Solving the Problems

[0005] This invention provides an information analysis means that monitors the state of items in a room using an information acquisition means to acquire images, and recognizes objects by analyzing the acquired images. Next, it uses a usage frequency estimation means to estimate the frequency of use of objects and understand the usage status of those items. Then, it uses a suggestion generation means that proposes organizing or disposing of the items based on their usage frequency to present the user with an efficient decluttering plan. Furthermore, it includes a processing means for processing unwanted items based on the suggestions, and by using a listing means that automatically lists unwanted items on an online platform, and a cooperation means that works with collection companies or donation recipients to dispose of them, it realizes a comfortable living space while reducing the burden on the user.

[0006] "Information acquisition means" refers to a device or method for acquiring the condition of items in a room in the form of images or other formats.

[0007] "Information analysis means" refers to a device or method for analyzing acquired image data to recognize objects within the image and extract their features.

[0008] "Usage frequency estimation means" refers to a device or method for estimating the usage frequency of a recognized object based on its usage history, exposure time, etc.

[0009] "Suggestion generation means" refers to a device or method for generating and presenting suggestions regarding the organization or disposal of items to a user, based on estimated usage frequency data.

[0010] "Processing means" refers to an apparatus or method for actually processing unwanted items based on the proposal, and specifically includes the automated listing and disposal arrangement of unwanted items.

[0011] "Listing method" refers to a device or method for automatically listing proposed unwanted items on an online platform and facilitating the sales process.

[0012] "Cooperative means" refers to a device or method for efficiently processing unwanted items by coordinating with collection companies or donation recipients. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a signed storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention provides a system for monitoring the usage of items within a household and efficiently decluttering. This system comprises information acquisition means, information analysis means, usage frequency estimation means, suggestion generation means, and processing means. A specific example of a system using these means is shown below.

[0035] First, the server periodically acquires images using an information acquisition device, namely an AI camera, installed in the room. The server receives this image data and uses an information analysis device to identify and recognize objects within the images. At this time, features such as the type and shape of the identified objects are also extracted and recorded in a database.

[0036] Next, the server uses a usage frequency estimation mechanism to calculate the usage frequency of each object based on the information recorded in the database. Based on this information, the server decides which items should be decluttered.

[0037] Based on the results, the server uses a suggestion generation mechanism to generate the optimal decluttering plan for the user. The server notifies the user of this suggestion via the terminal, and the user can review and modify the suggestion. For example, the server might present the user with the suggestion, "These clothes may not have been worn in the past six months. Shall we declutter them?"

[0038] Furthermore, if the user approves the proposal, the server automatically lists the unwanted items on the online platform using processing tools. The server also supports the process of arranging for the items to be sent to a recycling company or donation recipient by utilizing various integration tools.

[0039] In this way, the present invention enables efficient organization and disposal of items, making it easier to maintain a comfortable living space. Users can practice decluttering in their daily lives without feeling burdened.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The server controls AI cameras, which are information acquisition tools installed in the room, and acquires images at specific time intervals. The server receives these image data in real time and stores them in a database.

[0043] Step 2:

[0044] The server analyzes the received image data using information analysis tools. It recognizes objects within the image and identifies the type and location of each object. The server analyzes this recognition information and records the characteristics of the objects in a database.

[0045] Step 3:

[0046] The server operates a usage frequency estimation system and calculates the current usage frequency based on the previously recorded usage history of the object. Specifically, it evaluates whether the object is being used by taking into account changes in the object's position and exposure time. The results are reflected in the database.

[0047] Step 4:

[0048] The server uses a suggestion generation mechanism to generate a decluttering plan for the user based on usage frequency data. The terminal displays this plan and sends a notification, such as "This item hasn't been used recently, so we suggest disposing of it." The terminal accepts user confirmation or modifications.

[0049] Step 5:

[0050] After receiving user approval, the server uses processing tools to handle the disposal of unwanted items. Specifically, a listing process is initiated to automatically list the unwanted items on an online platform.

[0051] Step 6:

[0052] Furthermore, using collaborative processing methods, the system can arrange for unwanted items to be sent to recycling companies or donation recipients if the user so desires. The server manages these collaborations with external services, supporting the efficient disposal of unwanted items.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] In modern living environments, a large number of items accumulate in homes, posing a challenge in organizing and disposing of them. This can lead to cramped living spaces and a loss of comfort. Furthermore, the effort required to choose the optimal disposal method for unwanted items makes item management inefficient. Therefore, there is a need for efficient methods to manage and appropriately dispose of items accumulated in homes.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes: an image acquisition means that periodically acquires images of items using an information acquisition device installed in the room to acquire images; an information analysis means that analyzes the acquired image data using computing resources on the cloud to recognize the items; a usage frequency estimation means that records the type and characteristics of the recognized items and estimates the frequency of use of the items based on that data; a proposal generation means that uses a generation AI model to propose organizing or disposing of the items based on the estimated usage frequency; and a processing means that processes unwanted items after receiving user confirmation based on the proposal. This enables efficient management of items in living spaces and prompt and appropriate disposal.

[0058] An "information acquisition device" is a device installed in a room to acquire images of a target object.

[0059] "Image acquisition means" refers to a function that periodically acquires images of an object via an information acquisition device.

[0060] "Information analysis means" refers to a function that uses computing resources on the cloud to analyze acquired image data and recognize objects.

[0061] "Usage frequency estimation means" refers to a function that records the type and characteristics of recognized items as data and estimates the frequency of use of those items based on that data.

[0062] The "proposal generation means" is a function that uses a generation AI model to suggest to the user how to organize or dispose of items based on the estimated frequency of use.

[0063] "Processing means" refers to a function that processes unwanted items based on proposals presented by the proposal generation means, after receiving user confirmation.

[0064] The "listing function" is a feature that, after user approval, automatically lists unwanted items on an online exchange platform.

[0065] "Collaboration function" refers to a function that allows the processing system to dispose of unwanted items in cooperation with collection companies or donation recipients.

[0066] This invention is a system that supports the management, efficient organization, and disposal of household items. The server periodically acquires images of items using an information acquisition device installed in the room. Specifically, a general-purpose camera module is used as hardware, and the captured image data is transmitted to computing resources in the cloud.

[0067] The server sends this image data to an information analysis tool in the cloud, where it performs image analysis using common machine learning libraries, including GOOGLE TENSOR® and FLOW®. As a result, objects are recognized, and features such as the type and shape of the recognized objects are extracted. This information is recorded in a database, and database services such as Amazon RDS can be used.

[0068] Next, the server uses a usage frequency estimation mechanism to estimate the frequency of use of items based on the recorded information. This process uses the Python Pandas library to analyze item usage patterns based on past analysis data.

[0069] Regarding the suggestion generation method, a generative AI model provided by OpenAI® is used to suggest decluttering to the user based on acquired usage frequency data. For example, it generates a natural language prompt such as, "This cup hasn't been used in the last three months. Please consider decluttering it."

[0070] Users can receive and confirm suggestions via their devices. Based on feedback from the devices, the server executes the procedures for processing unwanted items. Using the processing methods, after user approval, unwanted items are automatically listed on online exchange platforms or a disposal plan is created. Coordination with recycling companies and donation recipients is also considered, and efficient communication and arrangements are made via the network.

[0071] A concrete example of a prompt message for a generative AI model would be: "Create a phrase that reports examples of items in the user's living space that haven't been used recently, and asks if the user is willing to organize those items."

[0072] This system allows users to efficiently manage their belongings and maintain a comfortable living space.

[0073] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0074] Step 1:

[0075] The server periodically acquires images of items using an information acquisition device (AI camera) installed in the room. The input is real-time image data acquired by the camera, and the output is an image file, such as JPEG, sent to the server for analysis. This image acquisition is performed automatically at a specified time each day.

[0076] Step 2:

[0077] The server sends image data to the cloud. Here, the input image data is transferred to the cloud analysis service using a secure protocol. The output is image data waiting on the cloud. Specifically, the image is securely transmitted using SSL / TLS and added to the analysis queue.

[0078] Step 3:

[0079] The server uses cloud-based information analysis tools to analyze image data and recognize objects. The input is the image data acquired in step 2, and the output is the analysis results, including the type and characteristics of the recognized objects. This process uses a machine learning library to label the types of objects. Specifically, objects are classified as "books," "clothing," "dishes," etc., and their characteristics such as shape and color are extracted.

[0080] Step 4:

[0081] The server records the recognized item data in the database. The input is the item analysis result generated in step 3, and the output is the updated database entry. Here, the ID, type, characteristics, and date / time information of each item are stored in Amazon RDS. Specifically, data is inserted using SQL queries.

[0082] Step 5:

[0083] The server uses a usage frequency estimation method to analyze recorded data and calculate the usage frequency of items. The input is historical item usage data stored in a database, and the output is an estimated usage frequency for each item. Using the Pandas library, the usage history of each item is analyzed to derive conclusions such as "This shirt has been used 3 times in the past 30 days."

[0084] Step 6:

[0085] The server uses an AI model to generate suggestions for the user based on item usage frequency data. The input is the estimated usage frequency, and the output is suggestions for decluttering. For example, it might generate a prompt such as, "This jacket has hardly been used in the last 6 months. Please consider getting rid of it."

[0086] Step 7:

[0087] The server generates a proposal and notifies the user's device. The input is the proposal content generated in step 6, and the output is a push notification to the user's device. The user can review and approve the proposal through the push notification. This operation is performed using the mobile app notification service.

[0088] Step 8:

[0089] If the user approves the proposal, the server processes the unwanted items. Specifically, the input is the user's approval, and the output is the procedure for listing the unwanted items on an online exchange platform or handing them over to a recycling company. At this point, product information is registered and listed via the online platform API, and the process is completed.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] The challenge lies in addressing the difficulty of efficiently utilizing living space, which arises from the increasing number of items in the home and the complexity of managing them. In particular, the lack of identification of unnecessary items and a plan for their disposal is a factor that reduces the quality of life.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes information gathering means, information analysis means, usage frequency estimation means, suggestion generation means, processing means, and presentation means. This enables effective organization and waste-free disposal of household items.

[0095] "Information gathering means" refers to devices or functions that acquire images, which are installed in home autonomous devices to understand the situation of the target object.

[0096] "Information analysis means" refers to techniques and methods for processing acquired image data and identifying specific objects.

[0097] A "usage frequency estimation means" is a method or device for analyzing the handling and usage conditions of an object and calculating its usage frequency.

[0098] The "proposal generation method" is a function that creates recommendations for organizing and disposing of items based on usage frequency data.

[0099] "Processing means" refers to the means of implementation for carrying out specific disposal or management actions based on the proposal.

[0100] "Presentation means" refers to a function that uses a home-use autonomous device to display or notify the user of suggested content.

[0101] An "online trading platform" is a general term for places and services where goods are bought and sold via the internet.

[0102] "Cooperative methods" refer to means of efficiently disposing of unnecessary items in cooperation with collection companies and donation recipients.

[0103] To realize this invention, the following configuration and technology are necessary for the autonomous home device. The server is equipped with an AI camera as an information gathering means and periodically photographs items in the home. This acquired image data is transmitted to the server and then analyzed using image analysis software such as TensorFlow as an information analysis means. This makes it possible to identify and distinguish objects.

[0104] The server calculates the frequency of use of the analyzed objects using a usage frequency estimation means. Based on the usage frequency data, a suggestion generation means operates to determine which items should be sorted or disposed of, and proposes the results to the user. The suggestion to the user is presented by a presentation means provided by the home autonomous device. This presentation means can also notify the user via a terminal such as a smartphone or tablet.

[0105] (Specific example)

[0106] For example, the server might notify the user's device with a suggestion such as, "These dishes appear to have not been used in the past year. Would you like to consider disposing of them or donating them?" If the user approves this suggestion, the unwanted items are automatically listed on an online trading platform or, if necessary, contacted by a collection agency or other relevant party through a collaborative means. An example of a specific prompt might be, "Identify items that have not been used in the past six months and generate a decluttering suggestion." This system not only allows users to efficiently organize their living spaces but also helps them clearly distinguish between what they need and what they don't.

[0107] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0108] Step 1:

[0109] The server uses an AI camera as a means of information gathering to periodically acquire images of items in the home. The AI ​​camera provides images of the home environment as input. These images are output directly as digital data and sent to the server.

[0110] Step 2:

[0111] The server processes the received image data using image analysis software such as TensorFlow. Image data is sent to the server as input, and information for identifying objects (such as their shape and color) is extracted from this data. The extracted information is then recorded in a database.

[0112] Step 3:

[0113] The server calculates the usage frequency of each item using a usage frequency estimation means, based on data obtained by the information analysis means. It uses previously recorded item characteristic data as input and performs calculations using a usage frequency model. As a result of the calculation, it outputs usage frequency information for each item and passes it to the proposal generation means.

[0114] Step 4:

[0115] The server uses a suggestion generation mechanism based on usage frequency information to generate suggestions for organization and disposal for the user. The input is usage frequency data for each item, and by executing the necessary programs for the suggestions, it outputs a suggestion statement that determines which items should be disposed of. This information is passed to the presentation mechanism.

[0116] Step 5:

[0117] The user's device (smartphone, tablet, etc.) notifies the user of suggestions sent from the server via a display mechanism. It receives the suggestion text from the server as input and displays the notification on the user's device as output. The user reviews the suggestion and makes a choice to organize it as needed.

[0118] Step 6:

[0119] If the user approves the proposal, the server uses processing tools to perform the corresponding action (e.g., listing on an online trading platform or contacting a collector). The input is the user's approval action, and the output is the execution of the selected disposal method.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] This invention provides a system that efficiently organizes and disposes of household items by offering suggestions based on frequency of use and taking into account the user's emotions. Specifically, it comprises information acquisition means, information analysis means, frequency of use estimation means, suggestion generation means, processing means, and an emotion engine. The system of this invention supports the user's decluttering process as follows.

[0122] First, the server periodically acquires images of the room through an AI camera, which is an information acquisition device installed in the room. These images are sent to the server, and objects within the images are identified by an information analysis device. The information of the identified objects is recorded in a database.

[0123] Next, the server uses a usage frequency estimation means to estimate the usage frequency of each object based on past data in the database. Based on this estimation result, the server uses a suggestion generation means to create suggestions regarding the organization or disposal of the items.

[0124] Furthermore, this system is equipped with an emotion engine that analyzes the user's emotional state when the user provides voice or video input to the system using a terminal. Once the user's emotions are collected, the emotion engine analyzes them to determine the user's current emotional state.

[0125] The results from the emotion engine are supplied to the suggestion generation system. The server adjusts the content and urgency of decluttering suggestions based on the emotional state. For example, if the user is stressed, the server will make simple suggestions and prioritize suggestions for items that are not used very often.

[0126] When a user reviews and approves a proposal using their device and submits feedback, the server automatically lists the selected unwanted items for sale via a processing system and arranges for their disposal in cooperation with a recycling company. This allows users to easily declutter their living space without having to go through cumbersome procedures.

[0127] This system supports users' lives and, through emotionally considerate suggestions, makes it possible to accept tidiness and organization more favorably.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The server activates an AI camera, which is a means of acquiring information, and takes pictures of the room at regular intervals. The server sends these images to a database and stores them there.

[0131] Step 2:

[0132] The server activates the information analysis system and recognizes objects in the room from the acquired images. It analyzes information such as the type, location, and size of the recognized objects and records it in a database.

[0133] Step 3:

[0134] The server uses a usage frequency estimation method to analyze movement and change information of objects stored in the database to estimate the usage frequency of each object. Based on this, it identifies which items may be unnecessary.

[0135] Step 4:

[0136] The server uses an emotion engine to analyze the user's voice and video data obtained from the terminal to assess the user's emotional state. For example, it can determine whether the user is relaxed or stressed.

[0137] Step 5:

[0138] The server utilizes a suggestion generation mechanism to create optimal decluttering suggestions based on object usage frequency data and the user's emotional state. These suggestions are then displayed on the device, specifically informing the user, "It appears this garment hasn't been worn for three months. Why not consider getting rid of it now?"

[0139] Step 6:

[0140] The user reviews the proposal using a device and approves or modifies the process. For example, the user decides on the final disposal method by agreeing to the proposal or selecting an alternative option.

[0141] Step 7:

[0142] Upon receiving user instructions, the server activates the processing system and automatically lists unwanted items online. Alternatively, the server coordinates with recycling companies or donation recipients via a linked system to ensure efficient disposal of items. The user is then notified when disposal is complete.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] Organizing belongings and disposing of unwanted items within a household requires considerable time, effort, and can also be emotionally burdensome. In particular, appropriate suggestions that consider the frequency of use of individual items and the user's feelings at the time are required, but conventional systems often fail to adequately address these issues.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes a recording means that operates as an information acquisition means, an analysis means that analyzes recorded images to classify objects, a frequency estimation means that estimates the frequency of use of objects, an emotion analysis means that analyzes the user's emotions from audio or video, an adjustment means that feeds back the results of the emotion analysis means to a suggestion means to adjust the suggested content, and a processing means that processes unwanted items based on the suggestions. This makes it possible to efficiently and appropriately organize items and dispose of unwanted items while taking into account the user's emotional state.

[0148] "Information acquisition means" refers to means of collecting data necessary to understand the items in a room, and includes cameras, etc.

[0149] "Analysis means" refers to means for analyzing collected data and identifying and classifying objects within an image.

[0150] A "frequency estimation method" is a means for calculating and predicting the frequency of use of an item based on historical information accumulated in a database.

[0151] "Suggestion methods" refer to means for creating suggestions that advise on organizing or disposing of items, taking into account the frequency of use of the items and other factors.

[0152] An "emotion analysis tool" is a means of analyzing a user's emotional state from their voice or video and providing that information.

[0153] "Adjustment methods" refer to means of adjusting the proposed content based on analyzed emotional information and selecting an approach that suits the user's current situation.

[0154] "Processing means" refers to means for processing unwanted items in a specific manner based on the proposal, such as automating procedures for listing items for sale or collection.

[0155] This system assists in the efficient organization and disposal of items within the home and consists of server, terminal, and user roles.

[0156] The server uses an AI camera installed in the room as a means of acquiring information. This camera can be, for example, a general surveillance camera or a smart home device, and it periodically records the room's state. The acquired images are sent to the server, where analysis software such as OpenCV or TensorFlow is used to identify and classify objects within the images. The identification information for each object is stored in a database.

[0157] The server applies machine learning techniques based on accumulated information and estimates the frequency of item use using software frameworks such as scikit-learn and PyTorch. The estimation results are output as suggestions to the user through a suggestion generation system using natural language generation tools.

[0158] The user receives this suggestion through their device. The device allows for voice and video input to analyze the user's emotions. Amazon Polly or general speech recognition software is used for emotion analysis, and the server receives the obtained emotion information and uses it to adjust the suggestion.

[0159] For example, if a frying pan that hasn't been used for a long time is identified in a user's living space, it will be judged as having low usage frequency. The server will then provide the user with a suggestion such as, "This frying pan hasn't been used for over six months. If you don't need it, consider disposing of it." If the user approves the suggestion, the server may proceed with the process of automatically listing the frying pan on an online platform.

[0160] An example of a prompt for the generating AI model is an input such as, "A scenario that generates decluttering suggestions when the user is feeling stressed." In this way, it is possible to support the creation of a comfortable space while taking into account the user's lifestyle and emotions.

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The server periodically acquires images of the room using an AI camera installed in the room. The image data obtained by the camera is input to the server. The server processes the data through analysis software to identify objects in the images and outputs the identified object information to a database.

[0164] Step 2:

[0165] The server uses machine learning algorithms to estimate the frequency of use of each item, referencing historical data based on item information in the database. The input data undergoes frequency analysis using scikit-learn or PyTorch to produce output representing the usage frequency of each item.

[0166] Step 3:

[0167] The server takes usage frequency data as input and runs a suggestion generation program to generate suggestions for organization or disposal for the user. The suggestion text is generated using natural language generation technology and output to the user's terminal.

[0168] Step 4:

[0169] The user receives generated suggestions through their device. Emotional information is obtained by the user inputting their voice or video, which activates the emotion analysis system. The input emotional data is analyzed through speech recognition software and output to the server as information about the emotional state.

[0170] Step 5:

[0171] The server receives the results of the sentiment analysis as input and provides feedback to the suggestion generation system, adjusting the suggested content according to the user's emotional state. The suggested content, taking the emotional state into account, is then output again to the user's terminal.

[0172] Step 6:

[0173] The user reviews the proposal and sends feedback to the server indicating approval or rejection. For approved proposals, the server automatically proceeds with the processing of the unwanted items using listing and integration methods. Listings are output as data to the online platform, and integrations are output as information to the recycling company.

[0174] (Application Example 2)

[0175] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0176] The problem this invention aims to solve is to provide more individualized and appropriate suggestions for organizing household items, not just based on frequency of use, but also tailored to the user's emotions and circumstances. This will enable users to organize their belongings efficiently while reducing their psychological burden.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0178] In this invention, the server includes information acquisition means for acquiring images, information analysis means for analyzing acquired images and recognizing objects, usage frequency estimation means for estimating the frequency of use of objects, state analysis means for analyzing the user's state, suggestion adjustment means for adjusting the suggestion content based on the user's state, and processing means for processing unnecessary items based on the suggestion. This makes it possible to provide suggestions that take into account the emotional state of each individual user.

[0179] "Information acquisition means for acquiring images" refers to means of capturing image data using cameras or sensors that are installed in an environment such as a room.

[0180] "Information analysis means for analyzing acquired images to recognize objects" refers to means of identifying objects from acquired image data using artificial intelligence or the like, and determining their characteristics and type.

[0181] A "usage frequency estimation means for estimating the usage frequency of an object" is a means for calculating and predicting how often a particular object is used, based on past data.

[0182] A "state analysis method for analyzing the user's condition" is a means of analyzing and understanding the user's emotions and mental state in real time.

[0183] A "proposal adjustment mechanism that adjusts the content of suggestions based on the user's state" is a means of changing the content of suggestions regarding the optimal organization and disposal of items according to the user's emotions and state.

[0184] "Processing means for disposing of unwanted items based on proposals" refers to means for automatically or semi-automatically organizing and disposing of items proposed by the user.

[0185] This invention provides a system for efficiently organizing belongings within a user's living space. This system utilizes AI-powered advanced analytical capabilities to achieve a personalized approach to organizing belongings that suits the user's lifestyle.

[0186] First, the server periodically acquires images of the room using an AI camera as an information acquisition tool. The acquired images are sent to the server, where objects are recognized by an information analysis tool. This recognized object information is stored in a data recording device and used for subsequent processing.

[0187] The server then uses an object usage frequency estimation mechanism to estimate how often an object is used. This estimation is based on past usage data and serves as a basis for evaluating the importance and priority of the item.

[0188] The user's device analyzes the user's voice and facial expressions to understand their emotional state at that time. The state analysis means can determine the user's emotions from these inputs and identify everyday stress, excitement, etc.

[0189] The acquired emotional data is supplied to the suggestion adjustment mechanism. The server generates coping suggestions based on the user's emotional state and past usage frequency, and displays them on the user's terminal. The content of the suggestions is designed to reduce the user's psychological burden. For example, if the user is stressed, easily actionable coping suggestions are prioritized.

[0190] As a concrete example, consider a situation where a user wants to try a new hobby and needs to secure workspace. In this case, the system prioritizes organizing the items necessary for the hobby and makes suggestions, including the collection and disposal of unnecessary items.

[0191] An example of a prompt for a generative AI model would be: "Create an example of an AI dialogue that suggests tidying up when the user is relaxed."

[0192] These features allow users to organize their living spaces comfortably and efficiently, enabling them to live with greater peace of mind.

[0193] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0194] Step 1:

[0195] The server uses an AI camera to acquire images of the room.

[0196] The input is video data from a camera.

[0197] The output is image data showing the current state of the room. The server stores this image data in a database for subsequent analysis.

[0198] Step 2:

[0199] The server uses information analysis tools to recognize objects from the acquired images.

[0200] The input is the image data obtained in step 1.

[0201] The output is a list of identified objects. The server uses image analysis techniques to analyze the shape, color, and arrangement of the objects, and records the characteristics of each object in a database.

[0202] Step 3:

[0203] The server uses a usage frequency estimation means to estimate the usage frequency of the identified object.

[0204] The input consists of past object usage history and object recognition data.

[0205] The output is usage frequency information for each object. The server performs time-series data analysis to understand the usage trends of the objects and calculate the frequency.

[0206] Step 4:

[0207] The device analyzes the user's voice and facial expressions to evaluate the user's emotional state.

[0208] The input consists of voice data and facial expression images from the user.

[0209] The output is data indicating the user's emotional state. The device uses a combination of voice processing technology and facial recognition to estimate emotions and sends the results to the server.

[0210] Step 5:

[0211] The server uses a suggestion adjustment mechanism to generate organized suggestions based on usage frequency information and emotional state.

[0212] The input consists of information on the frequency of use of objects and data on the user's emotional state.

[0213] The output consists of tailored sorting and disposal suggestions. The server utilizes an AI model to generate and send optimal suggestions to each user's device.

[0214] Step 6:

[0215] Users review the suggestions on their devices and send feedback as needed.

[0216] The input is suggested data from the server.

[0217] The output is user feedback data. Users provide feedback based on the displayed suggestions, and this information is used to improve future suggestions.

[0218] Step 7:

[0219] The server uses processing means to process unwanted items based on the proposal.

[0220] The input is user feedback data.

[0221] The output shows the completion status of the waste disposal process. The server disposes of items approved by the user in cooperation with online marketplaces and recycling companies, and reports the processing results.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] This invention provides a system for monitoring the usage of items within a household and efficiently decluttering. This system comprises information acquisition means, information analysis means, usage frequency estimation means, suggestion generation means, and processing means. A specific example of a system using these means is shown below.

[0239] First, the server periodically acquires images using an information acquisition device, namely an AI camera, installed in the room. The server receives this image data and uses an information analysis device to identify and recognize objects within the images. At this time, features such as the type and shape of the identified objects are also extracted and recorded in a database.

[0240] Next, the server uses a usage frequency estimation mechanism to calculate the usage frequency of each object based on the information recorded in the database. Based on this information, the server decides which items should be decluttered.

[0241] Based on the results, the server uses a suggestion generation mechanism to generate the optimal decluttering plan for the user. The server notifies the user of this suggestion via the terminal, and the user can review and modify the suggestion. For example, the server might present the user with the suggestion, "These clothes may not have been worn in the past six months. Shall we declutter them?"

[0242] Furthermore, if the user approves the proposal, the server automatically lists the unwanted items on the online platform using processing tools. The server also supports the process of arranging for the items to be sent to a recycling company or donation recipient by utilizing various integration tools.

[0243] In this way, the present invention enables efficient organization and disposal of items, making it easier to maintain a comfortable living space. Users can practice decluttering in their daily lives without feeling burdened.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The server controls AI cameras, which are information acquisition tools installed in the room, and acquires images at specific time intervals. The server receives these image data in real time and stores them in a database.

[0247] Step 2:

[0248] The server analyzes the received image data using information analysis tools. It recognizes objects within the image and identifies the type and location of each object. The server analyzes this recognition information and records the characteristics of the objects in a database.

[0249] Step 3:

[0250] The server operates a usage frequency estimation system and calculates the current usage frequency based on the previously recorded usage history of the object. Specifically, it evaluates whether the object is being used by taking into account changes in the object's position and exposure time. The results are reflected in the database.

[0251] Step 4:

[0252] The server uses a suggestion generation mechanism to generate a decluttering plan for the user based on usage frequency data. The terminal displays this plan and sends a notification, such as "This item hasn't been used recently, so we suggest disposing of it." The terminal accepts user confirmation or modifications.

[0253] Step 5:

[0254] After receiving user approval, the server uses processing tools to handle the disposal of unwanted items. Specifically, a listing process is initiated to automatically list the unwanted items on an online platform.

[0255] Step 6:

[0256] Furthermore, using collaborative processing methods, the system can arrange for unwanted items to be sent to recycling companies or donation recipients if the user so desires. The server manages these collaborations with external services, supporting the efficient disposal of unwanted items.

[0257] (Example 1)

[0258] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0259] In modern living environments, a large number of items accumulate in homes, posing a challenge in organizing and disposing of them. This can lead to cramped living spaces and a loss of comfort. Furthermore, the effort required to choose the optimal disposal method for unwanted items makes item management inefficient. Therefore, there is a need for efficient methods to manage and appropriately dispose of items accumulated in homes.

[0260] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0261] In this invention, the server includes: an image acquisition means that periodically acquires images of items using an information acquisition device installed in the room to acquire images; an information analysis means that analyzes the acquired image data using computing resources on the cloud to recognize the items; a usage frequency estimation means that records the type and characteristics of the recognized items and estimates the frequency of use of the items based on that data; a proposal generation means that uses a generation AI model to propose organizing or disposing of the items based on the estimated usage frequency; and a processing means that processes unwanted items after receiving user confirmation based on the proposal. This enables efficient management of items in living spaces and prompt and appropriate disposal.

[0262] An "information acquisition device" is a device installed in a room to acquire images of a target object.

[0263] "Image acquisition means" refers to a function that periodically acquires images of an object via an information acquisition device.

[0264] "Information analysis means" refers to a function that uses computing resources on the cloud to analyze acquired image data and recognize objects.

[0265] "Usage frequency estimation means" refers to a function that records the type and characteristics of recognized items as data and estimates the frequency of use of those items based on that data.

[0266] The "proposal generation means" is a function that uses a generation AI model to suggest to the user how to organize or dispose of items based on the estimated frequency of use.

[0267] "Processing means" refers to a function that processes unwanted items based on proposals presented by the proposal generation means, after receiving user confirmation.

[0268] The "listing function" is a feature that, after user approval, automatically lists unwanted items on an online exchange platform.

[0269] "Collaboration function" refers to a function that allows the processing system to dispose of unwanted items in cooperation with collection companies or donation recipients.

[0270] This invention is a system that supports the management, efficient organization, and disposal of household items. The server periodically acquires images of items using an information acquisition device installed in the room. Specifically, a general-purpose camera module is used as hardware, and the captured image data is transmitted to computing resources in the cloud.

[0271] The server sends this image data to an information analysis tool in the cloud, where it performs image analysis using common machine learning libraries, including Google® TensorFlow. As a result, objects are recognized, and features such as the type and shape of the recognized objects are extracted. This information is recorded in a database, and database services such as Amazon RDS can be used.

[0272] Next, the server uses a usage frequency estimation mechanism to estimate the frequency of use of items based on the recorded information. This process uses the Python Pandas library to analyze item usage patterns based on past analysis data.

[0273] Regarding the suggestion generation method, OpenAI's generative AI model is used to suggest decluttering to the user based on acquired usage frequency data. For example, it generates a natural language prompt such as, "This cup hasn't been used in the last three months. Please consider decluttering it."

[0274] Users can receive and confirm suggestions via their devices. Based on feedback from the devices, the server executes the procedures for processing unwanted items. Using the processing methods, after user approval, unwanted items are automatically listed on online exchange platforms or a disposal plan is created. Coordination with recycling companies and donation recipients is also considered, and efficient communication and arrangements are made via the network.

[0275] A concrete example of a prompt message for a generative AI model would be: "Create a phrase that reports examples of items in the user's living space that haven't been used recently, and asks if the user is willing to organize those items."

[0276] This system allows users to efficiently manage their belongings and maintain a comfortable living space.

[0277] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0278] Step 1:

[0279] The server periodically acquires images of items using an information acquisition device (AI camera) installed in the room. The input is real-time image data acquired by the camera, and the output is an image file, such as JPEG, sent to the server for analysis. This image acquisition is performed automatically at a specified time each day.

[0280] Step 2:

[0281] The server sends the image data to the cloud. Here, the input image data is transferred to the cloud's analysis service using a secure protocol. The output is the image data waiting on the cloud. Specifically, operations are performed to securely send the image using SSL / TLS and add it to the analysis queue.

[0282] Step 3:

[0283] The server uses the information analysis means on the cloud to analyze the image data and recognize the item. The input is the image data obtained in Step 2, and the output is the analysis result including the type and characteristics of the recognized item. For this process, a machine learning library is used to label the type of the item. Specifically, the item is classified as "book", "clothing", "dish", etc., and features such as its shape and color are extracted.

[0284] Step 4:

[0285] The server records the recognized item data in the database. The input is the analysis result of the item generated in Step 3, and the output is the updated database entry. Here, the ID, type, characteristics, and date-time information of each item are saved in Amazon RDS. As a specific operation, data is inserted using SQL queries.

[0286] Step 5:

[0287] The server uses the usage frequency estimation means to analyze the recorded data and calculate the usage frequency of the item. The input is the past usage data of the item accumulated in the database, and the output is the estimated value of the usage frequency of each item. Using the Pandas library, the usage history of each item is analyzed to draw conclusions such as "this shirt was used 3 times in the past 30 days".

[0288] Step 6:

[0289] The server uses an AI model to generate suggestions for the user based on item usage frequency data. The input is the estimated usage frequency, and the output is suggestions for decluttering. For example, it might generate a prompt such as, "This jacket has hardly been used in the last 6 months. Please consider getting rid of it."

[0290] Step 7:

[0291] The server generates a proposal and notifies the user's device. The input is the proposal content generated in step 6, and the output is a push notification to the user's device. The user can review and approve the proposal through the push notification. This operation is performed using the mobile app notification service.

[0292] Step 8:

[0293] If the user approves the proposal, the server processes the unwanted items. Specifically, the input is the user's approval, and the output is the procedure for listing the unwanted items on an online exchange platform or handing them over to a recycling company. At this point, product information is registered and listed via the online platform API, and the process is completed.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] The challenge lies in addressing the difficulty of efficiently utilizing living space, which arises from the increasing number of items in the home and the complexity of managing them. In particular, the lack of identification of unnecessary items and a plan for their disposal is a factor that reduces the quality of life.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes information gathering means, information analysis means, usage frequency estimation means, suggestion generation means, processing means, and presentation means. This enables effective organization and waste-free disposal of household items.

[0299] "Information gathering means" refers to devices or functions that acquire images, which are installed in home autonomous devices to understand the situation of the target object.

[0300] "Information analysis means" refers to techniques and methods for processing acquired image data and identifying specific objects.

[0301] A "usage frequency estimation means" is a method or device for analyzing the handling and usage conditions of an object and calculating its usage frequency.

[0302] The "proposal generation method" is a function that creates recommendations for organizing and disposing of items based on usage frequency data.

[0303] "Processing means" refers to the means of implementation for carrying out specific disposal or management actions based on the proposal.

[0304] "Presentation means" refers to a function that uses a home-use autonomous device to display or notify the user of suggested content.

[0305] An "online trading platform" is a general term for places and services where goods are bought and sold via the internet.

[0306] "Cooperative methods" refer to means of efficiently disposing of unnecessary items in cooperation with collection companies and donation recipients.

[0307] To implement this invention, the following configuration and technologies are required in the household autonomous device. The server is equipped with an AI camera as the information collection means and periodically takes pictures of the items in the household. The acquired image data is sent to the server and then analyzed using image analysis software such as TensorFlow as the information analysis means. This makes it possible to identify and distinguish the object.

[0308] Based on the analyzed data of the object, the server calculates its usage frequency using the usage frequency estimation means. Based on the usage frequency data, the proposal generation means operates to determine which items should be sorted or disposed of and proposes the results to the user. The presentation to the user is made by the presentation means provided in the household autonomous device. This presentation means can also notify the user through terminals such as smartphones and tablets.

[0309] (Specific example)

[0310] For example, the server notifies the user's terminal with a proposal such as "This tableware doesn't seem to have been used in the past year. Do you consider disposal or donation?" If the user approves this proposal, the unwanted items are automatically listed on the online trading platform or, if necessary, contacted to collectors etc. by the cooperation means. An example of a specific prompt sentence is "Identify the items that have not been used in the past 6 months and generate a proposal for sorting." By using this system, not only can the user efficiently organize the living space, but it also helps to clearly distinguish between necessary and unnecessary items.

[0311] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0312] Step 1:

[0313] The server uses an AI camera as a means of information gathering to periodically acquire images of items in the home. The AI ​​camera provides images of the home environment as input. These images are output directly as digital data and sent to the server.

[0314] Step 2:

[0315] The server processes the received image data using image analysis software such as TensorFlow. Image data is sent to the server as input, and information for identifying objects (such as their shape and color) is extracted from this data. The extracted information is then recorded in a database.

[0316] Step 3:

[0317] The server calculates the usage frequency of each item using a usage frequency estimation means, based on data obtained by the information analysis means. It uses previously recorded item characteristic data as input and performs calculations using a usage frequency model. As a result of the calculation, it outputs usage frequency information for each item and passes it to the proposal generation means.

[0318] Step 4:

[0319] The server uses a suggestion generation mechanism based on usage frequency information to generate suggestions for organization and disposal for the user. The input is usage frequency data for each item, and by executing the necessary programs for the suggestions, it outputs a suggestion statement that determines which items should be disposed of. This information is passed to the presentation mechanism.

[0320] Step 5:

[0321] The user's device (smartphone, tablet, etc.) notifies the user of suggestions sent from the server via a display mechanism. It receives the suggestion text from the server as input and displays the notification on the user's device as output. The user reviews the suggestion and makes a choice to organize it as needed.

[0322] Step 6:

[0323] If the user approves the proposal, the server uses processing tools to perform the corresponding action (e.g., listing on an online trading platform or contacting a collector). The input is the user's approval action, and the output is the execution of the selected disposal method.

[0324] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0325] This invention provides a system that efficiently organizes and disposes of household items by offering suggestions based on frequency of use and taking into account the user's emotions. Specifically, it comprises information acquisition means, information analysis means, frequency of use estimation means, suggestion generation means, processing means, and an emotion engine. The system of this invention supports the user's decluttering process as follows.

[0326] First, the server periodically acquires images of the room through an AI camera, which is an information acquisition device installed in the room. These images are sent to the server, and objects within the images are identified by an information analysis device. The information of the identified objects is recorded in a database.

[0327] Next, the server uses a usage frequency estimation means to estimate the usage frequency of each object based on past data in the database. Based on this estimation result, the server uses a suggestion generation means to create suggestions regarding the organization or disposal of the items.

[0328] Furthermore, this system is equipped with an emotion engine that analyzes the user's emotional state when the user provides voice or video input to the system using a terminal. Once the user's emotions are collected, the emotion engine analyzes them to determine the user's current emotional state.

[0329] The results from the emotion engine are supplied to the suggestion generation system. The server adjusts the content and urgency of decluttering suggestions based on the emotional state. For example, if the user is stressed, the server will make simple suggestions and prioritize suggestions for items that are not used very often.

[0330] When a user reviews and approves a proposal using their device and submits feedback, the server automatically lists the selected unwanted items for sale via a processing system and arranges for their disposal in cooperation with a recycling company. This allows users to easily declutter their living space without having to go through cumbersome procedures.

[0331] This system supports users' lives and, through emotionally considerate suggestions, makes it possible to accept tidiness and organization more favorably.

[0332] The following describes the processing flow.

[0333] Step 1:

[0334] The server activates an AI camera, which is a means of acquiring information, and takes pictures of the room at regular intervals. The server sends these images to a database and stores them there.

[0335] Step 2:

[0336] The server activates the information analysis system and recognizes objects in the room from the acquired images. It analyzes information such as the type, location, and size of the recognized objects and records it in a database.

[0337] Step 3:

[0338] The server uses a usage frequency estimation method to analyze movement and change information of objects stored in the database to estimate the usage frequency of each object. Based on this, it identifies which items may be unnecessary.

[0339] Step 4:

[0340] The server uses an emotion engine to analyze the user's voice and video data obtained from the terminal to assess the user's emotional state. For example, it can determine whether the user is relaxed or stressed.

[0341] Step 5:

[0342] The server utilizes a suggestion generation mechanism to create optimal decluttering suggestions based on object usage frequency data and the user's emotional state. These suggestions are then displayed on the device, specifically informing the user, "It appears this garment hasn't been worn for three months. Why not consider getting rid of it now?"

[0343] Step 6:

[0344] The user reviews the proposal using a device and approves or modifies the process. For example, the user decides on the final disposal method by agreeing to the proposal or selecting an alternative option.

[0345] Step 7:

[0346] Upon receiving user instructions, the server activates the processing system and automatically lists unwanted items online. Alternatively, the server coordinates with recycling companies or donation recipients via a linked system to ensure efficient disposal of items. The user is then notified when disposal is complete.

[0347] (Example 2)

[0348] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0349] Organizing belongings and disposing of unwanted items within a household requires considerable time, effort, and can also be emotionally burdensome. In particular, appropriate suggestions that consider the frequency of use of individual items and the user's feelings at the time are required, but conventional systems often fail to adequately address these issues.

[0350] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0351] In this invention, the server includes a recording means that operates as an information acquisition means, an analysis means that analyzes recorded images to classify objects, a frequency estimation means that estimates the frequency of use of objects, an emotion analysis means that analyzes the user's emotions from audio or video, an adjustment means that feeds back the results of the emotion analysis means to a suggestion means to adjust the suggested content, and a processing means that processes unwanted items based on the suggestions. This makes it possible to efficiently and appropriately organize items and dispose of unwanted items while taking into account the user's emotional state.

[0352] "Information acquisition means" refers to means of collecting data necessary to understand the items in a room, and includes cameras, etc.

[0353] "Analysis means" refers to means for analyzing collected data and identifying and classifying objects within an image.

[0354] A "frequency estimation method" is a means for calculating and predicting the frequency of use of an item based on historical information accumulated in a database.

[0355] "Suggestion methods" refer to means for creating suggestions that advise on organizing or disposing of items, taking into account the frequency of use of the items and other factors.

[0356] An "emotion analysis tool" is a means of analyzing a user's emotional state from their voice or video and providing that information.

[0357] "Adjustment methods" refer to means of adjusting the proposed content based on analyzed emotional information and selecting an approach that suits the user's current situation.

[0358] "Processing means" refers to means for processing unwanted items in a specific manner based on the proposal, such as automating procedures for listing items for sale or collection.

[0359] This system assists in the efficient organization and disposal of items within the home and consists of server, terminal, and user roles.

[0360] The server uses an AI camera installed in the room as a means of acquiring information. This camera can be, for example, a general surveillance camera or a smart home device, and it periodically records the room's state. The acquired images are sent to the server, where analysis software such as OpenCV or TensorFlow is used to identify and classify objects within the images. The identification information for each object is stored in a database.

[0361] The server applies machine learning techniques based on accumulated information and estimates the frequency of item use using software frameworks such as scikit-learn and PyTorch. The estimation results are output as suggestions to the user through a suggestion generation system using natural language generation tools.

[0362] The user receives this suggestion through their device. The device allows for voice and video input to analyze the user's emotions. Amazon Polly or general speech recognition software is used for emotion analysis, and the server receives the obtained emotion information and uses it to adjust the suggestion.

[0363] For example, if a frying pan that hasn't been used for a long time is identified in a user's living space, it will be judged as having low usage frequency. The server will then provide the user with a suggestion such as, "This frying pan hasn't been used for over six months. If you don't need it, consider disposing of it." If the user approves the suggestion, the server may proceed with the process of automatically listing the frying pan on an online platform.

[0364] An example of a prompt for the generating AI model is an input such as, "A scenario that generates decluttering suggestions when the user is feeling stressed." In this way, it is possible to support the creation of a comfortable space while taking into account the user's lifestyle and emotions.

[0365] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0366] Step 1:

[0367] The server periodically acquires images of the room using an AI camera installed in the room. The image data obtained by the camera is input to the server. The server processes the data through analysis software to identify objects in the images and outputs the identified object information to a database.

[0368] Step 2:

[0369] The server uses machine learning algorithms to estimate the frequency of use of each item, referencing historical data based on item information in the database. The input data undergoes frequency analysis using scikit-learn or PyTorch to produce output representing the usage frequency of each item.

[0370] Step 3:

[0371] The server takes usage frequency data as input and runs a suggestion generation program to generate suggestions for organization or disposal for the user. The suggestion text is generated using natural language generation technology and output to the user's terminal.

[0372] Step 4:

[0373] The user receives generated suggestions through their device. Emotional information is obtained by the user inputting their voice or video, which activates the emotion analysis system. The input emotional data is analyzed through speech recognition software and output to the server as information about the emotional state.

[0374] Step 5:

[0375] The server receives the results of the sentiment analysis as input and provides feedback to the suggestion generation system, adjusting the suggested content according to the user's emotional state. The suggested content, taking the emotional state into account, is then output again to the user's terminal.

[0376] Step 6:

[0377] The user reviews the proposal and sends feedback to the server indicating approval or rejection. For approved proposals, the server automatically proceeds with the processing of the unwanted items using listing and integration methods. Listings are output as data to the online platform, and integrations are output as information to the recycling company.

[0378] (Application Example 2)

[0379] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0380] The problem this invention aims to solve is to provide more individualized and appropriate suggestions for organizing household items, not just based on frequency of use, but also tailored to the user's emotions and circumstances. This will enable users to organize their belongings efficiently while reducing their psychological burden.

[0381] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0382] In this invention, the server includes information acquisition means for acquiring images, information analysis means for analyzing acquired images and recognizing objects, usage frequency estimation means for estimating the frequency of use of objects, state analysis means for analyzing the user's state, suggestion adjustment means for adjusting the suggestion content based on the user's state, and processing means for processing unnecessary items based on the suggestion. This makes it possible to provide suggestions that take into account the emotional state of each individual user.

[0383] "Information acquisition means for acquiring images" refers to means of capturing image data using cameras or sensors that are installed in an environment such as a room.

[0384] "Information analysis means for analyzing acquired images to recognize objects" refers to means for identifying objects from acquired image data using artificial intelligence or the like, and determining their characteristics and type.

[0385] A "usage frequency estimation means for estimating the usage frequency of an object" is a means for calculating and predicting how often a particular object is used, based on past data.

[0386] A "state analysis method for analyzing the user's condition" is a means of analyzing and understanding the user's emotions and mental state in real time.

[0387] A "proposal adjustment mechanism that adjusts the content of suggestions based on the user's state" is a means of changing the content of suggestions regarding the optimal organization and disposal of items according to the user's emotions and state.

[0388] "Processing means for disposing of unwanted items based on proposals" refers to means for automatically or semi-automatically organizing and disposing of items proposed by the user.

[0389] This invention provides a system for efficiently organizing belongings within a user's living space. This system utilizes AI-powered advanced analytical capabilities to achieve a personalized approach to organizing belongings that suits the user's lifestyle.

[0390] First, the server periodically acquires images of the room using an AI camera as an information acquisition tool. The acquired images are sent to the server, and objects are recognized by an information analysis tool. This recognized object information is stored in a data recording device and used for subsequent processing.

[0391] The server then uses an object usage frequency estimation mechanism to estimate how often an object is used. This estimation is based on past usage data and serves as a basis for evaluating the importance and priority of the item.

[0392] The user's device analyzes the user's voice and facial expressions to understand their emotional state at that time. The state analysis means can determine the user's emotions from these inputs and identify everyday stress, excitement, etc.

[0393] The acquired emotional data is supplied to the suggestion adjustment mechanism. The server generates coping suggestions based on the user's emotional state and past usage frequency, and displays them on the user's terminal. The content of the suggestions is designed to reduce the user's psychological burden. For example, if the user is stressed, easily actionable coping suggestions are prioritized.

[0394] As a concrete example, consider a situation where a user wants to try a new hobby and needs to secure workspace. In this case, the system prioritizes organizing the items necessary for the hobby and makes suggestions, including the collection and disposal of unnecessary items.

[0395] An example of a prompt for a generative AI model would be: "Create an example of an AI dialogue that suggests tidying up when the user is relaxed."

[0396] These features allow users to organize their living spaces comfortably and efficiently, enabling them to live with greater peace of mind.

[0397] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0398] Step 1:

[0399] The server uses an AI camera to acquire images of the room.

[0400] The input is video data from a camera.

[0401] The output is image data showing the current state of the room. The server stores this image data in a database for subsequent analysis.

[0402] Step 2:

[0403] The server uses information analysis tools to recognize objects from the acquired images.

[0404] The input is the image data obtained in step 1.

[0405] The output is a list of identified objects. The server uses image analysis techniques to analyze the shape, color, and arrangement of the objects, and records the characteristics of each object in a database.

[0406] Step 3:

[0407] The server uses a usage frequency estimation means to estimate the usage frequency of the identified object.

[0408] The input consists of past object usage history and object recognition data.

[0409] The output is usage frequency information for each object. The server performs time-series data analysis to understand the usage trends of the objects and calculate the frequency.

[0410] Step 4:

[0411] The device analyzes the user's voice and facial expressions to evaluate the user's emotional state.

[0412] The input consists of voice data and facial expression images from the user.

[0413] The output is data indicating the user's emotional state. The device uses a combination of voice processing technology and facial recognition to estimate emotions and sends the results to the server.

[0414] Step 5:

[0415] The server uses a suggestion adjustment mechanism to generate organized suggestions based on usage frequency information and emotional state.

[0416] The input consists of information on the frequency of use of objects and data on the user's emotional state.

[0417] The output consists of tailored sorting and disposal suggestions. The server utilizes an AI model to generate and send optimal suggestions to each user's device.

[0418] Step 6:

[0419] Users review the suggestions on their devices and send feedback as needed.

[0420] The input is suggested data from the server.

[0421] The output is user feedback data. Users provide feedback based on the displayed suggestions, and this information is used to improve future suggestions.

[0422] Step 7:

[0423] The server uses processing means to process unwanted items based on the proposal.

[0424] The input is user feedback data.

[0425] The output shows the completion status of the waste disposal process. The server disposes of items approved by the user in cooperation with online marketplaces and recycling companies, and reports the processing results.

[0426] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0427] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0428] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0429] [Third Embodiment]

[0430] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0431] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0432] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0433] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0434] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0435] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0436] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0437] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0438] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0439] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0440] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0441] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0442] This invention provides a system for monitoring the usage of items within a household and efficiently decluttering. This system comprises information acquisition means, information analysis means, usage frequency estimation means, suggestion generation means, and processing means. A specific example of a system using these means is shown below.

[0443] First, the server periodically acquires images using an information acquisition device, namely an AI camera, installed in the room. The server receives this image data and uses an information analysis device to identify and recognize objects within the images. At this time, features such as the type and shape of the identified objects are also extracted and recorded in a database.

[0444] Next, the server uses a usage frequency estimation mechanism to calculate the usage frequency of each object based on the information recorded in the database. Based on this information, the server decides which items should be decluttered.

[0445] Based on the results, the server uses a suggestion generation mechanism to generate the optimal decluttering plan for the user. The server notifies the user of this suggestion via the terminal, and the user can review and modify the suggestion. For example, the server might present the user with the suggestion, "These clothes may not have been worn in the past six months. Shall we declutter them?"

[0446] Furthermore, if the user approves the proposal, the server automatically lists the unwanted items on the online platform using processing tools. The server also supports the process of arranging for the items to be sent to a recycling company or donation recipient by utilizing various integration tools.

[0447] In this way, the present invention enables efficient organization and disposal of items, making it easier to maintain a comfortable living space. Users can practice decluttering in their daily lives without feeling burdened.

[0448] The following describes the processing flow.

[0449] Step 1:

[0450] The server controls AI cameras, which are information acquisition tools installed in the room, and acquires images at specific time intervals. The server receives these image data in real time and stores them in a database.

[0451] Step 2:

[0452] The server analyzes the received image data using information analysis tools. It recognizes objects within the image and identifies the type and location of each object. The server analyzes this recognition information and records the characteristics of the objects in a database.

[0453] Step 3:

[0454] The server operates a usage frequency estimation system and calculates the current usage frequency based on the previously recorded usage history of the object. Specifically, it evaluates whether the object is being used by taking into account changes in the object's position and exposure time. The results are reflected in the database.

[0455] Step 4:

[0456] The server uses a suggestion generation mechanism to generate a decluttering plan for the user based on usage frequency data. The terminal displays this plan and sends a notification, such as "This item hasn't been used recently, so we suggest disposing of it." The terminal accepts user confirmation or modifications.

[0457] Step 5:

[0458] After receiving user approval, the server uses processing tools to handle the disposal of unwanted items. Specifically, a listing process is initiated to automatically list the unwanted items on an online platform.

[0459] Step 6:

[0460] Furthermore, using collaborative processing methods, the system can arrange for unwanted items to be sent to recycling companies or donation recipients if the user so desires. The server manages these collaborations with external services, supporting the efficient disposal of unwanted items.

[0461] (Example 1)

[0462] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0463] In modern living environments, a large number of items accumulate in homes, posing a challenge in organizing and disposing of them. This can lead to cramped living spaces and a loss of comfort. Furthermore, the effort required to choose the optimal disposal method for unwanted items makes item management inefficient. Therefore, there is a need for efficient methods to manage and appropriately dispose of items accumulated in homes.

[0464] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0465] In this invention, the server includes: an image acquisition means that periodically acquires images of items using an information acquisition device installed in the room to acquire images; an information analysis means that analyzes the acquired image data using computing resources on the cloud to recognize the items; a usage frequency estimation means that records the type and characteristics of the recognized items and estimates the frequency of use of the items based on that data; a proposal generation means that uses a generation AI model to propose organizing or disposing of the items based on the estimated usage frequency; and a processing means that processes unwanted items after receiving user confirmation based on the proposal. This enables efficient management of items in living spaces and prompt and appropriate disposal.

[0466] An "information acquisition device" is a device installed in a room to acquire images of a target object.

[0467] "Image acquisition means" refers to a function that periodically acquires images of an object via an information acquisition device.

[0468] "Information analysis means" refers to a function that uses computing resources on the cloud to analyze acquired image data and recognize objects.

[0469] "Usage frequency estimation means" refers to a function that records the type and characteristics of recognized items as data and estimates the frequency of use of those items based on that data.

[0470] The "proposal generation means" is a function that uses a generation AI model to suggest to the user how to organize or dispose of items based on the estimated frequency of use.

[0471] "Processing means" refers to a function that processes unwanted items based on proposals presented by the proposal generation means, after receiving user confirmation.

[0472] The "listing function" is a feature that, after user approval, automatically lists unwanted items on an online exchange platform.

[0473] "Collaboration function" refers to a function that allows the processing system to dispose of unwanted items in cooperation with collection companies or donation recipients.

[0474] This invention is a system that supports the management, efficient organization, and disposal of household items. The server periodically acquires images of items using an information acquisition device installed in the room. Specifically, a general-purpose camera module is used as hardware, and the captured image data is transmitted to computing resources in the cloud.

[0475] The server sends this image data to an information analysis tool in the cloud, where it performs image analysis using common machine learning libraries, including Google TensorFlow. As a result, objects are recognized, and features such as the type and shape of the recognized objects are extracted. This information is recorded in a database, and database services such as Amazon RDS can be used.

[0476] Next, the server uses a usage frequency estimation mechanism to estimate the frequency of use of items based on the recorded information. This process uses the Python Pandas library to analyze item usage patterns based on past analysis data.

[0477] Regarding the suggestion generation method, OpenAI's generative AI model is used to suggest decluttering to the user based on acquired usage frequency data. For example, it generates a natural language prompt such as, "This cup hasn't been used in the last three months. Please consider decluttering it."

[0478] Users can receive and confirm suggestions via their devices. Based on feedback from the devices, the server executes the procedures for processing unwanted items. Using the processing methods, after user approval, unwanted items are automatically listed on online exchange platforms or a disposal plan is created. Coordination with recycling companies and donation recipients is also considered, and efficient communication and arrangements are made via the network.

[0479] A concrete example of a prompt message for a generative AI model would be: "Create a phrase that reports examples of items in the user's living space that haven't been used recently, and asks if the user is willing to organize those items."

[0480] This system allows users to efficiently manage their belongings and maintain a comfortable living space.

[0481] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0482] Step 1:

[0483] The server periodically acquires images of items using an information acquisition device (AI camera) installed in the room. The input is real-time image data acquired by the camera, and the output is an image file, such as JPEG, sent to the server for analysis. This image acquisition is performed automatically at a specified time each day.

[0484] Step 2:

[0485] The server sends image data to the cloud. Here, the input image data is transferred to the cloud analysis service using a secure protocol. The output is image data waiting on the cloud. Specifically, the image is securely transmitted using SSL / TLS and added to the analysis queue.

[0486] Step 3:

[0487] The server uses cloud-based information analysis tools to analyze image data and recognize objects. The input is the image data acquired in step 2, and the output is the analysis results, including the type and characteristics of the recognized objects. This process uses a machine learning library to label the types of objects. Specifically, objects are classified as "books," "clothing," "dishes," etc., and their characteristics such as shape and color are extracted.

[0488] Step 4:

[0489] The server records the recognized item data in the database. The input is the item analysis result generated in step 3, and the output is the updated database entry. Here, the ID, type, characteristics, and date / time information of each item are stored in Amazon RDS. Specifically, data is inserted using SQL queries.

[0490] Step 5:

[0491] The server uses a usage frequency estimation method to analyze recorded data and calculate the usage frequency of items. The input is historical item usage data stored in a database, and the output is an estimated usage frequency for each item. Using the Pandas library, the usage history of each item is analyzed to derive conclusions such as "This shirt has been used 3 times in the past 30 days."

[0492] Step 6:

[0493] The server uses an AI model to generate suggestions for the user based on item usage frequency data. The input is the estimated usage frequency, and the output is suggestions for decluttering. For example, it might generate a prompt such as, "This jacket has hardly been used in the last 6 months. Please consider getting rid of it."

[0494] Step 7:

[0495] The server generates a proposal and notifies the user's device. The input is the proposal content generated in step 6, and the output is a push notification to the user's device. The user can review and approve the proposal through the push notification. This operation is performed using the mobile app notification service.

[0496] Step 8:

[0497] If the user approves the proposal, the server processes the unwanted items. Specifically, the input is the user's approval, and the output is the procedure for listing the unwanted items on an online exchange platform or handing them over to a recycling company. At this point, product information is registered and listed via the online platform API, and the process is completed.

[0498] (Application Example 1)

[0499] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0500] The challenge lies in addressing the difficulty of efficiently utilizing living space, which arises from the increasing number of items in the home and the complexity of managing them. In particular, the lack of identification of unnecessary items and a plan for their disposal is a factor that reduces the quality of life.

[0501] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0502] In this invention, the server includes information gathering means, information analysis means, usage frequency estimation means, suggestion generation means, processing means, and presentation means. This enables effective organization and waste-free disposal of household items.

[0503] "Information gathering means" refers to devices or functions that acquire images, which are installed in home autonomous devices to understand the situation of an object.

[0504] "Information analysis means" refers to techniques and methods for processing acquired image data and identifying specific objects.

[0505] A "usage frequency estimation means" is a method or device for analyzing the handling and usage conditions of an object and calculating its usage frequency.

[0506] The "proposal generation method" is a function that creates recommendations for organizing and disposing of items based on usage frequency data.

[0507] "Processing means" refers to the means of implementation for carrying out specific disposal or management actions based on the proposal.

[0508] "Presentation means" refers to a function that uses a home-use autonomous device to display or notify the user of suggested content.

[0509] An "online trading platform" is a general term for places and services where goods are bought and sold via the internet.

[0510] "Cooperative methods" refer to means of efficiently disposing of unnecessary items in cooperation with collection companies and donation recipients.

[0511] To realize this invention, the following configuration and technology are necessary for the autonomous home device. The server is equipped with an AI camera as an information gathering means and periodically photographs items in the home. This acquired image data is transmitted to the server and then analyzed using image analysis software such as TensorFlow as an information analysis means. This makes it possible to identify and distinguish objects.

[0512] The server calculates the frequency of use of the analyzed objects using a usage frequency estimation means. Based on the usage frequency data, a suggestion generation means operates to determine which items should be sorted or disposed of, and proposes the results to the user. The suggestion to the user is presented by a presentation means provided by the home autonomous device. This presentation means can also notify the user via a terminal such as a smartphone or tablet.

[0513] (Specific example)

[0514] For example, the server might notify the user's device with a suggestion such as, "These dishes appear to have not been used in the past year. Would you like to consider disposing of them or donating them?" If the user approves this suggestion, the unwanted items are automatically listed on an online trading platform or, if necessary, contacted by a collection agency or other relevant party through a collaborative means. An example of a specific prompt might be, "Identify items that have not been used in the past six months and generate a decluttering suggestion." This system not only allows users to efficiently organize their living spaces but also helps them clearly distinguish between what they need and what they don't.

[0515] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0516] Step 1:

[0517] The server uses an AI camera as a means of information gathering to periodically acquire images of items in the home. The AI ​​camera provides images of the home environment as input. These images are output directly as digital data and sent to the server.

[0518] Step 2:

[0519] The server processes the received image data using image analysis software such as TensorFlow. Image data is sent to the server as input, and information for identifying objects (such as their shape and color) is extracted from this data. The extracted information is then recorded in a database.

[0520] Step 3:

[0521] The server calculates the usage frequency of each item using a usage frequency estimation means, based on data obtained by the information analysis means. It uses previously recorded item characteristic data as input and performs calculations using a usage frequency model. As a result of the calculation, it outputs usage frequency information for each item and passes it to the proposal generation means.

[0522] Step 4:

[0523] The server uses a suggestion generation mechanism based on usage frequency information to generate suggestions for organization and disposal for the user. The input is usage frequency data for each item, and by executing the necessary programs for the suggestions, it outputs a suggestion statement that determines which items should be disposed of. This information is passed to the presentation mechanism.

[0524] Step 5:

[0525] The user's device (smartphone, tablet, etc.) notifies the user of suggestions sent from the server via a display mechanism. It receives the suggestion text from the server as input and displays the notification on the user's device as output. The user reviews the suggestion and makes a choice to organize it as needed.

[0526] Step 6:

[0527] If the user approves the proposal, the server uses processing tools to perform the corresponding action (e.g., listing on an online trading platform or contacting a collector). The input is the user's approval action, and the output is the execution of the selected disposal method.

[0528] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0529] This invention provides a system that efficiently organizes and disposes of household items by offering suggestions based on frequency of use and taking into account the user's emotions. Specifically, it comprises information acquisition means, information analysis means, frequency of use estimation means, suggestion generation means, processing means, and an emotion engine. The system of this invention supports the user's decluttering process as follows.

[0530] First, the server periodically acquires images of the room through an AI camera, which is an information acquisition device installed in the room. These images are sent to the server, and objects within the images are identified by an information analysis device. The information of the identified objects is recorded in a database.

[0531] Next, the server uses a usage frequency estimation means to estimate the usage frequency of each object based on past data in the database. Based on this estimation result, the server uses a suggestion generation means to create suggestions regarding the organization or disposal of the items.

[0532] Furthermore, this system is equipped with an emotion engine that analyzes the user's emotional state when the user provides voice or video input to the system using a terminal. Once the user's emotions are collected, the emotion engine analyzes them to determine the user's current emotional state.

[0533] The results from the emotion engine are supplied to the suggestion generation system. The server adjusts the content and urgency of decluttering suggestions based on the emotional state. For example, if the user is stressed, the server will make simple suggestions and prioritize suggestions for items that are not used very often.

[0534] When a user reviews and approves a proposal using their device and submits feedback, the server automatically lists the selected unwanted items for sale via a processing system and arranges for their disposal in cooperation with a recycling company. This allows users to easily declutter their living space without having to go through cumbersome procedures.

[0535] This system supports users' lives and, through emotionally considerate suggestions, makes it possible to accept tidiness and organization more favorably.

[0536] The following describes the processing flow.

[0537] Step 1:

[0538] The server activates an AI camera, which is a means of acquiring information, and takes pictures of the room at regular intervals. The server sends these images to a database and stores them there.

[0539] Step 2:

[0540] The server activates the information analysis system and recognizes objects in the room from the acquired images. It analyzes information such as the type, location, and size of the recognized objects and records it in a database.

[0541] Step 3:

[0542] The server uses a usage frequency estimation method to analyze movement and change information of objects stored in the database to estimate the usage frequency of each object. Based on this, it identifies which items may be unnecessary.

[0543] Step 4:

[0544] The server uses an emotion engine to analyze the user's voice and video data obtained from the terminal to assess the user's emotional state. For example, it can determine whether the user is relaxed or stressed.

[0545] Step 5:

[0546] The server utilizes a suggestion generation mechanism to create optimal decluttering suggestions based on object usage frequency data and the user's emotional state. These suggestions are then displayed on the device, specifically informing the user, "It appears this garment hasn't been worn for three months. Why not consider getting rid of it now?"

[0547] Step 6:

[0548] The user reviews the proposal using a device and approves or modifies the process. For example, the user decides on the final disposal method by agreeing to the proposal or selecting an alternative option.

[0549] Step 7:

[0550] Upon receiving user instructions, the server activates the processing system and automatically lists unwanted items online. Alternatively, the server coordinates with recycling companies or donation recipients via a linked system to ensure efficient disposal of items. The user is then notified when disposal is complete.

[0551] (Example 2)

[0552] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0553] Organizing belongings and disposing of unwanted items within a household requires considerable time, effort, and can also be emotionally burdensome. In particular, appropriate suggestions that consider the frequency of use of individual items and the user's feelings at the time are required, but conventional systems often fail to adequately address these issues.

[0554] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0555] In this invention, the server includes a recording means that operates as an information acquisition means, an analysis means that analyzes recorded images to classify objects, a frequency estimation means that estimates the frequency of use of objects, an emotion analysis means that analyzes the user's emotions from audio or video, an adjustment means that feeds back the results of the emotion analysis means to a suggestion means to adjust the suggested content, and a processing means that processes unwanted items based on the suggestions. This makes it possible to efficiently and appropriately organize items and dispose of unwanted items while taking into account the user's emotional state.

[0556] "Information acquisition means" refers to means of collecting data necessary to understand the items in a room, and includes cameras, etc.

[0557] "Analysis means" refers to means for analyzing collected data and identifying and classifying objects within an image.

[0558] A "frequency estimation method" is a means for calculating and predicting the frequency of use of an item based on historical information accumulated in a database.

[0559] "Suggestion methods" refer to means for creating suggestions that advise on organizing or disposing of items, taking into account the frequency of use of the items and other factors.

[0560] An "emotion analysis tool" is a means of analyzing a user's emotional state from their voice or video and providing that information.

[0561] "Adjustment methods" refer to means of adjusting the proposed content based on analyzed emotional information and selecting an approach that suits the user's current situation.

[0562] "Processing means" refers to means for processing unwanted items in a specific manner based on the proposal, such as automating procedures for listing items for sale or collection.

[0563] This system assists in the efficient organization and disposal of items within the home and consists of server, terminal, and user roles.

[0564] The server uses an AI camera installed in the room as a means of acquiring information. This camera can be, for example, a general surveillance camera or a smart home device, and it periodically records the room's state. The acquired images are sent to the server, where analysis software such as OpenCV or TensorFlow is used to identify and classify objects within the images. The identification information for each object is stored in a database.

[0565] The server applies machine learning techniques based on accumulated information and estimates the frequency of item use using software frameworks such as scikit-learn and PyTorch. The estimation results are output as suggestions to the user through a suggestion generation system using natural language generation tools.

[0566] The user receives this suggestion through their device. The device allows for voice and video input to analyze the user's emotions. Amazon Polly or general speech recognition software is used for emotion analysis, and the server receives the obtained emotion information and uses it to adjust the suggestion.

[0567] For example, if a frying pan that hasn't been used for a long time is identified in a user's living space, it will be judged as having low usage frequency. The server will then provide the user with a suggestion such as, "This frying pan hasn't been used for over six months. If you don't need it, consider disposing of it." If the user approves the suggestion, the server may proceed with the process of automatically listing the frying pan on an online platform.

[0568] An example of a prompt for the generating AI model is an input such as, "A scenario that generates decluttering suggestions when the user is feeling stressed." In this way, it is possible to support the creation of a comfortable space while taking into account the user's lifestyle and emotions.

[0569] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0570] Step 1:

[0571] The server periodically acquires images of the room using an AI camera installed in the room. The image data obtained by the camera is input to the server. The server processes the data through analysis software to identify objects in the images and outputs the identified object information to a database.

[0572] Step 2:

[0573] The server uses machine learning algorithms to estimate the frequency of use of each item, referencing historical data based on item information in the database. The input data undergoes frequency analysis using scikit-learn or PyTorch to produce output representing the usage frequency of each item.

[0574] Step 3:

[0575] The server takes usage frequency data as input and runs a suggestion generation program to generate suggestions for organization or disposal for the user. The suggestion text is generated using natural language generation technology and output to the user's terminal.

[0576] Step 4:

[0577] The user receives generated suggestions through their device. Emotional information is obtained by the user inputting their voice or video, which activates the emotion analysis system. The input emotional data is analyzed through speech recognition software and output to the server as information about the emotional state.

[0578] Step 5:

[0579] The server receives the results of the sentiment analysis as input and provides feedback to the suggestion generation system, adjusting the suggested content according to the user's emotional state. The suggested content, taking the emotional state into account, is then output again to the user's terminal.

[0580] Step 6:

[0581] The user reviews the proposal and sends feedback to the server indicating approval or rejection. For approved proposals, the server automatically proceeds with the processing of the unwanted items using listing and integration methods. Listings are output as data to the online platform, and integrations are output as information to the recycling company.

[0582] (Application Example 2)

[0583] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0584] The problem this invention aims to solve is to provide more individualized and appropriate suggestions for organizing household items, not just based on frequency of use, but also tailored to the user's emotions and circumstances. This will enable users to organize their belongings efficiently while reducing their psychological burden.

[0585] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0586] In this invention, the server includes information acquisition means for acquiring images, information analysis means for analyzing acquired images and recognizing objects, usage frequency estimation means for estimating the frequency of use of objects, state analysis means for analyzing the user's state, suggestion adjustment means for adjusting the suggestion content based on the user's state, and processing means for processing unnecessary items based on the suggestion. This makes it possible to provide suggestions that take into account the emotional state of each individual user.

[0587] "Information acquisition means for acquiring images" refers to means of capturing image data using cameras or sensors that are installed in an environment such as a room.

[0588] "Information analysis means for analyzing acquired images to recognize objects" refers to means for identifying objects from acquired image data using artificial intelligence or the like, and determining their characteristics and type.

[0589] A "usage frequency estimation means for estimating the usage frequency of an object" is a means for calculating and predicting how often a particular object is used, based on past data.

[0590] A "state analysis method for analyzing the user's condition" is a means of analyzing and understanding the user's emotions and mental state in real time.

[0591] A "proposal adjustment mechanism that adjusts the content of suggestions based on the user's state" is a means of changing the content of suggestions regarding the optimal organization and disposal of items according to the user's emotions and state.

[0592] "Processing means for disposing of unwanted items based on proposals" refers to means for automatically or semi-automatically organizing and disposing of items proposed by the user.

[0593] This invention provides a system for efficiently organizing belongings within a user's living space. This system utilizes AI-powered advanced analytical capabilities to achieve a personalized approach to organizing belongings that suits the user's lifestyle.

[0594] First, the server periodically acquires images of the room using an AI camera as an information acquisition tool. The acquired images are sent to the server, and objects are recognized by an information analysis tool. This recognized object information is stored in a data recording device and used for subsequent processing.

[0595] The server then uses an object usage frequency estimation mechanism to estimate how often an object is used. This estimation is based on past usage data and serves as a basis for evaluating the importance and priority of the item.

[0596] The user's device analyzes the user's voice and facial expressions to understand their emotional state at that time. The state analysis means can determine the user's emotions from these inputs and identify everyday stress, excitement, etc.

[0597] The acquired emotional data is supplied to the suggestion adjustment mechanism. The server generates coping suggestions based on the user's emotional state and past usage frequency, and displays them on the user's terminal. The content of the suggestions is designed to reduce the user's psychological burden. For example, if the user is stressed, easily actionable coping suggestions are prioritized.

[0598] As a concrete example, consider a situation where a user wants to try a new hobby and needs to secure workspace. In this case, the system prioritizes organizing the items necessary for the hobby and makes suggestions, including the collection and disposal of unnecessary items.

[0599] An example of a prompt for a generative AI model would be: "Create an example of an AI dialogue that suggests tidying up when the user is relaxed."

[0600] These features allow users to organize their living spaces comfortably and efficiently, enabling them to live with greater peace of mind.

[0601] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0602] Step 1:

[0603] The server uses an AI camera to acquire images of the room.

[0604] The input is video data from a camera.

[0605] The output is image data showing the current state of the room. The server stores this image data in a database for subsequent analysis.

[0606] Step 2:

[0607] The server uses information analysis tools to recognize objects from the acquired images.

[0608] The input is the image data obtained in step 1.

[0609] The output is a list of identified objects. The server uses image analysis techniques to analyze the shape, color, and arrangement of the objects, and records the characteristics of each object in a database.

[0610] Step 3:

[0611] The server uses a usage frequency estimation means to estimate the usage frequency of the identified object.

[0612] The input consists of past object usage history and object recognition data.

[0613] The output is usage frequency information for each object. The server performs time-series data analysis to understand the usage trends of the objects and calculate the frequency.

[0614] Step 4:

[0615] The device analyzes the user's voice and facial expressions to evaluate the user's emotional state.

[0616] The input consists of voice data and facial expression images from the user.

[0617] The output is data indicating the user's emotional state. The device uses a combination of voice processing technology and facial recognition to estimate emotions and sends the results to the server.

[0618] Step 5:

[0619] The server uses a suggestion adjustment mechanism to generate organized suggestions based on usage frequency information and emotional state.

[0620] The input consists of information on the frequency of use of objects and data on the user's emotional state.

[0621] The output consists of tailored sorting and disposal suggestions. The server utilizes an AI model to generate and send optimal suggestions to each user's device.

[0622] Step 6:

[0623] Users review the suggestions on their devices and send feedback as needed.

[0624] The input is suggested data from the server.

[0625] The output is user feedback data. Users provide feedback based on the displayed suggestions, and this information is used to improve future suggestions.

[0626] Step 7:

[0627] The server uses processing means to process unwanted items based on the proposal.

[0628] The input is user feedback data.

[0629] The output shows the completion status of the waste disposal process. The server disposes of items approved by the user in cooperation with online marketplaces and recycling companies, and reports the processing results.

[0630] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0631] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0632] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0633] [Fourth Embodiment]

[0634] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0635] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0636] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0637] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0638] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0639] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0640] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0641] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0642] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0643] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0644] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0645] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0646] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0647] This invention provides a system for monitoring the usage of items within a household and efficiently decluttering. This system comprises information acquisition means, information analysis means, usage frequency estimation means, suggestion generation means, and processing means. A specific example of a system using these means is shown below.

[0648] First, the server periodically acquires images using an information acquisition device, namely an AI camera, installed in the room. The server receives this image data and uses an information analysis device to identify and recognize objects within the images. At this time, features such as the type and shape of the identified objects are also extracted and recorded in a database.

[0649] Next, the server uses a usage frequency estimation mechanism to calculate the usage frequency of each object based on the information recorded in the database. Based on this information, the server decides which items should be decluttered.

[0650] Based on the results, the server uses a suggestion generation mechanism to generate the optimal decluttering plan for the user. The server notifies the user of this suggestion via the terminal, and the user can review and modify the suggestion. For example, the server might present the user with the suggestion, "These clothes may not have been worn in the past six months. Shall we declutter them?"

[0651] Furthermore, if the user approves the proposal, the server automatically lists the unwanted items on the online platform using processing tools. The server also supports the process of arranging for the items to be sent to a recycling company or donation recipient by utilizing various integration tools.

[0652] In this way, the present invention enables efficient organization and disposal of items, making it easier to maintain a comfortable living space. Users can practice decluttering in their daily lives without feeling burdened.

[0653] The following describes the processing flow.

[0654] Step 1:

[0655] The server controls AI cameras, which are information acquisition tools installed in the room, and acquires images at specific time intervals. The server receives these image data in real time and stores them in a database.

[0656] Step 2:

[0657] The server analyzes the received image data using information analysis tools. It recognizes objects within the image and identifies the type and location of each object. The server analyzes this recognition information and records the characteristics of the objects in a database.

[0658] Step 3:

[0659] The server operates a usage frequency estimation system and calculates the current usage frequency based on the previously recorded usage history of the object. Specifically, it evaluates whether the object is being used by taking into account changes in the object's position and exposure time. The results are reflected in the database.

[0660] Step 4:

[0661] The server uses a suggestion generation mechanism to generate a decluttering plan for the user based on usage frequency data. The terminal displays this plan and sends a notification, such as "This item hasn't been used recently, so we suggest disposing of it." The terminal accepts user confirmation or modifications.

[0662] Step 5:

[0663] After receiving user approval, the server uses processing tools to handle the disposal of unwanted items. Specifically, a listing process is initiated to automatically list the unwanted items on an online platform.

[0664] Step 6:

[0665] Furthermore, using collaborative processing methods, the system can arrange for unwanted items to be sent to recycling companies or donation recipients if the user so desires. The server manages these collaborations with external services, supporting the efficient disposal of unwanted items.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] In modern living environments, a large number of items accumulate in homes, posing a challenge in organizing and disposing of them. This can lead to cramped living spaces and a loss of comfort. Furthermore, the effort required to choose the optimal disposal method for unwanted items makes item management inefficient. Therefore, there is a need for efficient methods to manage and appropriately dispose of items accumulated in homes.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes: an image acquisition means that periodically acquires images of items using an information acquisition device installed in the room to acquire images; an information analysis means that analyzes the acquired image data using computing resources on the cloud to recognize the items; a usage frequency estimation means that records the type and characteristics of the recognized items and estimates the frequency of use of the items based on that data; a proposal generation means that uses a generation AI model to propose organizing or disposing of the items based on the estimated usage frequency; and a processing means that processes unwanted items after receiving user confirmation based on the proposal. This enables efficient management of items in living spaces and prompt and appropriate disposal.

[0671] An "information acquisition device" is a device installed in a room to acquire images of a target object.

[0672] "Image acquisition means" refers to a function that periodically acquires images of an object via an information acquisition device.

[0673] "Information analysis means" refers to a function that uses computing resources on the cloud to analyze acquired image data and recognize objects.

[0674] "Usage frequency estimation means" refers to a function that records the type and characteristics of recognized items as data and estimates the frequency of use of those items based on that data.

[0675] The "proposal generation means" is a function that uses a generation AI model to suggest to the user how to organize or dispose of items based on the estimated frequency of use.

[0676] "Processing means" refers to a function that processes unwanted items based on proposals presented by the proposal generation means, after receiving user confirmation.

[0677] The "listing function" is a feature that, after user approval, automatically lists unwanted items on an online exchange platform.

[0678] "Collaboration function" refers to a function that allows the processing system to dispose of unwanted items in cooperation with collection companies or donation recipients.

[0679] This invention is a system that supports the management, efficient organization, and disposal of household items. The server periodically acquires images of items using an information acquisition device installed in the room. Specifically, a general-purpose camera module is used as hardware, and the captured image data is transmitted to computing resources in the cloud.

[0680] The server sends this image data to an information analysis tool in the cloud, where it performs image analysis using common machine learning libraries, including Google TensorFlow. As a result, objects are recognized, and features such as the type and shape of the recognized objects are extracted. This information is recorded in a database, and database services such as Amazon RDS can be used.

[0681] Next, the server uses a usage frequency estimation mechanism to estimate the frequency of use of items based on the recorded information. This process uses the Python Pandas library to analyze item usage patterns based on past analysis data.

[0682] Regarding the suggestion generation method, OpenAI's generative AI model is used to suggest decluttering to the user based on acquired usage frequency data. For example, it generates a natural language prompt such as, "This cup hasn't been used in the last three months. Please consider decluttering it."

[0683] Users can receive and confirm suggestions via their devices. Based on feedback from the devices, the server executes the procedures for processing unwanted items. Using the processing methods, after user approval, unwanted items are automatically listed on online exchange platforms or a disposal plan is created. Coordination with recycling companies and donation recipients is also considered, and efficient communication and arrangements are made via the network.

[0684] A concrete example of a prompt message for a generative AI model would be: "Create a phrase that reports examples of items in the user's living space that haven't been used recently, and asks if the user is willing to organize those items."

[0685] This system allows users to efficiently manage their belongings and maintain a comfortable living space.

[0686] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0687] Step 1:

[0688] The server periodically acquires images of items using an information acquisition device (AI camera) installed in the room. The input is real-time image data acquired by the camera, and the output is an image file, such as JPEG, sent to the server for analysis. This image acquisition is performed automatically at a specified time each day.

[0689] Step 2:

[0690] The server sends image data to the cloud. Here, the input image data is transferred to the cloud analysis service using a secure protocol. The output is image data waiting on the cloud. Specifically, the image is securely transmitted using SSL / TLS and added to the analysis queue.

[0691] Step 3:

[0692] The server uses cloud-based information analysis tools to analyze image data and recognize objects. The input is the image data acquired in step 2, and the output is the analysis results, including the type and characteristics of the recognized objects. This process uses a machine learning library to label the types of objects. Specifically, objects are classified as "books," "clothing," "dishes," etc., and their characteristics such as shape and color are extracted.

[0693] Step 4:

[0694] The server records the recognized item data in the database. The input is the item analysis result generated in step 3, and the output is the updated database entry. Here, the ID, type, characteristics, and date / time information of each item are stored in Amazon RDS. Specifically, data is inserted using SQL queries.

[0695] Step 5:

[0696] The server uses a usage frequency estimation method to analyze recorded data and calculate the usage frequency of items. The input is historical item usage data stored in a database, and the output is an estimated usage frequency for each item. Using the Pandas library, the usage history of each item is analyzed to derive conclusions such as "This shirt has been used 3 times in the past 30 days."

[0697] Step 6:

[0698] The server uses an AI model to generate suggestions for the user based on item usage frequency data. The input is the estimated usage frequency, and the output is suggestions for decluttering. For example, it might generate a prompt such as, "This jacket has hardly been used in the last 6 months. Please consider getting rid of it."

[0699] Step 7:

[0700] The server generates a proposal and notifies the user's device. The input is the proposal content generated in step 6, and the output is a push notification to the user's device. The user can review and approve the proposal through the push notification. This operation is performed using the mobile app notification service.

[0701] Step 8:

[0702] If the user approves the proposal, the server processes the unwanted items. Specifically, the input is the user's approval, and the output is the procedure for listing the unwanted items on an online exchange platform or handing them over to a recycling company. At this point, product information is registered and listed via the online platform API, and the process is completed.

[0703] (Application Example 1)

[0704] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0705] The challenge lies in addressing the difficulty of efficiently utilizing living space, which arises from the increasing number of items in the home and the complexity of managing them. In particular, the lack of identification of unnecessary items and a plan for their disposal is a factor that reduces the quality of life.

[0706] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0707] In this invention, the server includes information gathering means, information analysis means, usage frequency estimation means, suggestion generation means, processing means, and presentation means. This enables effective organization and waste-free disposal of household items.

[0708] "Information gathering means" refers to devices or functions that acquire images, which are installed in home autonomous devices to understand the situation of an object.

[0709] "Information analysis means" refers to techniques and methods for processing acquired image data and identifying specific objects.

[0710] A "usage frequency estimation means" is a method or device for analyzing the handling and usage conditions of an object and calculating its usage frequency.

[0711] The "proposal generation method" is a function that creates recommendations for organizing and disposing of items based on usage frequency data.

[0712] "Processing means" refers to the means of implementation for carrying out specific disposal or management actions based on the proposal.

[0713] "Presentation means" refers to a function that uses a home-use autonomous device to display or notify the user of suggested content.

[0714] An "online trading platform" is a general term for places and services where goods are bought and sold via the internet.

[0715] "Cooperative methods" refer to means of efficiently disposing of unnecessary items in cooperation with collection companies and donation recipients.

[0716] To realize this invention, the following configuration and technology are necessary for the autonomous home device. The server is equipped with an AI camera as an information gathering means and periodically photographs items in the home. This acquired image data is transmitted to the server and then analyzed using image analysis software such as TensorFlow as an information analysis means. This makes it possible to identify and distinguish objects.

[0717] The server calculates the frequency of use of the analyzed objects using a usage frequency estimation means. Based on the usage frequency data, a suggestion generation means operates to determine which items should be sorted or disposed of, and proposes the results to the user. The suggestion to the user is presented by a presentation means provided by the home autonomous device. This presentation means can also notify the user via a terminal such as a smartphone or tablet.

[0718] (Specific example)

[0719] For example, the server might notify the user's device with a suggestion such as, "These dishes appear to have not been used in the past year. Would you like to consider disposing of them or donating them?" If the user approves this suggestion, the unwanted items are automatically listed on an online trading platform or, if necessary, contacted by a collection agency or other relevant party through a collaborative means. An example of a specific prompt might be, "Identify items that have not been used in the past six months and generate a decluttering suggestion." This system not only allows users to efficiently organize their living spaces but also helps them clearly distinguish between what they need and what they don't.

[0720] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0721] Step 1:

[0722] The server uses an AI camera as a means of information gathering to periodically acquire images of items in the home. The AI ​​camera provides images of the home environment as input. These images are output directly as digital data and sent to the server.

[0723] Step 2:

[0724] The server processes the received image data using image analysis software such as TensorFlow. Image data is sent to the server as input, and information for identifying objects (such as their shape and color) is extracted from this data. The extracted information is then recorded in a database.

[0725] Step 3:

[0726] The server calculates the usage frequency of each item using a usage frequency estimation means, based on data obtained by the information analysis means. It uses previously recorded item characteristic data as input and performs calculations using a usage frequency model. As a result of the calculation, it outputs usage frequency information for each item and passes it to the proposal generation means.

[0727] Step 4:

[0728] The server uses a suggestion generation mechanism based on usage frequency information to generate suggestions for organization and disposal for the user. The input is usage frequency data for each item, and by executing the necessary programs for the suggestions, it outputs a suggestion statement that determines which items should be disposed of. This information is passed to the presentation mechanism.

[0729] Step 5:

[0730] The user's device (smartphone, tablet, etc.) notifies the user of suggestions sent from the server via a display mechanism. It receives the suggestion text from the server as input and displays the notification on the user's device as output. The user reviews the suggestion and makes a choice to organize it as needed.

[0731] Step 6:

[0732] If the user approves the proposal, the server uses processing tools to perform the corresponding action (e.g., listing on an online trading platform or contacting a collector). The input is the user's approval action, and the output is the execution of the selected disposal method.

[0733] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0734] This invention provides a system that efficiently organizes and disposes of household items by offering suggestions based on frequency of use and taking into account the user's emotions. Specifically, it comprises information acquisition means, information analysis means, frequency of use estimation means, suggestion generation means, processing means, and an emotion engine. The system of this invention supports the user's decluttering process as follows.

[0735] First, the server periodically acquires images of the room through an AI camera, which is an information acquisition device installed in the room. These images are sent to the server, and objects within the images are identified by an information analysis device. The information of the identified objects is recorded in a database.

[0736] Next, the server uses a usage frequency estimation means to estimate the usage frequency of each object based on past data in the database. Based on this estimation result, the server uses a suggestion generation means to create suggestions regarding the organization or disposal of the items.

[0737] Furthermore, this system is equipped with an emotion engine that analyzes the user's emotional state when the user provides voice or video input to the system using a terminal. Once the user's emotions are collected, the emotion engine analyzes them to determine the user's current emotional state.

[0738] The results from the emotion engine are supplied to the suggestion generation system. The server adjusts the content and urgency of decluttering suggestions based on the emotional state. For example, if the user is stressed, the server will make simple suggestions and prioritize suggestions for items that are not used very often.

[0739] When a user reviews and approves a proposal using their device and submits feedback, the server automatically lists the selected unwanted items for sale via a processing system and arranges for their disposal in cooperation with a recycling company. This allows users to easily declutter their living space without having to go through cumbersome procedures.

[0740] This system supports users' lives and, through emotionally considerate suggestions, makes it possible to accept tidiness and organization more favorably.

[0741] The following describes the processing flow.

[0742] Step 1:

[0743] The server activates an AI camera, which is a means of acquiring information, and takes pictures of the room at regular intervals. The server sends these images to a database and stores them there.

[0744] Step 2:

[0745] The server activates the information analysis system and recognizes objects in the room from the acquired images. It analyzes information such as the type, location, and size of the recognized objects and records it in a database.

[0746] Step 3:

[0747] The server uses a usage frequency estimation method to analyze movement and change information of objects stored in the database to estimate the usage frequency of each object. Based on this, it identifies which items may be unnecessary.

[0748] Step 4:

[0749] The server uses an emotion engine to analyze the user's voice and video data obtained from the terminal to assess the user's emotional state. For example, it can determine whether the user is relaxed or stressed.

[0750] Step 5:

[0751] The server utilizes a suggestion generation mechanism to create optimal decluttering suggestions based on object usage frequency data and the user's emotional state. These suggestions are then displayed on the device, specifically informing the user, "It appears this garment hasn't been worn for three months. Why not consider getting rid of it now?"

[0752] Step 6:

[0753] The user reviews the proposal using a device and approves or modifies the process. For example, the user decides on the final disposal method by agreeing to the proposal or selecting an alternative option.

[0754] Step 7:

[0755] Upon receiving user instructions, the server activates the processing system and automatically lists unwanted items online. Alternatively, the server coordinates with recycling companies or donation recipients via a linked system to ensure efficient disposal of items. The user is then notified when disposal is complete.

[0756] (Example 2)

[0757] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0758] Organizing belongings and disposing of unwanted items within a household requires considerable time, effort, and can also be emotionally burdensome. In particular, appropriate suggestions that consider the frequency of use of individual items and the user's feelings at the time are required, but conventional systems often fail to adequately address these issues.

[0759] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0760] In this invention, the server includes a recording means that operates as an information acquisition means, an analysis means that analyzes recorded images to classify objects, a frequency estimation means that estimates the frequency of use of objects, an emotion analysis means that analyzes the user's emotions from audio or video, an adjustment means that feeds back the results of the emotion analysis means to a suggestion means to adjust the suggested content, and a processing means that processes unwanted items based on the suggestions. This makes it possible to efficiently and appropriately organize items and dispose of unwanted items while taking into account the user's emotional state.

[0761] "Information acquisition means" refers to means of collecting data necessary to understand the items in a room, and includes cameras, etc.

[0762] "Analysis means" refers to means for analyzing collected data and identifying and classifying objects within an image.

[0763] A "frequency estimation method" is a means for calculating and predicting the frequency of use of an item based on historical information accumulated in a database.

[0764] "Suggestion methods" refer to means for creating suggestions that advise on organizing or disposing of items, taking into account the frequency of use of the items and other factors.

[0765] An "emotion analysis tool" is a means of analyzing a user's emotional state from their voice or video and providing that information.

[0766] "Adjustment methods" refer to means of adjusting the proposed content based on analyzed emotional information and selecting an approach that suits the user's current situation.

[0767] "Processing means" refers to means for processing unwanted items in a specific manner based on the proposal, such as automating procedures for listing items for sale or collection.

[0768] This system assists in the efficient organization and disposal of items within the home and consists of server, terminal, and user roles.

[0769] The server uses an AI camera installed in the room as a means of acquiring information. This camera can be, for example, a general surveillance camera or a smart home device, and it periodically records the room's state. The acquired images are sent to the server, where analysis software such as OpenCV or TensorFlow is used to identify and classify objects within the images. The identification information for each object is stored in a database.

[0770] The server applies machine learning techniques based on accumulated information and estimates the frequency of item use using software frameworks such as scikit-learn and PyTorch. The estimation results are output as suggestions to the user through a suggestion generation system using natural language generation tools.

[0771] The user receives this suggestion through their device. The device allows for voice and video input to analyze the user's emotions. Amazon Polly or general speech recognition software is used for emotion analysis, and the server receives the obtained emotion information and uses it to adjust the suggestion.

[0772] For example, if a frying pan that hasn't been used for a long time is identified in a user's living space, it will be judged as having low usage frequency. The server will then provide the user with a suggestion such as, "This frying pan hasn't been used for over six months. If you don't need it, consider disposing of it." If the user approves the suggestion, the server may proceed with the process of automatically listing the frying pan on an online platform.

[0773] An example of a prompt for the generating AI model is an input such as, "A scenario that generates decluttering suggestions when the user is feeling stressed." In this way, it is possible to support the creation of a comfortable space while taking into account the user's lifestyle and emotions.

[0774] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0775] Step 1:

[0776] The server periodically acquires images of the room using an AI camera installed in the room. The image data obtained by the camera is input to the server. The server processes the data through analysis software to identify objects in the images and outputs the identified object information to a database.

[0777] Step 2:

[0778] The server uses machine learning algorithms to estimate the frequency of use of each item, referencing historical data based on item information in the database. The input data undergoes frequency analysis using scikit-learn or PyTorch to produce output representing the usage frequency of each item.

[0779] Step 3:

[0780] The server takes usage frequency data as input and runs a suggestion generation program to generate suggestions for organization or disposal for the user. The suggestion text is generated using natural language generation technology and output to the user's terminal.

[0781] Step 4:

[0782] The user receives generated suggestions through their device. Emotional information is obtained by the user inputting their voice or video, which activates the emotion analysis system. The input emotional data is analyzed through speech recognition software and output to the server as information about the emotional state.

[0783] Step 5:

[0784] The server receives the results of the sentiment analysis as input and provides feedback to the suggestion generation system, adjusting the suggested content according to the user's emotional state. The suggested content, taking the emotional state into account, is then output again to the user's terminal.

[0785] Step 6:

[0786] The user reviews the proposal and sends feedback to the server indicating approval or rejection. For approved proposals, the server automatically proceeds with the processing of the unwanted items using listing and integration methods. Listings are output as data to the online platform, and integrations are output as information to the recycling company.

[0787] (Application Example 2)

[0788] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0789] The problem this invention aims to solve is to provide more individualized and appropriate suggestions for organizing household items, not just based on frequency of use, but also tailored to the user's emotions and circumstances. This will enable users to organize their belongings efficiently while reducing their psychological burden.

[0790] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0791] In this invention, the server includes information acquisition means for acquiring images, information analysis means for analyzing acquired images and recognizing objects, usage frequency estimation means for estimating the frequency of use of objects, state analysis means for analyzing the user's state, suggestion adjustment means for adjusting the suggestion content based on the user's state, and processing means for processing unnecessary items based on the suggestion. This makes it possible to provide suggestions that take into account the emotional state of each individual user.

[0792] "Information acquisition means for acquiring images" refers to means of capturing image data using cameras or sensors that are installed in an environment such as a room.

[0793] "Information analysis means for analyzing acquired images to recognize objects" refers to means for identifying objects from acquired image data using artificial intelligence or the like, and determining their characteristics and type.

[0794] A "usage frequency estimation means for estimating the usage frequency of an object" is a means for calculating and predicting how often a particular object is used, based on past data.

[0795] A "state analysis method for analyzing the user's condition" is a means of analyzing and understanding the user's emotions and mental state in real time.

[0796] A "proposal adjustment mechanism that adjusts the content of suggestions based on the user's state" is a means of changing the content of suggestions regarding the optimal organization and disposal of items according to the user's emotions and state.

[0797] "Processing means for disposing of unwanted items based on proposals" refers to means for automatically or semi-automatically organizing and disposing of items proposed by the user.

[0798] This invention provides a system for efficiently organizing belongings within a user's living space. This system utilizes AI-powered advanced analytical capabilities to achieve a personalized approach to organizing belongings that suits the user's lifestyle.

[0799] First, the server periodically acquires images of the room using an AI camera as an information acquisition tool. The acquired images are sent to the server, and objects are recognized by an information analysis tool. This recognized object information is stored in a data recording device and used for subsequent processing.

[0800] The server then uses an object usage frequency estimation mechanism to estimate how often an object is used. This estimation is based on past usage data and serves as a basis for evaluating the importance and priority of the item.

[0801] The user's device analyzes the user's voice and facial expressions to understand their emotional state at that time. The state analysis means can determine the user's emotions from these inputs and identify everyday stress, excitement, etc.

[0802] The acquired emotional data is supplied to the suggestion adjustment mechanism. The server generates coping suggestions based on the user's emotional state and past usage frequency, and displays them on the user's terminal. The content of the suggestions is designed to reduce the user's psychological burden. For example, if the user is stressed, easily actionable coping suggestions are prioritized.

[0803] As a concrete example, consider a situation where a user wants to try a new hobby and needs to secure workspace. In this case, the system prioritizes organizing the items necessary for the hobby and makes suggestions, including the collection and disposal of unnecessary items.

[0804] An example of a prompt for a generative AI model would be: "Create an example of an AI dialogue that suggests tidying up when the user is relaxed."

[0805] These features allow users to organize their living spaces comfortably and efficiently, enabling them to live with greater peace of mind.

[0806] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0807] Step 1:

[0808] The server uses an AI camera to acquire images of the room.

[0809] The input is video data from a camera.

[0810] The output is image data showing the current state of the room. The server stores this image data in a database for subsequent analysis.

[0811] Step 2:

[0812] The server uses information analysis tools to recognize objects from the acquired images.

[0813] The input is the image data obtained in step 1.

[0814] The output is a list of identified objects. The server uses image analysis techniques to analyze the shape, color, and arrangement of the objects, and records the characteristics of each object in a database.

[0815] Step 3:

[0816] The server uses a usage frequency estimation means to estimate the usage frequency of the identified object.

[0817] The input consists of past object usage history and object recognition data.

[0818] The output is usage frequency information for each object. The server performs time-series data analysis to understand the usage trends of the objects and calculate the frequency.

[0819] Step 4:

[0820] The device analyzes the user's voice and facial expressions to evaluate the user's emotional state.

[0821] The input consists of voice data and facial expression images from the user.

[0822] The output is data indicating the user's emotional state. The device uses a combination of voice processing technology and facial recognition to estimate emotions and sends the results to the server.

[0823] Step 5:

[0824] The server uses a suggestion adjustment mechanism to generate organized suggestions based on usage frequency information and emotional state.

[0825] The input consists of information on the frequency of use of objects and data on the user's emotional state.

[0826] The output consists of tailored sorting and disposal suggestions. The server utilizes an AI model to generate and send optimal suggestions to each user's device.

[0827] Step 6:

[0828] Users review the suggestions on their devices and send feedback as needed.

[0829] The input is suggested data from the server.

[0830] The output is user feedback data. Users provide feedback based on the displayed suggestions, and this information is used to improve future suggestions.

[0831] Step 7:

[0832] The server uses processing means to process unwanted items based on the proposal.

[0833] The input is user feedback data.

[0834] The output shows the completion status of the waste disposal process. The server disposes of items approved by the user in cooperation with online marketplaces and recycling companies, and reports the processing results.

[0835] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0836] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0837] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0838] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0839] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0840] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0841] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0842] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0843] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0844] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0845] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0846] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0847] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0848] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0849] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0850] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0851] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0852] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0853] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0854] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0855] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0856] The following is further disclosed regarding the embodiments described above.

[0857] (Claim 1)

[0858] Information acquisition means for acquiring images,

[0859] Information analysis means for analyzing acquired images to recognize objects,

[0860] A means for estimating the frequency of use of an object,

[0861] A proposal generation means that proposes the organization or disposal of an object based on its frequency of use,

[0862] A processing means for disposing of waste based on the proposal,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, wherein the processing means comprises listing means for automatically listing proposed unwanted items on an online platform.

[0866] (Claim 3)

[0867] The system according to claim 1, wherein the processing means comprises a coordinating means for disposing of the proposed waste in cooperation with a collection company or a donation recipient.

[0868] "Example 1"

[0869] (Claim 1)

[0870] An image acquisition means that periodically acquires images of objects using an information acquisition device installed in a room to acquire images,

[0871] An information analysis means that analyzes image data acquired using computing resources on the cloud to recognize objects,

[0872] A means for estimating the frequency of use of an item, which records the type and characteristics of the recognized item and estimates the frequency of use of the item based on that data,

[0873] A proposal generation means that uses a generative AI model to propose the organization or disposal of an object based on its estimated usage frequency,

[0874] Based on the proposal, a processing means for disposing of unwanted materials after user confirmation,

[0875] A system that includes this.

[0876] (Claim 2)

[0877] The system according to claim 1, further comprising a listing function in which, after user approval based on a proposal, the processing means automatically lists unwanted items on an online exchange medium.

[0878] (Claim 3)

[0879] The system according to claim 1, wherein the processing means has a coordinating function for disposing of proposed waste in cooperation with a collection company or a donation recipient.

[0880] "Application Example 1"

[0881] (Claim 1)

[0882] Information gathering means for acquiring images,

[0883] Information analysis means for analyzing acquired images to recognize the target,

[0884] A means for estimating the frequency of use of the target,

[0885] A proposal generation means that proposes the organization or disposal of an object based on its frequency of use,

[0886] A processing means for disposing of waste based on the proposal,

[0887] The processing means is a presentation means that presents the proposed content using a household autonomous device,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, wherein the processing means comprises a means for automatically listing the proposed unwanted items on an online trading platform.

[0891] (Claim 3)

[0892] The system according to claim 1, wherein the processing means comprises a means of cooperation for disposing of the proposed waste in cooperation with a collection company or a donation recipient.

[0893] "Example 2 of combining an emotion engine"

[0894] (Claim 1)

[0895] A recording means that operates as a means of acquiring information,

[0896] An analytical means for classifying objects by analyzing recorded images,

[0897] A frequency estimation means for estimating the frequency of use of an object,

[0898] A suggestion method for advising on the organization or disposal of objects based on their frequency of use,

[0899] An emotion analysis method that analyzes the user's emotions from audio or video,

[0900] An adjustment means that feeds back the results of the emotion analysis means to the proposal means and adjusts the proposed content,

[0901] A processing means for disposing of waste based on the proposal,

[0902] A system that includes this.

[0903] (Claim 2)

[0904] The system according to claim 1, comprising a means for automatically listing proposed unwanted items on an e-commerce platform.

[0905] (Claim 3)

[0906] The system according to claim 1, comprising a means for coordinating with an external organization to dispose of proposed waste materials.

[0907] "Application example 2 when combining with an emotional engine"

[0908] (Claim 1)

[0909] Information acquisition means for acquiring images,

[0910] Information analysis means for analyzing acquired images to recognize objects,

[0911] A means for estimating the frequency of use of an object,

[0912] A proposal generation means that proposes the organization or disposal of an object based on its frequency of use,

[0913] A state analysis means for analyzing the user's state,

[0914] A proposal adjustment mechanism that adjusts the proposal content based on the user's status,

[0915] A processing means for disposing of waste based on the proposal,

[0916] A system that includes this.

[0917] (Claim 2)

[0918] The system according to claim 1, wherein the processing means comprises a listing means for automatically listing proposed unwanted items on an information network.

[0919] (Claim 3)

[0920] The system according to claim 1, wherein the processing means comprises a coordinating means for disposing of proposed waste in cooperation with a collection company or a donation recipient. [Explanation of Symbols]

[0921] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Information gathering means for acquiring images, Information analysis means for analyzing acquired images to recognize the target, A means for estimating the frequency of use of the target, A proposal generation means that proposes the organization or disposal of an object based on its frequency of use, A processing means for disposing of waste based on the proposal, The processing means is a presentation means that presents the proposed content using a household autonomous device, A system that includes this.

2. The system according to claim 1, wherein the processing means comprises a means for automatically listing the proposed unwanted items on an online trading platform.

3. The system according to claim 1, wherein the processing means comprises a means of cooperation for disposing of the proposed waste in cooperation with a collection company or a donation recipient.

Images