System

Abstract

A system is provided.SOLUTION: A system including a terminal configured to capture an image, a server configured to transmit the image to the terminal, a server configured to receive the image and analyze the image using a generative model, a server configured to extract detailed information and a use history from the analysis, a server configured to determine an item to be discarded based on the extracted information, a server configured to transmit a proposal including a determination result and a cleanup method to the terminal, and a server configured to display the transmitted proposal and support a user to execute the proposal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The purpose of this invention is to solve the difficulty of sorting out and tidying up that is faced by busy working people, dual-income households, and companies that are unable to keep their offices tidy. In particular, there is a need for a means to provide efficient and convenient decluttering and tidying to address the problem of not being able to throw things away and causing pressure on living and office space. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a means for taking an image on a terminal, a means for transmitting the image to a server, a means for receiving the image on the server and analyzing it using a generative model, a means for extracting detailed information and usage history through the analysis, a means for determining items that should be decluttered based on the extracted information, a means for transmitting the determination results and a proposal including a tidying up method to the terminal, and a means for displaying the transmitted proposal and supporting the user in carrying out the proposal. The system also provides a means for automatically selecting a tidying up method based on usage history, a function for introducing links to sales sites for storage products and disposal of unwanted items, and a function for introducing tidying up support services. This allows users to easily declutter and achieve efficient organization and storage.

[0006] "Terminal" refers to a portable electronic device used by a user, and primarily includes smartphones and tablets.

[0007] "Means for taking images" refers to the functionality for taking pictures of items or objects using a camera built into the device.

[0008] A "server" is a computer system that stores and analyzes data over a network and can be accessed remotely.

[0009] A "generative model" refers to an algorithm or mathematical model that analyzes data and extracts information based on machine learning or deep learning.

[0010] "Means of analysis" refers to the process of inputting image data and extracting and recognizing specific information from that data.

[0011] "Detailed information" refers to specific data about the object, such as the title, author, and publication year for a book.

[0012] "Usage history" refers to data on how an object has been used in the past, such as frequency of use and the date and time of last use.

[0013] "Things to be decluttered" refers to items that are no longer needed in the user's life or work and should be thrown away or moved elsewhere.

[0014] "Tidying up method" refers to specific steps and methods for organizing, disposing of, or reusing unwanted items.

[0015] "Suggestions" refer to cleaning methods and related information that are shown to users based on the analysis results.

[0016] "Storage products" refers to various products used for the purpose of organizing and storing items.

[0017] "Unwanted item disposal" refers to the means and procedures for properly disposing of unwanted items.

[0018] "Sales Site" means a website that offers or sells products or services over the Internet.

[0019] "Clean-up support services" refers to services that include advice and actual work related to clean-up provided by professionals or specific services. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0042] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured. The device then compresses the image and sends it to the server.

[0043] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0044] The server determines whether the item should be decluttered based on the generated detailed information and usage history. If it is determined that the item should be decluttered, it selects the most appropriate method for the user to declutter. Methods include donating the item, selling it at a recycle shop, or taking it to a recycling center. This information is organized along with specific instructions and reference images and sent to the device.

[0045] The device will then display the received tidying method and reference images to the user. The user can refer to the displayed information to actually declutter and organize. For example, if a specific book is suggested as being sold at a recycle shop, the user can efficiently tidy up by following the suggestion and bringing the book. The app may also display links to related sales sites and tidying support services, which can be used to further smooth the process of decluttering.

[0046] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. It will then suggest selling the book at a recycle shop, and specific instructions, the store's location, and a sample image will be displayed on the device. The user can then use this information to go and sell the book.

[0047] In this way, the present invention provides a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[0048] The processing flow will be explained below.

[0049] Step 1:

[0050] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[0051] Step 2:

[0052] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[0053] Step 3:

[0054] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[0055] Step 4:

[0056] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[0057] Step 5:

[0058] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[0059] Step 6:

[0060] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[0061] Step 7:

[0062] The server determines whether an item should be discarded based on the extracted details and usage history. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[0063] Step 8:

[0064] If the server is determined to be an item that should be discarded, the system selects the most appropriate method for disposal, including donation or sale at a recycle shop, and provides detailed information and reference images.

[0065] Step 9:

[0066] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[0067] Step 10:

[0068] The device analyzes the cleaning method content and reference images received from the server and displays them on the user interface, allowing the user to check the specific cleaning steps and related images.

[0069] Step 11:

[0070] Users can refer to the tidying method displayed on the device and actually declutter and organize their belongings. For example, they can prepare items for donation according to the specified method.

[0071] Step 12:

[0072] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[0073] The above are the specific processing steps of this system.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] In modern society, the increase in the number of items has made decluttering and organizing important issues. However, many users often have difficulty deciding which items to dispose of and how. Furthermore, manually deciding how to organize items is time-consuming, laborious, and inefficient. Therefore, there is a need for a system that supports efficient decluttering and organizing.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes means for extracting detailed information about an item, means for determining whether the item should be decluttered based on the usage history, and means for selecting an optimal tidying method and transmitting it to the terminal, thereby enabling the user to declutter and organize efficiently.

[0079] A "terminal" is a device that a user directly operates to take an image of an item.

[0080] The "means for taking images" is a function for taking high-resolution images of an item using a camera built into the terminal.

[0081] The "server" is a device that receives image data sent from the terminal, analyzes it using a generative AI model, and makes a decision on whether to declutter based on detailed information about the item and its usage history.

[0082] A "generative AI model" is a model that uses machine learning and deep learning algorithms to analyze images and extract detailed information about items.

[0083] "Detailed information" refers to attribute information about an item, such as the title, author, and publication year in the case of a book.

[0084] "Usage history" is a record of how a user has used an item, and includes data such as frequency of use and the date and time of last use.

[0085] The "means for determining whether to declutter" is a function for determining whether an item should be decluttered based on detailed information and usage history.

[0086] The "optimal disposal method" refers to the method of selecting the most appropriate way to dispose of an item, including, for example, donating it, selling it at a thrift store, or taking it to a recycling center.

[0087] The "means for displaying suggestions" is a function that visualizes the information on tidying methods sent from the server on the terminal and displays it in a way that is easy for the user to understand.

[0088] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. Specifically, it works by combining a terminal, a server, and a generative AI model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0089] First, the user launches the application on their device and takes a picture of the items they want to organize. The device has a built-in camera that can capture high-resolution images. When the user taps the "shutter button," the image data is saved on the device and then compressed. This compressed image data is then sent from the device to the server.

[0090] The server passes the received image data to a generative AI model to begin analysis. This generative AI model uses machine learning and deep learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, the generative AI model would recognize the title, author, publication year, etc. The server also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0091] The server determines whether an item should be discarded based on detailed information and usage history. Based on this determination, the server selects the optimal method for discarding the item. Methods include donating, selling at a recycle shop, or taking it to a recycling center. The selected information is organized along with specific procedures and reference images and sent to the device.

[0092] The device then displays the received tidying methods and reference images to the user. The user then uses this information to actually declutter and organize their belongings. For example, if a specific book is suggested as being sold at a recycle shop, the user can follow the suggestion and bring the book to the shop to efficiently declutter. The app may also display links to related sales sites and decluttering support services, which can be used to further smooth the decluttering process.

[0093] For example, when a user takes a photo of a book on their bookshelf, the app extracts detailed information such as the book's title, author, and publication year. The server then determines that the book is infrequently used and identifies it as a candidate for decluttering. It then suggests selling the book at a recycle shop, and specific instructions, the store's location, and a reference image are displayed on the device. The user can use this information to go and sell the book.

[0094] Example prompt sentence:

[0095] "Analyze the images taken by the user, extract detailed information about the items, compare it with usage history data, determine whether decluttering is necessary, and suggest the best way to tidy up."

[0096] In this way, the present invention provides a system that allows users to easily and efficiently declutter and support organizing.

[0097] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0098] Step 1:

[0099] The user launches the app and takes a picture of the item.

[0100] Input: User action (launching an app, launching the camera, tapping the shutter button)

[0101] Output: High-resolution image data

[0102] Specific operation: A user takes a picture of an object using the camera on their smartphone or tablet. When the user taps the "shutter button," the device captures the image data and stores it locally.

[0103] Step 2:

[0104] The device compresses the images it takes and sends them to the server.

[0105] Input: High-resolution image data

[0106] Output: Compressed image data (e.g., JPEG format)

[0107] Specific operation: The image data captured by the device is compressed using an appropriate compression algorithm (e.g., JPEG compression) to reduce the data size. The compressed image data is then sent to the server.

[0108] Step 3:

[0109] The server receives the image data and passes it to the generative AI model.

[0110] Input: Compressed image data

[0111] Output: Data input to a generative AI model

[0112] Specific operation: The server receives the compressed image data sent from the device and stores it in a temporary storage location. The image data is then passed to the generative AI model and prepared for analysis.

[0113] Step 4:

[0114] A generative AI model analyzes the image and extracts detailed information about the item.

[0115] Input: Compressed image data

[0116] Output: Detailed information about the item (e.g. title, author, publication year, etc.)

[0117] How it works: The generative AI model analyzes the image data it receives and applies image recognition algorithms to identify detailed information about the item. The resulting information is then stored in a database or passed on to the next processing step.

[0118] Step 5:

[0119] The server checks the usage history and evaluates the usage of the item.

[0120] Input: Item details

[0121] Output: Usage history data (e.g., frequency of use, last used date and time, etc.)

[0122] Specific operation: The server accesses the usage history database based on the detailed information of the item. It obtains the relevant usage history data and analyzes it to evaluate the usage of the item. The evaluation results are used in the next judgment process.

[0123] Step 6:

[0124] The server determines whether or not decluttering is possible based on detailed information and usage history.

[0125] Input: Detailed information on the item, usage history data

[0126] Output: Decision result of whether or not to declutter

[0127] How it works: The server combines the detailed information and usage history data and determines whether an item should be decluttered based on a pre-defined algorithm. For example, the algorithm is set up to identify items that are used infrequently as candidates for decluttering.

[0128] Step 7:

[0129] The server selects the optimal cleaning method and sends it to the device.

[0130] Input: Decision result of whether or not to declutter

[0131] Output: Data suggesting how to clean up (e.g., information on recycle shops, specific steps, reference images)

[0132] Specific operation: The server selects the optimal tidying method based on the results of the decluttering assessment. It generates information about the selected tidying method (e.g., information about the recycle shop, procedures for bringing in items, images, etc.) and sends that data to the device.

[0133] Step 8:

[0134] The device displays the cleaning method and reference images to the user.

[0135] Input: Tidying method suggestion data

[0136] Output: Visual display information to the user

[0137] Specific operation: The device analyzes the data on tidying methods received from the server and displays it through the user interface. Specific steps, reference images, related links, etc. are displayed to make it easy for the user to understand.

[0138] Step 9:

[0139] The user performs the decluttering process as suggested.

[0140] Input: Visual information on how to tidy up

[0141] Output: Items discarded

[0142] Specific actions: The user follows the tidying method displayed on the device and actually declutters. For example, they can take items to a recycle shop or dispose of them in a specified way, efficiently organizing their belongings.

[0143] In this way, a system is realized in which users, devices, servers, and generative AI models work together to efficiently declutter and organize.

[0144] (Application example 1)

[0145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] Conventional decluttering and organizing systems only extract detailed information about items and their usage history, but lack the functionality to provide market value or location information for nearby recycle shops, making it difficult for users to decide how to dispose of their items and to dispose of them efficiently. The purpose of this invention is to solve these problems and provide a system that allows users to dispose of items efficiently and rationally.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0148] In this invention, the server includes means for taking images on the terminal, means for transmitting the images to the server, means for receiving the images on the server and analyzing them using a generative model, means for extracting detailed information and usage history through the analysis, means for determining items that should be decluttered based on the extracted information, means for transmitting the determination results and a proposal including a tidying up method to the terminal, means for displaying the transmitted proposal and supporting the user in carrying out the proposal, means for evaluating the market value of the items, and means for providing location information of nearby recycle shops. This allows the user to know the details and market value of the items and select the optimal disposal method, as well as obtain information on nearby recycle shops, enabling efficient decluttering and organizing.

[0149] A "terminal" is an electronic device that can be carried by a user and has the function of taking pictures and communicating with a server.

[0150] The "means for taking an image" is a function for taking an image of an item using a camera or a photography application installed on the terminal.

[0151] The "means for transmitting to a server" is a function for transmitting captured image data to a server via the Internet or other communication means.

[0152] A "generative model" is an artificial intelligence technology that uses methods such as deep learning and machine learning to analyze data and apply the results.

[0153] The "means for analyzing" is a function that analyzes the received image using a generative model on the server and extracts the necessary information.

[0154] "Detailed information" refers to the identification information and characteristics of an item, specifically, information such as the title, author, and publication year for a book.

[0155] "Usage history" is a record of how a user has used a particular item, including frequency of use and the date and time of last use.

[0156] The "means for determining items that should be decluttered" is a function that evaluates the usefulness of an item based on the extracted detailed information and usage history, and determines whether it should be subject to decluttering.

[0157] "Cleaning methods" are specific ways to dispose of or reuse items, such as donating them, selling them at a recycling shop, or taking them to a recycling center.

[0158] The "means for sending suggestions to the terminal" is a function for sending the suggestions and tidying methods generated by the server to the user's terminal.

[0159] "Means for displaying suggestions and assisting the user in implementing them" refers to a function that visually displays the received suggestions on the device and supports the user in implementing them.

[0160] The "means for assessing market value" is a function for estimating the market value of an item based on the analyzed detailed information of the item.

[0161] "Means for providing location information of recycle shops" is a function that provides users with the location and contact information of the nearest recycle shop where they can drop off their items.

[0162] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0163] First, the user launches the application on their device. Next, they take a picture of the items they want to organize using the device's built-in camera. When the user taps the "shutter button," high-resolution image data is captured. The device then compresses the captured image data and sends it to the server.

[0164] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses deep learning and machine learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, publication year, etc. Based on this detailed information, the market value of the item is also estimated.

[0165] The server then accesses a usage history database to check how the user has used the item in the past. This includes frequency of use and the date and time of the last use. Based on this, the server determines whether the item is suitable for decluttering. If it is determined that the item should be decluttered, the server suggests the best way to declutter (for example, donate it, sell it at a recycle shop, or take it to a recycling center).

[0166] The server sends the generated detailed information, the market value of the items, how to organize them, and the location of nearby recycle shops to the device. The device visually displays this information to the user. The user can refer to the displayed information to actually declutter and organize their belongings. In addition, by taking the items to the recycle shop indicated, they can dispose of them effectively.

[0167] As a specific example of processing, the following flow can be considered.

[0168] 1. The user takes a photo of a book on a bookshelf.

[0169] 2. The image is sent to the server, and the generative model extracts detailed information such as "Title: 'Book Title', Author: 'Author Name', Year of Publication: 1997".

[0170] 3. The server checks the usage history and recognizes that the usage frequency is low.

[0171] 4. The server identifies the items as candidates for decluttering and suggests selling them at a recycle shop.

[0172] 5. The server sends the location and business information of the recycle shop to the terminal.

[0173] 6. The user takes the book to a thrift store based on the suggestions.

[0174] An example of a prompt for a generative AI model is as follows:

[0175] Extract detailed information about the following object. The object belongs to the category "Book". Specifically, output the title, author, and publication year.

[0176] As described above, the present invention is a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[0177] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0178] Step 1:

[0179] The user uses the device's camera to take a picture of the item they want to organize. The user's input is the image of the item, and the output is high-resolution image data. The user launches the camera application on the device, frames the item, and taps the shutter button. This action generates image data.

[0180] Step 2:

[0181] The device compresses the captured image and sends it to the server. The input is high-resolution image data, and the output is compressed image data. The image compression algorithm in the device operates to compress the image data while maintaining a certain level of quality. The image data is then sent to the server via the Internet.

[0182] Step 3:

[0183] The server passes the received image data to the generative model and begins analysis. The input is compressed image data, and the output is detailed information about the item. The server first decompresses the compressed data and analyzes the image of the item using a generative AI model (for example, a deep learning model using TENSORFLOW (registered trademark)). Specifically, it extracts detailed information such as the item's title, author, and publication year.

[0184] Step 4:

[0185] The server accesses the usage history database and checks the usage history based on the extracted detailed information. The input is the detailed information about the item, and the output is the usage history information. The server queries the database using the detailed information as a key to obtain information such as the frequency of use of the item and the date and time of its last use.

[0186] Step 5:

[0187] The server determines whether an item should be decluttered and evaluates its market value based on detailed information and usage history. The input is detailed information and usage history information, and the output is the decluttering decision result and market value assessment result. The server's algorithm evaluates the input data and, if the item is not used frequently, it is considered for decluttering and its market value is also evaluated.

[0188] Step 6:

[0189] The server selects the optimal tidying method and generates a proposal along with the location information of the recycle shop. The input is the results of the decluttering assessment and the market value assessment, and the output is the tidying method and proposal. The server takes into account the condition and market price of the items and selects the optimal tidying method (e.g., selling at a recycle shop). It also collects the location information of nearby recycle shops and includes it in the proposal.

[0190] Step 7:

[0191] The server sends the generated proposal to the terminal. The input is the proposal, and the output is data sent to the terminal. The server sends the proposal to the user's terminal and notifies them in real time.

[0192] Step 8:

[0193] The terminal displays the received proposals and information about the recycle shop to the user. The input is proposal data from the server, and the output is visual information displayed on the terminal screen. The proposals are displayed using the terminal's display function in a way that allows the user to intuitively understand them.

[0194] Step 9:

[0195] The user refers to the suggestions displayed on the device and takes action to actually dispose of the items. The input is the suggestion information displayed on the device, and the output is the actual disposal of the items. By bringing the items to a recycle shop, the user can efficiently declutter.

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] The present invention is a system that efficiently uses a terminal to declutter and organize items, and further adjusts tidying suggestions by recognizing the user's emotions. This system operates by combining a terminal, a server, a generative model, and an emotion engine, and suggests decluttering and tidying methods based on detailed information, usage history, and the user's emotional state.

[0198] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured, and then the device compresses the image and sends it to the server.

[0199] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0200] Furthermore, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine recognizes emotions based on the user's voice, facial expressions, and behavioral data to determine the user's current emotional state. For example, it can recognize whether the user is feeling stressed or relaxed.

[0201] The server determines whether an item should be decluttered based on the generated detailed information, usage history, and the user's emotional state. If it determines that decluttering is necessary, it selects the most appropriate tidying method for the user. In doing so, it refers to the emotion recognition results and adjusts the suggestions. For example, if the user is feeling stressed, it will suggest simple tidying methods or methods that are expected to have a positive effect. The suggestions include specific steps and reference images and are sent to the device.

[0202] The device displays the tidying method and reference images received from the server to the user. The user can refer to the displayed information to actually declutter and organize their belongings. For example, if a book is suggested as "sold at a recycle shop," the user can proceed with preparations to take it to the recycle shop. Advice based on emotion recognition results is also displayed, allowing the user to maintain positive emotions while tidying up.

[0203] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. Furthermore, the emotion engine will recognize that the user is relaxed and suggest selling the book at a recycle shop. The suggestion includes specific steps, the store's location, and a reference image, and is displayed on the device. The user can use this information to go and sell the book.

[0204] In this way, the present invention is a system that provides powerful support for decluttering and tidying up easily and efficiently while taking into consideration the user's feelings.

[0205] The processing flow will be explained below.

[0206] Step 1:

[0207] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[0208] Step 2:

[0209] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[0210] Step 3:

[0211] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[0212] Step 4:

[0213] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[0214] Step 5:

[0215] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[0216] Step 6:

[0217] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[0218] Step 7:

[0219] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes emotions based on the user's voice, facial expressions, and behavioral data to determine their current emotional state. It analyzes in real time whether the user is feeling stressed or relaxed.

[0220] Step 8:

[0221] The server determines whether an item should be discarded based on detailed information, usage history, and the user's emotional state. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[0222] Step 9:

[0223] If the server determines that an item should be discarded, it selects the optimal method for decluttering, including donating it or selling it at a recycle shop. It also refers to the emotion recognition results and suggests simple decluttering methods if the user is stressed, or more proactive suggestions if the user is relaxed.

[0224] Step 10:

[0225] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[0226] Step 11:

[0227] The device analyzes the tidying method content and reference images received from the server and displays them on the user interface, allowing the user to check specific tidying steps and related images. In addition, positive advice provided by the emotion engine is also displayed.

[0228] Step 12:

[0229] The user actually declutters and organizes their belongings by referring to the tidying method displayed on the device. For example, they put into practice the recommendation made by the emotion engine in a relaxed state to take their belongings to a recycling shop.

[0230] Step 13:

[0231] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[0232] The above are the specific processing steps of this system.

[0233] Example 2

[0234] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0235] Conventional decluttering and tidying systems provide uniform suggestions without considering the user's emotional state, which can be stressful for many users. Furthermore, they do not effectively utilize detailed information about items or usage history, making it difficult to provide optimal suggestions for users. Furthermore, if the tidying methods provided by the system are not appropriate for a particular emotional state, users may find it difficult to follow the suggestions.

[0236] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0237] In this invention, the server includes means for receiving images and analyzing them using a generative AI model, means for extracting detailed information and usage history of items through analysis, means for determining items to be decluttered based on the extracted information and the user's emotional state, means for sending suggestions including tidying methods according to the user's emotional state to the terminal, and means for displaying the sent suggestions and supporting the user in carrying them out. This enables efficient and stress-free decluttering and tidying while taking the user's emotions into consideration.

[0238] 1. "Terminal" means a type of electronic device operated by a user, which has hardware such as a camera, microphone, and display, and has the ability to run applications.

[0239] 2. "Server" means a computer system that receives data sent from a terminal via a network and performs processing such as analysis and database management.

[0240] 3. A "generative AI model" is a type of artificial intelligence that uses machine learning and deep learning algorithms to analyze data and extract detailed information.

[0241] 4. "Image analysis" is the process of taking image data as input and extracting information about an item or other object from it.

[0242] 5. "Detailed information" refers to specific characteristics and attributes of the photographed item, such as the title, author, and publication year.

[0243] 6. "Usage history" refers to data about how a user has used an item in the past, including frequency of use and the date and time of last use.

[0244] 7. An "emotion engine" is a software or hardware system that analyzes a user's emotional state based on their voice, facial expression, and behavioral data.

[0245] 8. "Danshari" is the process of organizing and disposing of items that users have determined to be unnecessary.

[0246] 9. "Tidying up method" refers to the specific means and procedures for organizing and tidying up items.

[0247] 10. "Suggestions" are specific advice or steps regarding decluttering and tidying up that are sent from the server to the device.

[0248] This invention is a system that uses a device to efficiently declutter and organize items, and further recognizes the user's emotions and adjusts tidying suggestions accordingly. This system operates by combining a device, a server, a generative AI model, and an emotion engine.

[0249] First, the user launches the dedicated Danshari application using their device. The initial screen of the application is displayed, providing an interface for starting to organize items. The user then takes a photo of the item they want to organize using the device's camera. Specifically, when the user taps the "photo button," the device's camera takes a high-resolution image. The device's camera module captures the image and stores it in the device's internal memory.

[0250] Next, the device compresses the captured image data in JPEG format or similar and sends it to the server. The device's network module sends the image data to the server using an HTTP POST request. The server receives the HTTP POST request and uses web server software (e.g., NGINX or Apache (registered trademark)) to save the image data in a storage directory.

[0251] The server then runs the generative AI model to analyze the received image. During this analysis, detailed information about the object contained in the image (e.g., title, author, publication year, etc.) is extracted. The server then runs a Python script to apply the generative AI model using a machine learning library such as TensorFlow.

[0252] The server then accesses a usage history database to see how the user has used the item in the past, and executes an SQL query to retrieve the relevant usage history from the database, including frequency of use and the last time it was used.

[0253] The device then runs an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotional state by collecting data on their voice, facial expressions, and behavior. Specifically, the device collects data using its microphone and camera, and passes it to an emotion engine such as Affectiva's SDK for analysis.

[0254] The server determines whether items should be decluttered based on detailed information, usage history, and emotion recognition results generated by the server. The server executes logic and evaluates the need for decluttering according to conditional branching. If it determines that decluttering is necessary, it suggests the optimal tidying method based on the user's emotional state. The server inputs prompts into the generative AI model and obtains the output results. This makes it possible to suggest simple tidying methods or methods that are expected to have a positive effect if the user is feeling stressed.

[0255] A specific example is given below.

[0256] Example: When a user takes a photo of a book on a bookshelf, the device's camera captures an image of the book. The device compresses the image and sends it to the server. The server uses a generative AI model to extract detailed information such as "Harry Potter and the Philosopher's Stone, published in 1997, by J.K. Rowling." The server determines that the book is infrequently used and makes it a candidate for decluttering. The emotion engine recognizes that the user is relaxed and suggests "selling at a recycle shop." The suggestion includes specific steps, the store's location, and a reference image. The device displays the suggestion to the user, who then begins preparing to take the book to the recycle shop.

[0257] Example prompt for a generative AI model:

[0258] "I took a picture of a book on my bookshelf. Please extract detailed information about this book and help me decide whether to get rid of it."

[0259] "Please suggest the best way for users to clean up when they are feeling stressed."

[0260] As described above, this system takes into consideration the user's emotions and enables efficient and stress-free decluttering and tidying.

[0261] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0262] Step 1:

[0263] The user operates the device and launches the dedicated Danshari application. The device displays the initial screen of the application.

[0264] Input: Launch application

[0265] Output: Display of initial screen

[0266] Specific operations: The device operating system loads the application and renders the user interface.

[0267] Step 2:

[0268] The user takes a photo of the item they want to organize using their device's camera. When the user taps the "photo button," the device's camera captures a high-resolution image.

[0269] Input: Photo of item

[0270] Output: Captured image data

[0271] Specific operation: The camera module captures an image and saves it in the internal memory in JPEG format.

[0272] Step 3:

[0273] The image data captured by the terminal is compressed and sent to the server.

[0274] Input: Photographed image data

[0275] Output: Compressed image data and HTTP POST request

[0276] Specific operation: The terminal software compresses the image in JPEG format and uses the network module to send an HTTP POST request.

[0277] Step 4:

[0278] The server receives the HTTP POST request and saves the image data.

[0279] Input: Compressed image data

[0280] Output: Image data stored on the server

[0281] What happens: Web server software (e.g., NGINX or Apache) processes the request and stores the image in a database or on the file system.

[0282] Step 5:

[0283] The server runs the generative AI model to analyze the received image and extract detailed information (title, author, publication year, etc.).

[0284] Input: Image data stored on the server

[0285] Output: Extracted details

[0286] What it does: A Python script applies a generative AI model using a machine learning library such as TensorFlow to perform image analysis.

[0287] Step 6:

[0288] The server accesses the usage history database and extracts the usage history of the item.

[0289] Input: Extracted details

[0290] Output: Item usage history

[0291] Specific operation: Executes an SQL query and retrieves the relevant usage history from the database.

[0292] Step 7:

[0293] The device runs an emotion engine to analyze the user's emotional state, collecting the user's voice, facial expression, and behavioral data.

[0294] Input: User's voice, facial expression, and behavior data

[0295] Output: User's emotional state

[0296] Specific operation: Collects data using the device's microphone and camera and passes it to an emotion engine such as Affectiva's SDK.

[0297] Step 8:

[0298] The server determines whether an item should be decluttered based on detailed information, usage history, and emotion recognition results.

[0299] Input: detailed information, usage history, emotion recognition results

[0300] Output: Decluttering results

[0301] What it does: The server logic synthesizes the information and evaluates the need for decluttering based on conditional branching.

[0302] Step 9:

[0303] The server uses a generative AI model to suggest the optimal tidying method.

[0304] Input: Decision result of decluttering, user's emotional state

[0305] Output: Tidying up suggestions

[0306] Specific operation: Input a prompt into the generative AI model, obtain the output, and generate a cleaning method.

[0307] Step 10:

[0308] The server sends the generated proposal to the terminal.

[0309] Input: Tidying up suggestions, reference images

[0310] Output: Proposal as HTTP response

[0311] Specific operation: The proposal content is sent to the terminal as an HTTP response.

[0312] Step 11:

[0313] The terminal displays the received proposal to the user.

[0314] Input: Proposal content, reference image

[0315] Output: Display of proposal

[0316] Specific behavior: The suggestions are displayed as text and images on the device display, and the user interface is updated.

[0317] Through the above steps, this system is able to support efficient decluttering and tidying while taking into consideration the user's emotions.

[0318] (Application example 2)

[0319] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0320] In modern society, many people find it difficult to organize and declutter. Deciding which items to discard or how to proceed with decluttering can be particularly difficult and stressful, as emotions are often involved. Conventional systems lack the technology to provide optimal decluttering recommendations while taking into account the user's emotional state, preventing efficient and positive decluttering and tidying.

[0321] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0322] In this invention, the server includes means for taking an image on the terminal, means for transmitting the image to the server, means for receiving the image on the server and analyzing it using a generative model, means for extracting detailed information and usage history through the analysis, means including an emotion recognition engine for analyzing the user's emotional state in real time, means for adjusting the content of decluttering suggestions based on the emotional state, means for transmitting suggestions including the assessment results and tidying methods to the terminal, and means for displaying the transmitted suggestions and supporting the user in implementing them. This enables efficient and positive decluttering and tidying while taking the user's emotional state into consideration.

[0323] A "terminal" is a computing device that allows a user to take images and communicate with a server.

[0324] "Image" refers to visual information captured by a user using a device, and is the data to be analyzed.

[0325] A "server" is a central computing system for receiving image data sent from a terminal and for analyzing and processing the information.

[0326] A "generative model" is an algorithmic system that uses machine learning and deep learning to analyze image data and extract detailed information.

[0327] "Detailed information" refers to the identification data of an item obtained through image analysis (e.g., book title, author, publication year, etc.).

[0328] "Usage history" refers to a record of how a user has used the item in the past (e.g., frequency of use, date of last use, etc.).

[0329] "Danshari" is the process of organizing your home by throwing away unnecessary items and keeping only what is necessary.

[0330] An "emotion recognition engine" is a system that analyzes a user's voice, facial expression, and behavioral data in real time to determine the user's emotional state.

[0331] "Suggestions" are advice on how to tidy up or declutter that the server provides to the user based on the generative model and emotion recognition results.

[0332] "Sending" is the communication process by which the server delivers image data, analysis results, and proposals to the terminal.

[0333] "Display" refers to the act of visually presenting the suggestions and information received by the terminal to the user.

[0334] "Assistance" is the guidance or help provided to help users implement the suggested tidying methods.

[0335] The system based on this invention provides support for users to declutter and tidy up efficiently and positively. Specifically, the system allows users to use a device to take pictures of items, send the image data to a server, and the server generates decluttering suggestions based on detailed information, usage history, and the user's emotional state.

[0336] The system program is configured as follows:

[0337] 1. Image capture and transmission

[0338] The device is equipped with a camera function that allows the user to take pictures of the items they want to organize. The user activates the camera, takes a picture of the items they want to organize (e.g., books, clothes, furniture, etc.), and taps the shutter button. The image data is compressed and sent to the server, which then receives high-resolution visual data.

[0339] 2. Analysis of image data

[0340] The server then passes the received image data to a generative model based on machine learning or deep learning to extract detailed information. This model uses a known database to determine the item's category and characteristics (e.g., book title, author, publication year, etc.). The server then accesses a database of the user's usage history to determine how the item has been used in the past. This usage history includes frequency of use and the date and time of last use.

[0341] 3. Recognizing emotional states

[0342] The device is equipped with an emotion engine for emotion recognition, which analyzes the user's voice, facial expressions, and behavioral data in real time. This emotion engine determines the user's current emotional state (e.g., relaxed, stressed, sad, happy). For example, the emotion recognition engine can suggest "sale at a recycle shop" for an image taken while the user is relaxing in front of a bookshelf.

[0343] 4. Proposal Generation and Notification

[0344] The server determines whether an item should be decluttered based on the analyzed details, usage history, and emotional state. Once the determination is made, it selects the optimal decluttering method (e.g., recycle, discard, or preserve) and generates appropriate suggestions for the user. These suggestions take into account the user's emotional state and include advice to reduce stress. For example, specific steps such as "It would be a good idea to sell this book at a thrift store" are presented, along with the location of the thrift store and reference images.

[0345] Specific examples

[0346] For example, if a user takes a photo of a book on a bookshelf, the device extracts detailed information about the book (such as the book title, author name, and publication year). The server determines that the book is not frequently used, and uses an emotion engine to recognize that the user is in a relaxed state. As a result, the server suggests "selling at a recycle shop" and provides directions to the recycle shop and specific steps to follow.

[0347] Prompt Sentence Examples

[0348] "Please advise how this item should be disposed of based on the user's state of relaxation."

[0349] "What is the best way to declutter if the user is stressed?"

[0350] In this way, the present invention makes decluttering suggestions while taking into account the user's emotional state, enabling more efficient and less stressful tidying up.

[0351] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0352] Step 1:

[0353] The user takes a picture of an item on the device. The device's camera function is activated, and when the user presses the capture button, high-resolution image data is acquired. The image data is input and passed to the next step in a compressed form.

[0354] Step 2:

[0355] The device sends the captured image data to the server. The compressed image data is sent to the server using HTTP or HTTPS as the communication protocol. The sent image data becomes input and is passed to the analysis module on the server side.

[0356] Step 3:

[0357] The server passes the received image data to a generative model for detailed analysis. The generative model uses algorithms based on machine learning or deep learning to extract attributes of the item (e.g., book title, author, publication year, etc.). At this stage, the image data is input and detailed information about the item is output.

[0358] Step 4:

[0359] Based on the detailed information analyzed by the server, the server accesses the user's usage history database to obtain the item's usage status (e.g., frequency of use, date and time of last use, etc.). The extracted detailed information of the item is input, and usage history information is output.

[0360] Step 5:

[0361] The device uses an emotion recognition engine to analyze the user's current emotional state in real time, taking the user's voice, facial expression, and behavioral data as input. The emotion recognition engine analyzes and outputs the user's emotional state (e.g., relaxed, stressed, sad, happy).

[0362] Step 6:

[0363] Based on the detailed information, usage history, and the user's emotional state, the server determines whether an item should be decluttered. It then generates the optimal tidying method based on the user's emotional state. To achieve this, a generative AI model is used, which inputs the detailed information, usage history, and emotional state, and outputs decluttering suggestions and tidying procedures.

[0364] Step 7:

[0365] The server generates decluttering suggestions and tidying procedures, which are then sent to the device. The suggestions are adjusted based on the user's emotional state. The suggestions and tidying procedures are input, converted into a data format that can be sent to the device, and then output.

[0366] Step 8:

[0367] The device receives the decluttering suggestions and tidying procedures from the server and displays them to the user. This allows the user to actually declutter and tidy up based on the information displayed on the screen. The suggestions sent to the device are input, and the information displayed to the user is output.

[0368] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0369] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0370] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0371] [Second embodiment]

[0372] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0373] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0374] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0375] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0376] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0377] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0378] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0379] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0380] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0381] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0382] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0383] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0384] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0385] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured. The device then compresses the image and sends it to the server.

[0386] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0387] The server determines whether the item should be decluttered based on the generated detailed information and usage history. If it is determined that the item should be decluttered, it selects the most appropriate method for the user to declutter. Methods include donating the item, selling it at a recycle shop, or taking it to a recycling center. This information is organized along with specific instructions and reference images and sent to the device.

[0388] The device will then display the received tidying method and reference images to the user. The user can refer to the displayed information to actually declutter and organize. For example, if a specific book is suggested as being sold at a recycle shop, the user can efficiently tidy up by following the suggestion and bringing the book. The app may also display links to related sales sites and tidying support services, which can be used to further smooth the process of decluttering.

[0389] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. It will then suggest selling the book at a recycle shop, and specific instructions, the store's location, and a sample image will be displayed on the device. The user can then use this information to go and sell the book.

[0390] In this way, the present invention provides a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[0391] The processing flow will be explained below.

[0392] Step 1:

[0393] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[0394] Step 2:

[0395] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[0396] Step 3:

[0397] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[0398] Step 4:

[0399] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[0400] Step 5:

[0401] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[0402] Step 6:

[0403] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[0404] Step 7:

[0405] The server determines whether an item should be discarded based on the extracted details and usage history. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[0406] Step 8:

[0407] If the server is determined to be an item that should be discarded, the system selects the most appropriate method for disposal, including donation or sale at a recycle shop, and provides detailed information and reference images.

[0408] Step 9:

[0409] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[0410] Step 10:

[0411] The device analyzes the cleaning method content and reference images received from the server and displays them on the user interface, allowing the user to check the specific cleaning steps and related images.

[0412] Step 11:

[0413] Users can refer to the tidying method displayed on the device and actually declutter and organize their belongings. For example, they can prepare items for donation according to the specified method.

[0414] Step 12:

[0415] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[0416] The above are the specific processing steps of this system.

[0417] Example 1

[0418] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0419] In modern society, the increase in the number of items has made decluttering and organizing important issues. However, many users often have difficulty deciding which items to dispose of and how. Furthermore, manually deciding how to organize items is time-consuming, laborious, and inefficient. Therefore, there is a need for a system that supports efficient decluttering and organizing.

[0420] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0421] In this invention, the server includes means for extracting detailed information about an item, means for determining whether the item should be decluttered based on the usage history, and means for selecting an optimal tidying method and transmitting it to the terminal, thereby enabling the user to declutter and organize efficiently.

[0422] A "terminal" is a device that a user directly operates to take an image of an item.

[0423] The "means for taking images" is a function for taking high-resolution images of an item using a camera built into the terminal.

[0424] The "server" is a device that receives image data sent from the terminal, analyzes it using a generative AI model, and makes a decision on whether to declutter based on detailed information about the item and its usage history.

[0425] A "generative AI model" is a model that uses machine learning and deep learning algorithms to analyze images and extract detailed information about items.

[0426] "Detailed information" refers to attribute information about an item, such as the title, author, and publication year in the case of a book.

[0427] "Usage history" is a record of how a user has used an item, and includes data such as frequency of use and the date and time of last use.

[0428] The "means for determining whether to declutter" is a function for determining whether an item should be decluttered based on detailed information and usage history.

[0429] The "optimal disposal method" refers to the method of selecting the most appropriate way to dispose of an item, including, for example, donating it, selling it at a thrift store, or taking it to a recycling center.

[0430] The "means for displaying suggestions" is a function that visualizes the information on tidying methods sent from the server on the terminal and displays it in a way that is easy for the user to understand.

[0431] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. Specifically, it works by combining a terminal, a server, and a generative AI model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0432] First, the user launches the application on their device and takes a picture of the items they want to organize. The device has a built-in camera that can capture high-resolution images. When the user taps the "shutter button," the image data is saved on the device and then compressed. This compressed image data is then sent from the device to the server.

[0433] The server passes the received image data to a generative AI model to begin analysis. This generative AI model uses machine learning and deep learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, the generative AI model would recognize the title, author, publication year, etc. The server also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0434] The server determines whether an item should be discarded based on detailed information and usage history. Based on this determination, the server selects the optimal method for discarding the item. Methods include donating, selling at a recycle shop, or taking it to a recycling center. The selected information is organized along with specific procedures and reference images and sent to the device.

[0435] The device then displays the received tidying methods and reference images to the user. The user then uses this information to actually declutter and organize their belongings. For example, if a specific book is suggested as being sold at a recycle shop, the user can follow the suggestion and bring the book to the shop to efficiently declutter. The app may also display links to related sales sites and decluttering support services, which can be used to further smooth the decluttering process.

[0436] For example, when a user takes a photo of a book on their bookshelf, the app extracts detailed information such as the book's title, author, and publication year. The server then determines that the book is infrequently used and identifies it as a candidate for decluttering. It then suggests selling the book at a recycle shop, and specific instructions, the store's location, and a reference image are displayed on the device. The user can use this information to go and sell the book.

[0437] Example prompt sentence:

[0438] "Analyze the images taken by the user, extract detailed information about the items, compare it with usage history data, determine whether decluttering is necessary, and suggest the best way to tidy up."

[0439] In this way, the present invention provides a system that allows users to easily and efficiently declutter and support organizing.

[0440] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0441] Step 1:

[0442] The user launches the app and takes a picture of the item.

[0443] Input: User action (launching an app, launching the camera, tapping the shutter button)

[0444] Output: High-resolution image data

[0445] Specific operation: A user takes a picture of an object using the camera on their smartphone or tablet. When the user taps the "shutter button," the device captures the image data and stores it locally.

[0446] Step 2:

[0447] The device compresses the images it takes and sends them to the server.

[0448] Input: High-resolution image data

[0449] Output: Compressed image data (e.g., JPEG format)

[0450] Specific operation: The image data captured by the device is compressed using an appropriate compression algorithm (e.g., JPEG compression) to reduce the data size. The compressed image data is then sent to the server.

[0451] Step 3:

[0452] The server receives the image data and passes it to the generative AI model.

[0453] Input: Compressed image data

[0454] Output: Data input to a generative AI model

[0455] Specific operation: The server receives the compressed image data sent from the device and stores it in a temporary storage location. The image data is then passed to the generative AI model and prepared for analysis.

[0456] Step 4:

[0457] A generative AI model analyzes the image and extracts detailed information about the item.

[0458] Input: Compressed image data

[0459] Output: Detailed information about the item (e.g. title, author, publication year, etc.)

[0460] How it works: The generative AI model analyzes the image data it receives and applies image recognition algorithms to identify detailed information about the item. The resulting information is then stored in a database or passed on to the next processing step.

[0461] Step 5:

[0462] The server checks the usage history and evaluates the usage of the item.

[0463] Input: Item details

[0464] Output: Usage history data (e.g., frequency of use, last used date and time, etc.)

[0465] Specific operation: The server accesses the usage history database based on the detailed information of the item. It obtains the relevant usage history data and analyzes it to evaluate the usage of the item. The evaluation results are used in the next judgment process.

[0466] Step 6:

[0467] The server determines whether or not decluttering is possible based on detailed information and usage history.

[0468] Input: Detailed information on the item, usage history data

[0469] Output: Decision result of whether or not to declutter

[0470] How it works: The server combines the detailed information and usage history data and determines whether an item should be decluttered based on a pre-defined algorithm. For example, the algorithm is set up to identify items that are used infrequently as candidates for decluttering.

[0471] Step 7:

[0472] The server selects the optimal cleaning method and sends it to the device.

[0473] Input: Decision result of whether or not to declutter

[0474] Output: Data suggesting how to clean up (e.g., information on recycle shops, specific steps, reference images)

[0475] Specific operation: The server selects the optimal tidying method based on the results of the decluttering assessment. It generates information about the selected tidying method (e.g., information about the recycle shop, procedures for bringing in items, images, etc.) and sends that data to the device.

[0476] Step 8:

[0477] The device displays the cleaning method and reference images to the user.

[0478] Input: Tidying method suggestion data

[0479] Output: Visual display information to the user

[0480] Specific operation: The device analyzes the data on tidying methods received from the server and displays it through the user interface. Specific steps, reference images, related links, etc. are displayed to make it easy for the user to understand.

[0481] Step 9:

[0482] The user performs the decluttering process as suggested.

[0483] Input: Visual information on how to tidy up

[0484] Output: Items discarded

[0485] Specific actions: The user follows the tidying method displayed on the device and actually declutters. For example, they can take items to a recycle shop or dispose of them in a specified way, efficiently organizing their belongings.

[0486] In this way, a system is realized in which users, devices, servers, and generative AI models work together to efficiently declutter and organize.

[0487] (Application example 1)

[0488] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0489] Conventional decluttering and organizing systems only extract detailed information about items and their usage history, but lack the functionality to provide market value or location information for nearby recycle shops, making it difficult for users to decide how to dispose of their items and to dispose of them efficiently. The purpose of this invention is to solve these problems and provide a system that allows users to dispose of items efficiently and rationally.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0491] In this invention, the server includes means for taking images on the terminal, means for transmitting the images to the server, means for receiving the images on the server and analyzing them using a generative model, means for extracting detailed information and usage history through the analysis, means for determining items that should be decluttered based on the extracted information, means for transmitting the determination results and a proposal including a tidying up method to the terminal, means for displaying the transmitted proposal and supporting the user in carrying out the proposal, means for evaluating the market value of the items, and means for providing location information of nearby recycle shops. This allows the user to know the details and market value of the items and select the optimal disposal method, as well as obtain information on nearby recycle shops, enabling efficient decluttering and organizing.

[0492] A "terminal" is an electronic device that can be carried by a user and has the function of taking pictures and communicating with a server.

[0493] The "means for taking an image" is a function for taking an image of an item using a camera or a photography application installed on the terminal.

[0494] The "means for transmitting to a server" is a function for transmitting captured image data to a server via the Internet or other communication means.

[0495] A "generative model" is an artificial intelligence technology that uses methods such as deep learning and machine learning to analyze data and apply the results.

[0496] The "means for analyzing" is a function that analyzes the received image using a generative model on the server and extracts the necessary information.

[0497] "Detailed information" refers to the identification information and characteristics of an item, specifically, information such as the title, author, and publication year for a book.

[0498] "Usage history" is a record of how a user has used a particular item, including frequency of use and the date and time of last use.

[0499] The "means for determining items that should be decluttered" is a function that evaluates the usefulness of an item based on the extracted detailed information and usage history, and determines whether it should be subject to decluttering.

[0500] "Cleaning methods" are specific ways to dispose of or reuse items, such as donating them, selling them at a recycling shop, or taking them to a recycling center.

[0501] The "means for sending suggestions to the terminal" is a function for sending the suggestions and tidying methods generated by the server to the user's terminal.

[0502] "Means for displaying suggestions and assisting the user in implementing them" refers to a function that visually displays the received suggestions on the device and supports the user in implementing them.

[0503] The "means for assessing market value" is a function for estimating the market value of an item based on the analyzed detailed information of the item.

[0504] "Means for providing location information of recycle shops" is a function that provides users with the location and contact information of the nearest recycle shop where they can drop off their items.

[0505] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0506] First, the user launches the application on their device. Next, they take a picture of the items they want to organize using the device's built-in camera. When the user taps the "shutter button," high-resolution image data is captured. The device then compresses the captured image data and sends it to the server.

[0507] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses deep learning and machine learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, publication year, etc. Based on this detailed information, the market value of the item is also estimated.

[0508] The server then accesses a usage history database to check how the user has used the item in the past. This includes frequency of use and the date and time of the last use. Based on this, the server determines whether the item is suitable for decluttering. If it is determined that the item should be decluttered, the server suggests the best way to declutter (for example, donate it, sell it at a recycle shop, or take it to a recycling center).

[0509] The server sends the generated detailed information, the market value of the items, how to organize them, and the location of nearby recycle shops to the device. The device visually displays this information to the user. The user can refer to the displayed information to actually declutter and organize their belongings. In addition, by taking the items to the recycle shop indicated, they can dispose of them effectively.

[0510] As a specific example of processing, the following flow can be considered.

[0511] 1. The user takes a photo of a book on a bookshelf.

[0512] 2. The image is sent to the server, and the generative model extracts detailed information such as "Title: 'Book Title', Author: 'Author Name', Year of Publication: 1997".

[0513] 3. The server checks the usage history and recognizes that the usage frequency is low.

[0514] 4. The server identifies the items as candidates for decluttering and suggests selling them at a recycle shop.

[0515] 5. The server sends the location and business information of the recycle shop to the terminal.

[0516] 6. The user takes the book to a thrift store based on the suggestions.

[0517] An example of a prompt for a generative AI model is as follows:

[0518] Extract detailed information about the following object. The object belongs to the category "Book". Specifically, output the title, author, and publication year.

[0519] As described above, the present invention is a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[0520] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0521] Step 1:

[0522] The user uses the device's camera to take a picture of the item they want to organize. The user's input is the image of the item, and the output is high-resolution image data. The user launches the camera application on the device, frames the item, and taps the shutter button. This action generates image data.

[0523] Step 2:

[0524] The device compresses the captured image and sends it to the server. The input is high-resolution image data, and the output is compressed image data. The image compression algorithm in the device operates to compress the image data while maintaining a certain level of quality. The image data is then sent to the server via the Internet.

[0525] Step 3:

[0526] The server passes the received image data to a generative model and begins analysis. The input is compressed image data, and the output is detailed information about the item. The server first decompresses the compressed data and analyzes the image of the item using a generative AI model (for example, a deep learning model using TensorFlow). Specifically, it extracts detailed information such as the item's title, author, and publication year.

[0527] Step 4:

[0528] The server accesses the usage history database and checks the usage history based on the extracted detailed information. The input is the detailed information about the item, and the output is the usage history information. The server queries the database using the detailed information as a key to obtain information such as the frequency of use of the item and the date and time of its last use.

[0529] Step 5:

[0530] The server determines whether an item should be decluttered and evaluates its market value based on detailed information and usage history. The input is detailed information and usage history information, and the output is the decluttering decision result and market value assessment result. The server's algorithm evaluates the input data and, if the item is not used frequently, it is considered for decluttering and its market value is also evaluated.

[0531] Step 6:

[0532] The server selects the optimal tidying method and generates a proposal along with the location information of the recycle shop. The input is the results of the decluttering assessment and the market value assessment, and the output is the tidying method and proposal. The server takes into account the condition and market price of the items and selects the optimal tidying method (e.g., selling at a recycle shop). It also collects the location information of nearby recycle shops and includes it in the proposal.

[0533] Step 7:

[0534] The server sends the generated proposal to the terminal. The input is the proposal, and the output is data sent to the terminal. The server sends the proposal to the user's terminal and notifies them in real time.

[0535] Step 8:

[0536] The terminal displays the received proposals and information about the recycle shop to the user. The input is proposal data from the server, and the output is visual information displayed on the terminal screen. The proposals are displayed using the terminal's display function in a way that allows the user to intuitively understand them.

[0537] Step 9:

[0538] The user refers to the suggestions displayed on the device and takes action to actually dispose of the items. The input is the suggestion information displayed on the device, and the output is the actual disposal of the items. By bringing the items to a recycle shop, the user can efficiently declutter.

[0539] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0540] The present invention is a system that efficiently uses a terminal to declutter and organize items, and further adjusts tidying suggestions by recognizing the user's emotions. This system operates by combining a terminal, a server, a generative model, and an emotion engine, and suggests decluttering and tidying methods based on detailed information, usage history, and the user's emotional state.

[0541] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured, and then the device compresses the image and sends it to the server.

[0542] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0543] Furthermore, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine recognizes emotions based on the user's voice, facial expressions, and behavioral data to determine the user's current emotional state. For example, it can recognize whether the user is feeling stressed or relaxed.

[0544] The server determines whether an item should be decluttered based on the generated detailed information, usage history, and the user's emotional state. If it determines that decluttering is necessary, it selects the most appropriate tidying method for the user. In doing so, it refers to the emotion recognition results and adjusts the suggestions. For example, if the user is feeling stressed, it will suggest simple tidying methods or methods that are expected to have a positive effect. The suggestions include specific steps and reference images and are sent to the device.

[0545] The device displays the tidying method and reference images received from the server to the user. The user can refer to the displayed information to actually declutter and organize their belongings. For example, if a book is suggested as "sold at a recycle shop," the user can proceed with preparations to take it to the recycle shop. Advice based on emotion recognition results is also displayed, allowing the user to maintain positive emotions while tidying up.

[0546] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. Furthermore, the emotion engine will recognize that the user is relaxed and suggest selling the book at a recycle shop. The suggestion includes specific steps, the store's location, and a reference image, and is displayed on the device. The user can use this information to go and sell the book.

[0547] In this way, the present invention is a system that provides powerful support for decluttering and tidying up easily and efficiently while taking into consideration the user's feelings.

[0548] The processing flow will be explained below.

[0549] Step 1:

[0550] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[0551] Step 2:

[0552] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[0553] Step 3:

[0554] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[0555] Step 4:

[0556] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[0557] Step 5:

[0558] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[0559] Step 6:

[0560] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[0561] Step 7:

[0562] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes emotions based on the user's voice, facial expressions, and behavioral data to determine their current emotional state. It analyzes in real time whether the user is feeling stressed or relaxed.

[0563] Step 8:

[0564] The server determines whether an item should be discarded based on detailed information, usage history, and the user's emotional state. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[0565] Step 9:

[0566] If the server determines that an item should be discarded, it selects the optimal method for decluttering, including donating it or selling it at a recycle shop. It also refers to the emotion recognition results and suggests simple decluttering methods if the user is stressed, or more proactive suggestions if the user is relaxed.

[0567] Step 10:

[0568] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[0569] Step 11:

[0570] The device analyzes the tidying method content and reference images received from the server and displays them on the user interface, allowing the user to check specific tidying steps and related images. In addition, positive advice provided by the emotion engine is also displayed.

[0571] Step 12:

[0572] The user actually declutters and organizes their belongings by referring to the tidying method displayed on the device. For example, they put into practice the recommendation made by the emotion engine in a relaxed state to take their belongings to a recycling shop.

[0573] Step 13:

[0574] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[0575] The above are the specific processing steps of this system.

[0576] Example 2

[0577] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] Conventional decluttering and tidying systems provide uniform suggestions without considering the user's emotional state, which can be stressful for many users. Furthermore, they do not effectively utilize detailed information about items or usage history, making it difficult to provide optimal suggestions for users. Furthermore, if the tidying methods provided by the system are not appropriate for a particular emotional state, users may find it difficult to follow the suggestions.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0580] In this invention, the server includes means for receiving images and analyzing them using a generative AI model, means for extracting detailed information and usage history of items through analysis, means for determining items to be decluttered based on the extracted information and the user's emotional state, means for sending suggestions including tidying methods according to the user's emotional state to the terminal, and means for displaying the sent suggestions and supporting the user in carrying them out. This enables efficient and stress-free decluttering and tidying while taking the user's emotions into consideration.

[0581] 1. "Terminal" means a type of electronic device operated by a user, which has hardware such as a camera, microphone, and display, and has the ability to run applications.

[0582] 2. "Server" means a computer system that receives data sent from a terminal via a network and performs processing such as analysis and database management.

[0583] 3. A "generative AI model" is a type of artificial intelligence that uses machine learning and deep learning algorithms to analyze data and extract detailed information.

[0584] 4. "Image analysis" is the process of taking image data as input and extracting information about an item or other object from it.

[0585] 5. "Detailed information" refers to specific characteristics and attributes of the photographed item, such as the title, author, and publication year.

[0586] 6. "Usage history" refers to data about how a user has used an item in the past, including frequency of use and the date and time of last use.

[0587] 7. An "emotion engine" is a software or hardware system that analyzes a user's emotional state based on their voice, facial expression, and behavioral data.

[0588] 8. "Danshari" is the process of organizing and disposing of items that users have determined to be unnecessary.

[0589] 9. "Tidying up method" refers to the specific means and procedures for organizing and tidying up items.

[0590] 10. "Suggestions" are specific advice or steps regarding decluttering and tidying up that are sent from the server to the device.

[0591] This invention is a system that uses a device to efficiently declutter and organize items, and further recognizes the user's emotions and adjusts tidying suggestions accordingly. This system operates by combining a device, a server, a generative AI model, and an emotion engine.

[0592] First, the user launches the dedicated Danshari application using their device. The initial screen of the application is displayed, providing an interface for starting to organize items. The user then takes a photo of the item they want to organize using the device's camera. Specifically, when the user taps the "photo button," the device's camera takes a high-resolution image. The device's camera module captures the image and stores it in the device's internal memory.

[0593] Next, the device compresses the captured image data in JPEG format or similar and sends it to the server. The device's network module sends the image data to the server using an HTTP POST request. The server receives the HTTP POST request and uses web server software (e.g., NGINX or Apache) to save the image data in a storage directory.

[0594] The server then runs the generative AI model to analyze the received image. During this analysis, detailed information about the object contained in the image (e.g., title, author, publication year, etc.) is extracted. The server then runs a Python script to apply the generative AI model using a machine learning library such as TensorFlow.

[0595] The server then accesses a usage history database to see how the user has used the item in the past, and executes an SQL query to retrieve the relevant usage history from the database, including frequency of use and the last time it was used.

[0596] The device then runs an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotional state by collecting data on their voice, facial expressions, and behavior. Specifically, the device collects data using its microphone and camera, and passes it to an emotion engine such as Affectiva's SDK for analysis.

[0597] The server determines whether items should be decluttered based on detailed information, usage history, and emotion recognition results generated by the server. The server executes logic and evaluates the need for decluttering according to conditional branching. If it determines that decluttering is necessary, it suggests the optimal tidying method based on the user's emotional state. The server inputs prompts into the generative AI model and obtains the output results. This makes it possible to suggest simple tidying methods or methods that are expected to have a positive effect if the user is feeling stressed.

[0598] A specific example is given below.

[0599] Example: When a user takes a photo of a book on a bookshelf, the device's camera captures an image of the book. The device compresses the image and sends it to the server. The server uses a generative AI model to extract detailed information such as "Harry Potter and the Philosopher's Stone, published in 1997, by J.K. Rowling." The server determines that the book is infrequently used and makes it a candidate for decluttering. The emotion engine recognizes that the user is relaxed and suggests "selling at a recycle shop." The suggestion includes specific steps, the store's location, and a reference image. The device displays the suggestion to the user, who then begins preparing to take the book to the recycle shop.

[0600] Example prompt for a generative AI model:

[0601] "I took a picture of a book on my bookshelf. Please extract detailed information about this book and help me decide whether to get rid of it."

[0602] "Please suggest the best way for users to clean up when they are feeling stressed."

[0603] As described above, this system takes into consideration the user's emotions and enables efficient and stress-free decluttering and tidying.

[0604] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0605] Step 1:

[0606] The user operates the device and launches the dedicated Danshari application. The device displays the initial screen of the application.

[0607] Input: Launch application

[0608] Output: Display of initial screen

[0609] Specific operations: The device operating system loads the application and renders the user interface.

[0610] Step 2:

[0611] The user takes a photo of the item they want to organize using their device's camera. When the user taps the "photo button," the device's camera captures a high-resolution image.

[0612] Input: Photo of item

[0613] Output: Captured image data

[0614] Specific operation: The camera module captures an image and saves it in the internal memory in JPEG format.

[0615] Step 3:

[0616] The image data captured by the terminal is compressed and sent to the server.

[0617] Input: Photographed image data

[0618] Output: Compressed image data and HTTP POST request

[0619] Specific operation: The terminal software compresses the image in JPEG format and uses the network module to send an HTTP POST request.

[0620] Step 4:

[0621] The server receives the HTTP POST request and saves the image data.

[0622] Input: Compressed image data

[0623] Output: Image data stored on the server

[0624] What happens: Web server software (e.g., NGINX or Apache) processes the request and stores the image in a database or on the file system.

[0625] Step 5:

[0626] The server runs the generative AI model to analyze the received image and extract detailed information (title, author, publication year, etc.).

[0627] Input: Image data stored on the server

[0628] Output: Extracted details

[0629] What it does: A Python script applies a generative AI model using a machine learning library such as TensorFlow to perform image analysis.

[0630] Step 6:

[0631] The server accesses the usage history database and extracts the usage history of the item.

[0632] Input: Extracted details

[0633] Output: Item usage history

[0634] Specific operation: Executes an SQL query and retrieves the relevant usage history from the database.

[0635] Step 7:

[0636] The device runs an emotion engine to analyze the user's emotional state, collecting the user's voice, facial expression, and behavioral data.

[0637] Input: User's voice, facial expression, and behavior data

[0638] Output: User's emotional state

[0639] Specific operation: Collects data using the device's microphone and camera and passes it to an emotion engine such as Affectiva's SDK.

[0640] Step 8:

[0641] The server determines whether an item should be decluttered based on detailed information, usage history, and emotion recognition results.

[0642] Input: detailed information, usage history, emotion recognition results

[0643] Output: Decluttering results

[0644] What it does: The server logic synthesizes the information and evaluates the need for decluttering based on conditional branching.

[0645] Step 9:

[0646] The server uses a generative AI model to suggest the optimal tidying method.

[0647] Input: Decision result of decluttering, user's emotional state

[0648] Output: Tidying up suggestions

[0649] Specific operation: Input a prompt into the generative AI model, obtain the output, and generate a cleaning method.

[0650] Step 10:

[0651] The server sends the generated proposal to the terminal.

[0652] Input: Tidying up suggestions, reference images

[0653] Output: Proposal as HTTP response

[0654] Specific operation: The proposal content is sent to the terminal as an HTTP response.

[0655] Step 11:

[0656] The terminal displays the received proposal to the user.

[0657] Input: Proposal content, reference image

[0658] Output: Display of proposal

[0659] Specific behavior: The suggestions are displayed as text and images on the device display, and the user interface is updated.

[0660] Through the above steps, this system is able to support efficient decluttering and tidying while taking into consideration the user's emotions.

[0661] (Application example 2)

[0662] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0663] In modern society, many people find it difficult to organize and declutter. Deciding which items to discard or how to proceed with decluttering can be particularly difficult and stressful, as emotions are often involved. Conventional systems lack the technology to provide optimal decluttering recommendations while taking into account the user's emotional state, preventing efficient and positive decluttering and tidying.

[0664] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0665] In this invention, the server includes means for taking an image on the terminal, means for transmitting the image to the server, means for receiving the image on the server and analyzing it using a generative model, means for extracting detailed information and usage history through the analysis, means including an emotion recognition engine for analyzing the user's emotional state in real time, means for adjusting the content of decluttering suggestions based on the emotional state, means for transmitting suggestions including the assessment results and tidying methods to the terminal, and means for displaying the transmitted suggestions and supporting the user in implementing them. This enables efficient and positive decluttering and tidying while taking the user's emotional state into consideration.

[0666] A "terminal" is a computing device that allows a user to take images and communicate with a server.

[0667] "Image" refers to visual information captured by a user using a device, and is the data to be analyzed.

[0668] A "server" is a central computing system for receiving image data sent from a terminal and for analyzing and processing the information.

[0669] A "generative model" is an algorithmic system that uses machine learning and deep learning to analyze image data and extract detailed information.

[0670] "Detailed information" refers to the identification data of an item obtained through image analysis (e.g., book title, author, publication year, etc.).

[0671] "Usage history" refers to a record of how a user has used the item in the past (e.g., frequency of use, date of last use, etc.).

[0672] "Danshari" is the process of organizing your home by throwing away unnecessary items and keeping only what is necessary.

[0673] An "emotion recognition engine" is a system that analyzes a user's voice, facial expression, and behavioral data in real time to determine the user's emotional state.

[0674] "Suggestions" are advice on how to tidy up or declutter that the server provides to the user based on the generative model and emotion recognition results.

[0675] "Sending" is the communication process by which the server delivers image data, analysis results, and proposals to the terminal.

[0676] "Display" refers to the act of visually presenting the suggestions and information received by the terminal to the user.

[0677] "Assistance" is the guidance or help provided to help users implement the suggested tidying methods.

[0678] The system based on this invention provides support for users to declutter and tidy up efficiently and positively. Specifically, the system allows users to use a device to take pictures of items, send the image data to a server, and the server generates decluttering suggestions based on detailed information, usage history, and the user's emotional state.

[0679] The system program is configured as follows:

[0680] 1. Image capture and transmission

[0681] The device is equipped with a camera function that allows the user to take pictures of the items they want to organize. The user activates the camera, takes a picture of the items they want to organize (e.g., books, clothes, furniture, etc.), and taps the shutter button. The image data is compressed and sent to the server, which then receives high-resolution visual data.

[0682] 2. Analysis of image data

[0683] The server then passes the received image data to a generative model based on machine learning or deep learning to extract detailed information. This model uses a known database to determine the item's category and characteristics (e.g., book title, author, publication year, etc.). The server then accesses a database of the user's usage history to determine how the item has been used in the past. This usage history includes frequency of use and the date and time of last use.

[0684] 3. Recognizing emotional states

[0685] The device is equipped with an emotion engine for emotion recognition, which analyzes the user's voice, facial expressions, and behavioral data in real time. This emotion engine determines the user's current emotional state (e.g., relaxed, stressed, sad, happy). For example, the emotion recognition engine can suggest "sale at a recycle shop" for an image taken while the user is relaxing in front of a bookshelf.

[0686] 4. Proposal Generation and Notification

[0687] The server determines whether an item should be decluttered based on the analyzed details, usage history, and emotional state. Once the determination is made, it selects the optimal decluttering method (e.g., recycle, discard, or preserve) and generates appropriate suggestions for the user. These suggestions take into account the user's emotional state and include advice to reduce stress. For example, specific steps such as "It would be a good idea to sell this book at a thrift store" are presented, along with the location of the thrift store and reference images.

[0688] Specific examples

[0689] For example, if a user takes a photo of a book on a bookshelf, the device extracts detailed information about the book (such as the book title, author name, and publication year). The server determines that the book is not frequently used, and uses an emotion engine to recognize that the user is in a relaxed state. As a result, the server suggests "selling at a recycle shop" and provides directions to the recycle shop and specific steps to follow.

[0690] Prompt Sentence Examples

[0691] "Please advise how this item should be disposed of based on the user's state of relaxation."

[0692] "What is the best way to declutter if the user is stressed?"

[0693] In this way, the present invention makes decluttering suggestions while taking into account the user's emotional state, enabling more efficient and less stressful tidying up.

[0694] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0695] Step 1:

[0696] The user takes a picture of an item on the device. The device's camera function is activated, and when the user presses the capture button, high-resolution image data is acquired. The image data is input and passed to the next step in a compressed form.

[0697] Step 2:

[0698] The device sends the captured image data to the server. The compressed image data is sent to the server using HTTP or HTTPS as the communication protocol. The sent image data becomes input and is passed to the analysis module on the server side.

[0699] Step 3:

[0700] The server passes the received image data to a generative model for detailed analysis. The generative model uses algorithms based on machine learning or deep learning to extract attributes of the item (e.g., book title, author, publication year, etc.). At this stage, the image data is input and detailed information about the item is output.

[0701] Step 4:

[0702] Based on the detailed information analyzed by the server, the server accesses the user's usage history database to obtain the item's usage status (e.g., frequency of use, date and time of last use, etc.). The extracted detailed information of the item is input, and usage history information is output.

[0703] Step 5:

[0704] The device uses an emotion recognition engine to analyze the user's current emotional state in real time, taking the user's voice, facial expression, and behavioral data as input. The emotion recognition engine analyzes and outputs the user's emotional state (e.g., relaxed, stressed, sad, happy).

[0705] Step 6:

[0706] Based on the detailed information, usage history, and the user's emotional state, the server determines whether an item should be decluttered. It then generates the optimal tidying method based on the user's emotional state. To achieve this, a generative AI model is used, which inputs the detailed information, usage history, and emotional state, and outputs decluttering suggestions and tidying procedures.

[0707] Step 7:

[0708] The server generates decluttering suggestions and tidying procedures, which are then sent to the device. The suggestions are adjusted based on the user's emotional state. The suggestions and tidying procedures are input, converted into a data format that can be sent to the device, and then output.

[0709] Step 8:

[0710] The device receives the decluttering suggestions and tidying procedures from the server and displays them to the user. This allows the user to actually declutter and tidy up based on the information displayed on the screen. The suggestions sent to the device are input, and the information displayed to the user is output.

[0711] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0712] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0713] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0714] [Third embodiment]

[0715] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0716] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0717] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0718] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0719] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0720] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0721] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0722] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0723] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0724] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0725] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0726] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0727] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0728] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured. The device then compresses the image and sends it to the server.

[0729] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0730] The server determines whether the item should be decluttered based on the generated detailed information and usage history. If it is determined that the item should be decluttered, it selects the most appropriate method for the user to declutter. Methods include donating the item, selling it at a recycle shop, or taking it to a recycling center. This information is organized along with specific instructions and reference images and sent to the device.

[0731] The device will then display the received tidying method and reference images to the user. The user can refer to the displayed information to actually declutter and organize. For example, if a specific book is suggested as being sold at a recycle shop, the user can efficiently tidy up by following the suggestion and bringing the book. The app may also display links to related sales sites and tidying support services, which can be used to further smooth the process of decluttering.

[0732] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. It will then suggest selling the book at a recycle shop, and specific instructions, the store's location, and a sample image will be displayed on the device. The user can then use this information to go and sell the book.

[0733] In this way, the present invention provides a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[0734] The processing flow will be explained below.

[0735] Step 1:

[0736] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[0737] Step 2:

[0738] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[0739] Step 3:

[0740] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[0741] Step 4:

[0742] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[0743] Step 5:

[0744] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[0745] Step 6:

[0746] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[0747] Step 7:

[0748] The server determines whether an item should be discarded based on the extracted details and usage history. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[0749] Step 8:

[0750] If the server is determined to be an item that should be discarded, the system selects the most appropriate method for disposal, including donation or sale at a recycle shop, and provides detailed information and reference images.

[0751] Step 9:

[0752] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[0753] Step 10:

[0754] The device analyzes the cleaning method content and reference images received from the server and displays them on the user interface, allowing the user to check the specific cleaning steps and related images.

[0755] Step 11:

[0756] Users can refer to the tidying method displayed on the device and actually declutter and organize their belongings. For example, they can prepare items for donation according to the specified method.

[0757] Step 12:

[0758] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[0759] The above are the specific processing steps of this system.

[0760] Example 1

[0761] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0762] In modern society, the increase in the number of items has made decluttering and organizing important issues. However, many users often have difficulty deciding which items to dispose of and how. Furthermore, manually deciding how to organize items is time-consuming, laborious, and inefficient. Therefore, there is a need for a system that supports efficient decluttering and organizing.

[0763] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0764] In this invention, the server includes means for extracting detailed information about an item, means for determining whether the item should be decluttered based on the usage history, and means for selecting an optimal tidying method and transmitting it to the terminal, thereby enabling the user to declutter and organize efficiently.

[0765] A "terminal" is a device that a user directly operates to take an image of an item.

[0766] The "means for taking images" is a function for taking high-resolution images of an item using a camera built into the terminal.

[0767] The "server" is a device that receives image data sent from the terminal, analyzes it using a generative AI model, and makes a decision on whether to declutter based on detailed information about the item and its usage history.

[0768] A "generative AI model" is a model that uses machine learning and deep learning algorithms to analyze images and extract detailed information about items.

[0769] "Detailed information" refers to attribute information about an item, such as the title, author, and publication year in the case of a book.

[0770] "Usage history" is a record of how a user has used an item, and includes data such as frequency of use and the date and time of last use.

[0771] The "means for determining whether to declutter" is a function for determining whether an item should be decluttered based on detailed information and usage history.

[0772] The "optimal disposal method" refers to the method of selecting the most appropriate way to dispose of an item, including, for example, donating it, selling it at a thrift store, or taking it to a recycling center.

[0773] The "means for displaying suggestions" is a function that visualizes the information on tidying methods sent from the server on the terminal and displays it in a way that is easy for the user to understand.

[0774] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. Specifically, it works by combining a terminal, a server, and a generative AI model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0775] First, the user launches the application on their device and takes a picture of the items they want to organize. The device has a built-in camera that can capture high-resolution images. When the user taps the "shutter button," the image data is saved on the device and then compressed. This compressed image data is then sent from the device to the server.

[0776] The server passes the received image data to a generative AI model to begin analysis. This generative AI model uses machine learning and deep learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, the generative AI model would recognize the title, author, publication year, etc. The server also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0777] The server determines whether an item should be discarded based on detailed information and usage history. Based on this determination, the server selects the optimal method for discarding the item. Methods include donating, selling at a recycle shop, or taking it to a recycling center. The selected information is organized along with specific procedures and reference images and sent to the device.

[0778] The device then displays the received tidying methods and reference images to the user. The user then uses this information to actually declutter and organize their belongings. For example, if a specific book is suggested as being sold at a recycle shop, the user can follow the suggestion and bring the book to the shop to efficiently declutter. The app may also display links to related sales sites and decluttering support services, which can be used to further smooth the decluttering process.

[0779] For example, when a user takes a photo of a book on their bookshelf, the app extracts detailed information such as the book's title, author, and publication year. The server then determines that the book is infrequently used and identifies it as a candidate for decluttering. It then suggests selling the book at a recycle shop, and specific instructions, the store's location, and a reference image are displayed on the device. The user can use this information to go and sell the book.

[0780] Example prompt sentence:

[0781] "Analyze the images taken by the user, extract detailed information about the items, compare it with usage history data, determine whether decluttering is necessary, and suggest the best way to tidy up."

[0782] In this way, the present invention provides a system that allows users to easily and efficiently declutter and support organizing.

[0783] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0784] Step 1:

[0785] The user launches the app and takes a picture of the item.

[0786] Input: User action (launching an app, launching the camera, tapping the shutter button)

[0787] Output: High-resolution image data

[0788] Specific operation: A user takes a picture of an object using the camera on their smartphone or tablet. When the user taps the "shutter button," the device captures the image data and stores it locally.

[0789] Step 2:

[0790] The device compresses the images it takes and sends them to the server.

[0791] Input: High-resolution image data

[0792] Output: Compressed image data (e.g., JPEG format)

[0793] Specific operation: The image data captured by the device is compressed using an appropriate compression algorithm (e.g., JPEG compression) to reduce the data size. The compressed image data is then sent to the server.

[0794] Step 3:

[0795] The server receives the image data and passes it to the generative AI model.

[0796] Input: Compressed image data

[0797] Output: Data input to a generative AI model

[0798] Specific operation: The server receives the compressed image data sent from the device and stores it in a temporary storage location. The image data is then passed to the generative AI model and prepared for analysis.

[0799] Step 4:

[0800] A generative AI model analyzes the image and extracts detailed information about the item.

[0801] Input: Compressed image data

[0802] Output: Detailed information about the item (e.g. title, author, publication year, etc.)

[0803] How it works: The generative AI model analyzes the image data it receives and applies image recognition algorithms to identify detailed information about the item. The resulting information is then stored in a database or passed on to the next processing step.

[0804] Step 5:

[0805] The server checks the usage history and evaluates the usage of the item.

[0806] Input: Item details

[0807] Output: Usage history data (e.g., frequency of use, last used date and time, etc.)

[0808] Specific operation: The server accesses the usage history database based on the detailed information of the item. It obtains the relevant usage history data and analyzes it to evaluate the usage of the item. The evaluation results are used in the next judgment process.

[0809] Step 6:

[0810] The server determines whether or not decluttering is possible based on detailed information and usage history.

[0811] Input: Detailed information on the item, usage history data

[0812] Output: Decision result of whether or not to declutter

[0813] How it works: The server combines the detailed information and usage history data and determines whether an item should be decluttered based on a pre-defined algorithm. For example, the algorithm is set up to identify items that are used infrequently as candidates for decluttering.

[0814] Step 7:

[0815] The server selects the optimal cleaning method and sends it to the device.

[0816] Input: Decision result of whether or not to declutter

[0817] Output: Data suggesting how to clean up (e.g., information on recycle shops, specific steps, reference images)

[0818] Specific operation: The server selects the optimal tidying method based on the results of the decluttering assessment. It generates information about the selected tidying method (e.g., information about the recycle shop, procedures for bringing in items, images, etc.) and sends that data to the device.

[0819] Step 8:

[0820] The device displays the cleaning method and reference images to the user.

[0821] Input: Tidying method suggestion data

[0822] Output: Visual display information to the user

[0823] Specific operation: The device analyzes the data on tidying methods received from the server and displays it through the user interface. Specific steps, reference images, related links, etc. are displayed to make it easy for the user to understand.

[0824] Step 9:

[0825] The user performs the decluttering process as suggested.

[0826] Input: Visual information on how to tidy up

[0827] Output: Items discarded

[0828] Specific actions: The user follows the tidying method displayed on the device and actually declutters. For example, they can take items to a recycle shop or dispose of them in a specified way, efficiently organizing their belongings.

[0829] In this way, a system is realized in which users, devices, servers, and generative AI models work together to efficiently declutter and organize.

[0830] (Application example 1)

[0831] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0832] Conventional decluttering and organizing systems only extract detailed information about items and their usage history, but lack the functionality to provide market value or location information for nearby recycle shops, making it difficult for users to decide how to dispose of their items and to dispose of them efficiently. The purpose of this invention is to solve these problems and provide a system that allows users to dispose of items efficiently and rationally.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0834] In this invention, the server includes means for taking images on the terminal, means for transmitting the images to the server, means for receiving the images on the server and analyzing them using a generative model, means for extracting detailed information and usage history through the analysis, means for determining items that should be decluttered based on the extracted information, means for transmitting the determination results and a proposal including a tidying up method to the terminal, means for displaying the transmitted proposal and supporting the user in carrying out the proposal, means for evaluating the market value of the items, and means for providing location information of nearby recycle shops. This allows the user to know the details and market value of the items and select the optimal disposal method, as well as obtain information on nearby recycle shops, enabling efficient decluttering and organizing.

[0835] A "terminal" is an electronic device that can be carried by a user and has the function of taking pictures and communicating with a server.

[0836] The "means for taking an image" is a function for taking an image of an item using a camera or a photography application installed on the terminal.

[0837] The "means for transmitting to a server" is a function for transmitting captured image data to a server via the Internet or other communication means.

[0838] A "generative model" is an artificial intelligence technology that uses methods such as deep learning and machine learning to analyze data and apply the results.

[0839] The "means for analyzing" is a function that analyzes the received image using a generative model on the server and extracts the necessary information.

[0840] "Detailed information" refers to the identification information and characteristics of an item, specifically, information such as the title, author, and publication year for a book.

[0841] "Usage history" is a record of how a user has used a particular item, including frequency of use and the date and time of last use.

[0842] The "means for determining items that should be decluttered" is a function that evaluates the usefulness of an item based on the extracted detailed information and usage history, and determines whether it should be subject to decluttering.

[0843] "Cleaning methods" are specific ways to dispose of or reuse items, such as donating them, selling them at a recycling shop, or taking them to a recycling center.

[0844] The "means for sending suggestions to the terminal" is a function for sending the suggestions and tidying methods generated by the server to the user's terminal.

[0845] "Means for displaying suggestions and assisting the user in implementing them" refers to a function that visually displays the received suggestions on the device and supports the user in implementing them.

[0846] The "means for assessing market value" is a function for estimating the market value of an item based on the analyzed detailed information of the item.

[0847] "Means for providing location information of recycle shops" is a function that provides users with the location and contact information of the nearest recycle shop where they can drop off their items.

[0848] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[0849] First, the user launches the application on their device. Next, they take a picture of the items they want to organize using the device's built-in camera. When the user taps the "shutter button," high-resolution image data is captured. The device then compresses the captured image data and sends it to the server.

[0850] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses deep learning and machine learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, publication year, etc. Based on this detailed information, the market value of the item is also estimated.

[0851] The server then accesses a usage history database to check how the user has used the item in the past. This includes frequency of use and the date and time of the last use. Based on this, the server determines whether the item is suitable for decluttering. If it is determined that the item should be decluttered, the server suggests the best way to declutter (for example, donate it, sell it at a recycle shop, or take it to a recycling center).

[0852] The server sends the generated detailed information, the market value of the items, how to organize them, and the location of nearby recycle shops to the device. The device visually displays this information to the user. The user can refer to the displayed information to actually declutter and organize their belongings. In addition, by taking the items to the recycle shop indicated, they can dispose of them effectively.

[0853] As a specific example of processing, the following flow can be considered.

[0854] 1. The user takes a photo of a book on a bookshelf.

[0855] 2. The image is sent to the server, and the generative model extracts detailed information such as "Title: 'Book Title', Author: 'Author Name', Year of Publication: 1997".

[0856] 3. The server checks the usage history and recognizes that the usage frequency is low.

[0857] 4. The server identifies the items as candidates for decluttering and suggests selling them at a recycle shop.

[0858] 5. The server sends the location and business information of the recycle shop to the terminal.

[0859] 6. The user takes the book to a thrift store based on the suggestions.

[0860] An example of a prompt for a generative AI model is as follows:

[0861] Extract detailed information about the following object. The object belongs to the category "Book". Specifically, output the title, author, and publication year.

[0862] As described above, the present invention is a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[0863] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0864] Step 1:

[0865] The user uses the device's camera to take a picture of the item they want to organize. The user's input is the image of the item, and the output is high-resolution image data. The user launches the camera application on the device, frames the item, and taps the shutter button. This action generates image data.

[0866] Step 2:

[0867] The device compresses the captured image and sends it to the server. The input is high-resolution image data, and the output is compressed image data. The image compression algorithm in the device operates to compress the image data while maintaining a certain level of quality. The image data is then sent to the server via the Internet.

[0868] Step 3:

[0869] The server passes the received image data to a generative model and begins analysis. The input is compressed image data, and the output is detailed information about the item. The server first decompresses the compressed data and analyzes the image of the item using a generative AI model (for example, a deep learning model using TensorFlow). Specifically, it extracts detailed information such as the item's title, author, and publication year.

[0870] Step 4:

[0871] The server accesses the usage history database and checks the usage history based on the extracted detailed information. The input is the detailed information about the item, and the output is the usage history information. The server queries the database using the detailed information as a key to obtain information such as the frequency of use of the item and the date and time of its last use.

[0872] Step 5:

[0873] The server determines whether an item should be decluttered and evaluates its market value based on detailed information and usage history. The input is detailed information and usage history information, and the output is the decluttering decision result and market value assessment result. The server's algorithm evaluates the input data and, if the item is not used frequently, it is considered for decluttering and its market value is also evaluated.

[0874] Step 6:

[0875] The server selects the optimal tidying method and generates a proposal along with the location information of the recycle shop. The input is the results of the decluttering assessment and the market value assessment, and the output is the tidying method and proposal. The server takes into account the condition and market price of the items and selects the optimal tidying method (e.g., selling at a recycle shop). It also collects the location information of nearby recycle shops and includes it in the proposal.

[0876] Step 7:

[0877] The server sends the generated proposal to the terminal. The input is the proposal, and the output is data sent to the terminal. The server sends the proposal to the user's terminal and notifies them in real time.

[0878] Step 8:

[0879] The terminal displays the received proposals and information about the recycle shop to the user. The input is proposal data from the server, and the output is visual information displayed on the terminal screen. The proposals are displayed using the terminal's display function in a way that allows the user to intuitively understand them.

[0880] Step 9:

[0881] The user refers to the suggestions displayed on the device and takes action to actually dispose of the items. The input is the suggestion information displayed on the device, and the output is the actual disposal of the items. By bringing the items to a recycle shop, the user can efficiently declutter.

[0882] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0883] The present invention is a system that efficiently uses a terminal to declutter and organize items, and further adjusts tidying suggestions by recognizing the user's emotions. This system operates by combining a terminal, a server, a generative model, and an emotion engine, and suggests decluttering and tidying methods based on detailed information, usage history, and the user's emotional state.

[0884] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured, and then the device compresses the image and sends it to the server.

[0885] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[0886] Furthermore, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine recognizes emotions based on the user's voice, facial expressions, and behavioral data to determine the user's current emotional state. For example, it can recognize whether the user is feeling stressed or relaxed.

[0887] The server determines whether an item should be decluttered based on the generated detailed information, usage history, and the user's emotional state. If it determines that decluttering is necessary, it selects the most appropriate tidying method for the user. In doing so, it refers to the emotion recognition results and adjusts the suggestions. For example, if the user is feeling stressed, it will suggest simple tidying methods or methods that are expected to have a positive effect. The suggestions include specific steps and reference images and are sent to the device.

[0888] The device displays the tidying method and reference images received from the server to the user. The user can refer to the displayed information to actually declutter and organize their belongings. For example, if a book is suggested as "sold at a recycle shop," the user can proceed with preparations to take it to the recycle shop. Advice based on emotion recognition results is also displayed, allowing the user to maintain positive emotions while tidying up.

[0889] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. Furthermore, the emotion engine will recognize that the user is relaxed and suggest selling the book at a recycle shop. The suggestion includes specific steps, the store's location, and a reference image, and is displayed on the device. The user can use this information to go and sell the book.

[0890] In this way, the present invention is a system that provides powerful support for decluttering and tidying up easily and efficiently while taking into consideration the user's feelings.

[0891] The processing flow will be explained below.

[0892] Step 1:

[0893] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[0894] Step 2:

[0895] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[0896] Step 3:

[0897] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[0898] Step 4:

[0899] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[0900] Step 5:

[0901] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[0902] Step 6:

[0903] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[0904] Step 7:

[0905] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes emotions based on the user's voice, facial expressions, and behavioral data to determine their current emotional state. It analyzes in real time whether the user is feeling stressed or relaxed.

[0906] Step 8:

[0907] The server determines whether an item should be discarded based on detailed information, usage history, and the user's emotional state. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[0908] Step 9:

[0909] If the server determines that an item should be discarded, it selects the optimal method for decluttering, including donating it or selling it at a recycle shop. It also refers to the emotion recognition results and suggests simple decluttering methods if the user is stressed, or more proactive suggestions if the user is relaxed.

[0910] Step 10:

[0911] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[0912] Step 11:

[0913] The device analyzes the tidying method content and reference images received from the server and displays them on the user interface, allowing the user to check specific tidying steps and related images. In addition, positive advice provided by the emotion engine is also displayed.

[0914] Step 12:

[0915] The user actually declutters and organizes their belongings by referring to the tidying method displayed on the device. For example, they put into practice the recommendation made by the emotion engine in a relaxed state to take their belongings to a recycling shop.

[0916] Step 13:

[0917] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[0918] The above are the specific processing steps of this system.

[0919] Example 2

[0920] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0921] Conventional decluttering and tidying systems provide uniform suggestions without considering the user's emotional state, which can be stressful for many users. Furthermore, they do not effectively utilize detailed information about items or usage history, making it difficult to provide optimal suggestions for users. Furthermore, if the tidying methods provided by the system are not appropriate for a particular emotional state, users may find it difficult to follow the suggestions.

[0922] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0923] In this invention, the server includes means for receiving images and analyzing them using a generative AI model, means for extracting detailed information and usage history of items through analysis, means for determining items to be decluttered based on the extracted information and the user's emotional state, means for sending suggestions including tidying methods according to the user's emotional state to the terminal, and means for displaying the sent suggestions and supporting the user in carrying them out. This enables efficient and stress-free decluttering and tidying while taking the user's emotions into consideration.

[0924] 1. "Terminal" means a type of electronic device operated by a user, which has hardware such as a camera, microphone, and display, and has the ability to run applications.

[0925] 2. "Server" means a computer system that receives data sent from a terminal via a network and performs processing such as analysis and database management.

[0926] 3. A "generative AI model" is a type of artificial intelligence that uses machine learning and deep learning algorithms to analyze data and extract detailed information.

[0927] 4. "Image analysis" is the process of taking image data as input and extracting information about an item or other object from it.

[0928] 5. "Detailed information" refers to specific characteristics and attributes of the photographed item, such as the title, author, and publication year.

[0929] 6. "Usage history" refers to data about how a user has used an item in the past, including frequency of use and the date and time of last use.

[0930] 7. An "emotion engine" is a software or hardware system that analyzes a user's emotional state based on their voice, facial expression, and behavioral data.

[0931] 8. "Danshari" is the process of organizing and disposing of items that users have determined to be unnecessary.

[0932] 9. "Tidying up method" refers to the specific means and procedures for organizing and tidying up items.

[0933] 10. "Suggestions" are specific advice or steps regarding decluttering and tidying up that are sent from the server to the device.

[0934] This invention is a system that uses a device to efficiently declutter and organize items, and further recognizes the user's emotions and adjusts tidying suggestions accordingly. This system operates by combining a device, a server, a generative AI model, and an emotion engine.

[0935] First, the user launches the dedicated Danshari application using their device. The initial screen of the application is displayed, providing an interface for starting to organize items. The user then takes a photo of the item they want to organize using the device's camera. Specifically, when the user taps the "photo button," the device's camera takes a high-resolution image. The device's camera module captures the image and stores it in the device's internal memory.

[0936] Next, the device compresses the captured image data in JPEG format or similar and sends it to the server. The device's network module sends the image data to the server using an HTTP POST request. The server receives the HTTP POST request and uses web server software (e.g., NGINX or Apache) to save the image data in a storage directory.

[0937] The server then runs the generative AI model to analyze the received image. During this analysis, detailed information about the object contained in the image (e.g., title, author, publication year, etc.) is extracted. The server then runs a Python script to apply the generative AI model using a machine learning library such as TensorFlow.

[0938] The server then accesses a usage history database to see how the user has used the item in the past, and executes an SQL query to retrieve the relevant usage history from the database, including frequency of use and the last time it was used.

[0939] The device then runs an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotional state by collecting data on their voice, facial expressions, and behavior. Specifically, the device collects data using its microphone and camera, and passes it to an emotion engine such as Affectiva's SDK for analysis.

[0940] The server determines whether items should be decluttered based on detailed information, usage history, and emotion recognition results generated by the server. The server executes logic and evaluates the need for decluttering according to conditional branching. If it determines that decluttering is necessary, it suggests the optimal tidying method based on the user's emotional state. The server inputs prompts into the generative AI model and obtains the output results. This makes it possible to suggest simple tidying methods or methods that are expected to have a positive effect if the user is feeling stressed.

[0941] A specific example is given below.

[0942] Example: When a user takes a photo of a book on a bookshelf, the device's camera captures an image of the book. The device compresses the image and sends it to the server. The server uses a generative AI model to extract detailed information such as "Harry Potter and the Philosopher's Stone, published in 1997, by J.K. Rowling." The server determines that the book is infrequently used and makes it a candidate for decluttering. The emotion engine recognizes that the user is relaxed and suggests "selling at a recycle shop." The suggestion includes specific steps, the store's location, and a reference image. The device displays the suggestion to the user, who then begins preparing to take the book to the recycle shop.

[0943] Example prompt for a generative AI model:

[0944] "I took a picture of a book on my bookshelf. Please extract detailed information about this book and help me decide whether to get rid of it."

[0945] "Please suggest the best way for users to clean up when they are feeling stressed."

[0946] As described above, this system takes into consideration the user's emotions and enables efficient and stress-free decluttering and tidying.

[0947] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0948] Step 1:

[0949] The user operates the device and launches the dedicated Danshari application. The device displays the initial screen of the application.

[0950] Input: Launch application

[0951] Output: Display of initial screen

[0952] Specific operations: The device operating system loads the application and renders the user interface.

[0953] Step 2:

[0954] The user takes a photo of the item they want to organize using their device's camera. When the user taps the "photo button," the device's camera captures a high-resolution image.

[0955] Input: Photo of item

[0956] Output: Captured image data

[0957] Specific operation: The camera module captures an image and saves it in the internal memory in JPEG format.

[0958] Step 3:

[0959] The image data captured by the terminal is compressed and sent to the server.

[0960] Input: Photographed image data

[0961] Output: Compressed image data and HTTP POST request

[0962] Specific operation: The terminal software compresses the image in JPEG format and uses the network module to send an HTTP POST request.

[0963] Step 4:

[0964] The server receives the HTTP POST request and saves the image data.

[0965] Input: Compressed image data

[0966] Output: Image data stored on the server

[0967] What happens: Web server software (e.g., NGINX or Apache) processes the request and stores the image in a database or on the file system.

[0968] Step 5:

[0969] The server runs the generative AI model to analyze the received image and extract detailed information (title, author, publication year, etc.).

[0970] Input: Image data stored on the server

[0971] Output: Extracted details

[0972] What it does: A Python script applies a generative AI model using a machine learning library such as TensorFlow to perform image analysis.

[0973] Step 6:

[0974] The server accesses the usage history database and extracts the usage history of the item.

[0975] Input: Extracted details

[0976] Output: Item usage history

[0977] Specific operation: Executes an SQL query and retrieves the relevant usage history from the database.

[0978] Step 7:

[0979] The device runs an emotion engine to analyze the user's emotional state, collecting the user's voice, facial expression, and behavioral data.

[0980] Input: User's voice, facial expression, and behavior data

[0981] Output: User's emotional state

[0982] Specific operation: Collects data using the device's microphone and camera and passes it to an emotion engine such as Affectiva's SDK.

[0983] Step 8:

[0984] The server determines whether an item should be decluttered based on detailed information, usage history, and emotion recognition results.

[0985] Input: detailed information, usage history, emotion recognition results

[0986] Output: Decluttering results

[0987] What it does: The server logic synthesizes the information and evaluates the need for decluttering based on conditional branching.

[0988] Step 9:

[0989] The server uses a generative AI model to suggest the optimal tidying method.

[0990] Input: Decision result of decluttering, user's emotional state

[0991] Output: Tidying up suggestions

[0992] Specific operation: Input a prompt into the generative AI model, obtain the output, and generate a cleaning method.

[0993] Step 10:

[0994] The server sends the generated proposal to the terminal.

[0995] Input: Tidying up suggestions, reference images

[0996] Output: Proposal as HTTP response

[0997] Specific operation: The proposal content is sent to the terminal as an HTTP response.

[0998] Step 11:

[0999] The terminal displays the received proposal to the user.

[1000] Input: Proposal content, reference image

[1001] Output: Display of proposal

[1002] Specific behavior: The suggestions are displayed as text and images on the device display, and the user interface is updated.

[1003] Through the above steps, this system is able to support efficient decluttering and tidying while taking into consideration the user's emotions.

[1004] (Application example 2)

[1005] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1006] In modern society, many people find it difficult to organize and declutter. Deciding which items to discard or how to proceed with decluttering can be particularly difficult and stressful, as emotions are often involved. Conventional systems lack the technology to provide optimal decluttering recommendations while taking into account the user's emotional state, preventing efficient and positive decluttering and tidying.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1008] In this invention, the server includes means for taking an image on the terminal, means for transmitting the image to the server, means for receiving the image on the server and analyzing it using a generative model, means for extracting detailed information and usage history through the analysis, means including an emotion recognition engine for analyzing the user's emotional state in real time, means for adjusting the content of decluttering suggestions based on the emotional state, means for transmitting suggestions including the assessment results and tidying methods to the terminal, and means for displaying the transmitted suggestions and supporting the user in implementing them. This enables efficient and positive decluttering and tidying while taking the user's emotional state into consideration.

[1009] A "terminal" is a computing device that allows a user to take images and communicate with a server.

[1010] "Image" refers to visual information captured by a user using a device, and is the data to be analyzed.

[1011] A "server" is a central computing system for receiving image data sent from a terminal and for analyzing and processing the information.

[1012] A "generative model" is an algorithmic system that uses machine learning and deep learning to analyze image data and extract detailed information.

[1013] "Detailed information" refers to the identification data of an item obtained through image analysis (e.g., book title, author, publication year, etc.).

[1014] "Usage history" refers to a record of how a user has used the item in the past (e.g., frequency of use, date of last use, etc.).

[1015] "Danshari" is the process of organizing your home by throwing away unnecessary items and keeping only what is necessary.

[1016] An "emotion recognition engine" is a system that analyzes a user's voice, facial expression, and behavioral data in real time to determine the user's emotional state.

[1017] "Suggestions" are advice on how to tidy up or declutter that the server provides to the user based on the generative model and emotion recognition results.

[1018] "Sending" is the communication process by which the server delivers image data, analysis results, and proposals to the terminal.

[1019] "Display" refers to the act of visually presenting the suggestions and information received by the terminal to the user.

[1020] "Assistance" is the guidance or help provided to help users implement the suggested tidying methods.

[1021] The system based on this invention provides support for users to declutter and tidy up efficiently and positively. Specifically, the system allows users to use a device to take pictures of items, send the image data to a server, and the server generates decluttering suggestions based on detailed information, usage history, and the user's emotional state.

[1022] The system program is configured as follows:

[1023] 1. Image capture and transmission

[1024] The device is equipped with a camera function that allows the user to take pictures of the items they want to organize. The user activates the camera, takes a picture of the items they want to organize (e.g., books, clothes, furniture, etc.), and taps the shutter button. The image data is compressed and sent to the server, which then receives high-resolution visual data.

[1025] 2. Analysis of image data

[1026] The server then passes the received image data to a generative model based on machine learning or deep learning to extract detailed information. This model uses a known database to determine the item's category and characteristics (e.g., book title, author, publication year, etc.). The server then accesses a database of the user's usage history to determine how the item has been used in the past. This usage history includes frequency of use and the date and time of last use.

[1027] 3. Recognizing emotional states

[1028] The device is equipped with an emotion engine for emotion recognition, which analyzes the user's voice, facial expressions, and behavioral data in real time. This emotion engine determines the user's current emotional state (e.g., relaxed, stressed, sad, happy). For example, the emotion recognition engine can suggest "sale at a recycle shop" for an image taken while the user is relaxing in front of a bookshelf.

[1029] 4. Proposal Generation and Notification

[1030] The server determines whether an item should be decluttered based on the analyzed details, usage history, and emotional state. Once the determination is made, it selects the optimal decluttering method (e.g., recycle, discard, or preserve) and generates appropriate suggestions for the user. These suggestions take into account the user's emotional state and include advice to reduce stress. For example, specific steps such as "It would be a good idea to sell this book at a thrift store" are presented, along with the location of the thrift store and reference images.

[1031] Specific examples

[1032] For example, if a user takes a photo of a book on a bookshelf, the device extracts detailed information about the book (such as the book title, author name, and publication year). The server determines that the book is not frequently used, and uses an emotion engine to recognize that the user is in a relaxed state. As a result, the server suggests "selling at a recycle shop" and provides directions to the recycle shop and specific steps to follow.

[1033] Prompt Sentence Examples

[1034] "Please advise how this item should be disposed of based on the user's state of relaxation."

[1035] "What is the best way to declutter if the user is stressed?"

[1036] In this way, the present invention makes decluttering suggestions while taking into account the user's emotional state, enabling more efficient and less stressful tidying up.

[1037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1038] Step 1:

[1039] The user takes a picture of an item on the device. The device's camera function is activated, and when the user presses the capture button, high-resolution image data is acquired. The image data is input and passed to the next step in a compressed form.

[1040] Step 2:

[1041] The device sends the captured image data to the server. The compressed image data is sent to the server using HTTP or HTTPS as the communication protocol. The sent image data becomes input and is passed to the analysis module on the server side.

[1042] Step 3:

[1043] The server passes the received image data to a generative model for detailed analysis. The generative model uses algorithms based on machine learning or deep learning to extract attributes of the item (e.g., book title, author, publication year, etc.). At this stage, the image data is input and detailed information about the item is output.

[1044] Step 4:

[1045] Based on the detailed information analyzed by the server, the server accesses the user's usage history database to obtain the item's usage status (e.g., frequency of use, date and time of last use, etc.). The extracted detailed information of the item is input, and usage history information is output.

[1046] Step 5:

[1047] The device uses an emotion recognition engine to analyze the user's current emotional state in real time, taking the user's voice, facial expression, and behavioral data as input. The emotion recognition engine analyzes and outputs the user's emotional state (e.g., relaxed, stressed, sad, happy).

[1048] Step 6:

[1049] Based on the detailed information, usage history, and the user's emotional state, the server determines whether an item should be decluttered. It then generates the optimal tidying method based on the user's emotional state. To achieve this, a generative AI model is used, which inputs the detailed information, usage history, and emotional state, and outputs decluttering suggestions and tidying procedures.

[1050] Step 7:

[1051] The server generates decluttering suggestions and tidying procedures, which are then sent to the device. The suggestions are adjusted based on the user's emotional state. The suggestions and tidying procedures are input, converted into a data format that can be sent to the device, and then output.

[1052] Step 8:

[1053] The device receives the decluttering suggestions and tidying procedures from the server and displays them to the user. This allows the user to actually declutter and tidy up based on the information displayed on the screen. The suggestions sent to the device are input, and the information displayed to the user is output.

[1054] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1055] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1056] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1057] [Fourth embodiment]

[1058] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1059] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1060] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1061] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1062] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1063] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1064] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1065] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1066] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1067] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1068] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1069] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1070] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1071] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[1072] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured. The device then compresses the image and sends it to the server.

[1073] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[1074] The server determines whether the item should be decluttered based on the generated detailed information and usage history. If it is determined that the item should be decluttered, it selects the most appropriate method for the user to declutter. Methods include donating the item, selling it at a recycle shop, or taking it to a recycling center. This information is organized along with specific instructions and reference images and sent to the device.

[1075] The device will then display the received tidying method and reference images to the user. The user can refer to the displayed information to actually declutter and organize. For example, if a specific book is suggested as being sold at a recycle shop, the user can efficiently tidy up by following the suggestion and bringing the book. The app may also display links to related sales sites and tidying support services, which can be used to further smooth the process of decluttering.

[1076] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. It will then suggest selling the book at a recycle shop, and specific instructions, the store's location, and a sample image will be displayed on the device. The user can then use this information to go and sell the book.

[1077] In this way, the present invention provides a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[1078] The processing flow will be explained below.

[1079] Step 1:

[1080] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[1081] Step 2:

[1082] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[1083] Step 3:

[1084] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[1085] Step 4:

[1086] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[1087] Step 5:

[1088] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[1089] Step 6:

[1090] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[1091] Step 7:

[1092] The server determines whether an item should be discarded based on the extracted details and usage history. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[1093] Step 8:

[1094] If the server is determined to be an item that should be discarded, the system selects the most appropriate method for disposal, including donation or sale at a recycle shop, and provides detailed information and reference images.

[1095] Step 9:

[1096] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[1097] Step 10:

[1098] The device analyzes the cleaning method content and reference images received from the server and displays them on the user interface, allowing the user to check the specific cleaning steps and related images.

[1099] Step 11:

[1100] Users can refer to the tidying method displayed on the device and actually declutter and organize their belongings. For example, they can prepare items for donation according to the specified method.

[1101] Step 12:

[1102] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[1103] The above are the specific processing steps of this system.

[1104] Example 1

[1105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1106] In modern society, the increase in the number of items has made decluttering and organizing important issues. However, many users often have difficulty deciding which items to dispose of and how. Furthermore, manually deciding how to organize items is time-consuming, laborious, and inefficient. Therefore, there is a need for a system that supports efficient decluttering and organizing.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1108] In this invention, the server includes means for extracting detailed information about an item, means for determining whether the item should be decluttered based on the usage history, and means for selecting an optimal tidying method and transmitting it to the terminal, thereby enabling the user to declutter and organize efficiently.

[1109] A "terminal" is a device that a user directly operates to take an image of an item.

[1110] The "means for taking images" is a function for taking high-resolution images of an item using a camera built into the terminal.

[1111] The "server" is a device that receives image data sent from the terminal, analyzes it using a generative AI model, and makes a decision on whether to declutter based on detailed information about the item and its usage history.

[1112] A "generative AI model" is a model that uses machine learning and deep learning algorithms to analyze images and extract detailed information about items.

[1113] "Detailed information" refers to attribute information about an item, such as the title, author, and publication year in the case of a book.

[1114] "Usage history" is a record of how a user has used an item, and includes data such as frequency of use and the date and time of last use.

[1115] The "means for determining whether to declutter" is a function for determining whether an item should be decluttered based on detailed information and usage history.

[1116] The "optimal disposal method" refers to the method of selecting the most appropriate way to dispose of an item, including, for example, donating it, selling it at a thrift store, or taking it to a recycling center.

[1117] The "means for displaying suggestions" is a function that visualizes the information on tidying methods sent from the server on the terminal and displays it in a way that is easy for the user to understand.

[1118] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. Specifically, it works by combining a terminal, a server, and a generative AI model, and suggests decluttering and tidying methods based on detailed information and usage history.

[1119] First, the user launches the application on their device and takes a picture of the items they want to organize. The device has a built-in camera that can capture high-resolution images. When the user taps the "shutter button," the image data is saved on the device and then compressed. This compressed image data is then sent from the device to the server.

[1120] The server passes the received image data to a generative AI model to begin analysis. This generative AI model uses machine learning and deep learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, the generative AI model would recognize the title, author, publication year, etc. The server also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[1121] The server determines whether an item should be discarded based on detailed information and usage history. Based on this determination, the server selects the optimal method for discarding the item. Methods include donating, selling at a recycle shop, or taking it to a recycling center. The selected information is organized along with specific procedures and reference images and sent to the device.

[1122] The device then displays the received tidying methods and reference images to the user. The user then uses this information to actually declutter and organize their belongings. For example, if a specific book is suggested as being sold at a recycle shop, the user can follow the suggestion and bring the book to the shop to efficiently declutter. The app may also display links to related sales sites and decluttering support services, which can be used to further smooth the decluttering process.

[1123] For example, when a user takes a photo of a book on their bookshelf, the app extracts detailed information such as the book's title, author, and publication year. The server then determines that the book is infrequently used and identifies it as a candidate for decluttering. It then suggests selling the book at a recycle shop, and specific instructions, the store's location, and a reference image are displayed on the device. The user can use this information to go and sell the book.

[1124] Example prompt sentence:

[1125] "Analyze the images taken by the user, extract detailed information about the items, compare it with usage history data, determine whether decluttering is necessary, and suggest the best way to tidy up."

[1126] In this way, the present invention provides a system that allows users to easily and efficiently declutter and support organizing.

[1127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1128] Step 1:

[1129] The user launches the app and takes a picture of the item.

[1130] Input: User action (launching an app, launching the camera, tapping the shutter button)

[1131] Output: High-resolution image data

[1132] Specific operation: A user takes a picture of an object using the camera on their smartphone or tablet. When the user taps the "shutter button," the device captures the image data and stores it locally.

[1133] Step 2:

[1134] The device compresses the images it takes and sends them to the server.

[1135] Input: High-resolution image data

[1136] Output: Compressed image data (e.g., JPEG format)

[1137] Specific operation: The image data captured by the device is compressed using an appropriate compression algorithm (e.g., JPEG compression) to reduce the data size. The compressed image data is then sent to the server.

[1138] Step 3:

[1139] The server receives the image data and passes it to the generative AI model.

[1140] Input: Compressed image data

[1141] Output: Data input to a generative AI model

[1142] Specific operation: The server receives the compressed image data sent from the device and stores it in a temporary storage location. The image data is then passed to the generative AI model and prepared for analysis.

[1143] Step 4:

[1144] A generative AI model analyzes the image and extracts detailed information about the item.

[1145] Input: Compressed image data

[1146] Output: Detailed information about the item (e.g. title, author, publication year, etc.)

[1147] How it works: The generative AI model analyzes the image data it receives and applies image recognition algorithms to identify detailed information about the item. The resulting information is then stored in a database or passed on to the next processing step.

[1148] Step 5:

[1149] The server checks the usage history and evaluates the usage of the item.

[1150] Input: Item details

[1151] Output: Usage history data (e.g., frequency of use, last used date and time, etc.)

[1152] Specific operation: The server accesses the usage history database based on the detailed information of the item. It obtains the relevant usage history data and analyzes it to evaluate the usage of the item. The evaluation results are used in the next judgment process.

[1153] Step 6:

[1154] The server determines whether or not decluttering is possible based on detailed information and usage history.

[1155] Input: Detailed information on the item, usage history data

[1156] Output: Decision result of whether or not to declutter

[1157] How it works: The server combines the detailed information and usage history data and determines whether an item should be decluttered based on a pre-defined algorithm. For example, the algorithm is set up to identify items that are used infrequently as candidates for decluttering.

[1158] Step 7:

[1159] The server selects the optimal cleaning method and sends it to the device.

[1160] Input: Decision result of whether or not to declutter

[1161] Output: Data suggesting how to clean up (e.g., information on recycle shops, specific steps, reference images)

[1162] Specific operation: The server selects the optimal tidying method based on the results of the decluttering assessment. It generates information about the selected tidying method (e.g., information about the recycle shop, procedures for bringing in items, images, etc.) and sends that data to the device.

[1163] Step 8:

[1164] The device displays the cleaning method and reference images to the user.

[1165] Input: Tidying method suggestion data

[1166] Output: Visual display information to the user

[1167] Specific operation: The device analyzes the data on tidying methods received from the server and displays it through the user interface. Specific steps, reference images, related links, etc. are displayed to make it easy for the user to understand.

[1168] Step 9:

[1169] The user performs the decluttering process as suggested.

[1170] Input: Visual information on how to tidy up

[1171] Output: Items discarded

[1172] Specific actions: The user follows the tidying method displayed on the device and actually declutters. For example, they can take items to a recycle shop or dispose of them in a specified way, efficiently organizing their belongings.

[1173] In this way, a system is realized in which users, devices, servers, and generative AI models work together to efficiently declutter and organize.

[1174] (Application example 1)

[1175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1176] Conventional decluttering and organizing systems only extract detailed information about items and their usage history, but lack the functionality to provide market value or location information for nearby recycle shops, making it difficult for users to decide how to dispose of their items and to dispose of them efficiently. The purpose of this invention is to solve these problems and provide a system that allows users to dispose of items efficiently and rationally.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1178] In this invention, the server includes means for taking images on the terminal, means for transmitting the images to the server, means for receiving the images on the server and analyzing them using a generative model, means for extracting detailed information and usage history through the analysis, means for determining items that should be decluttered based on the extracted information, means for transmitting the determination results and a proposal including a tidying up method to the terminal, means for displaying the transmitted proposal and supporting the user in carrying out the proposal, means for evaluating the market value of the items, and means for providing location information of nearby recycle shops. This allows the user to know the details and market value of the items and select the optimal disposal method, as well as obtain information on nearby recycle shops, enabling efficient decluttering and organizing.

[1179] A "terminal" is an electronic device that can be carried by a user and has the function of taking pictures and communicating with a server.

[1180] The "means for taking an image" is a function for taking an image of an item using a camera or a photography application installed on the terminal.

[1181] The "means for transmitting to a server" is a function for transmitting captured image data to a server via the Internet or other communication means.

[1182] A "generative model" is an artificial intelligence technology that uses methods such as deep learning and machine learning to analyze data and apply the results.

[1183] The "means for analyzing" is a function that analyzes the received image using a generative model on the server and extracts the necessary information.

[1184] "Detailed information" refers to the identification information and characteristics of an item, specifically, information such as the title, author, and publication year for a book.

[1185] "Usage history" is a record of how a user has used a particular item, including frequency of use and the date and time of last use.

[1186] The "means for determining items that should be decluttered" is a function that evaluates the usefulness of an item based on the extracted detailed information and usage history, and determines whether it should be subject to decluttering.

[1187] "Cleaning methods" are specific ways to dispose of or reuse items, such as donating them, selling them at a recycling shop, or taking them to a recycling center.

[1188] The "means for sending suggestions to the terminal" is a function for sending the suggestions and tidying methods generated by the server to the user's terminal.

[1189] "Means for displaying suggestions and assisting the user in implementing them" refers to a function that visually displays the received suggestions on the device and supports the user in implementing them.

[1190] The "means for assessing market value" is a function for estimating the market value of an item based on the analyzed detailed information of the item.

[1191] "Means for providing location information of recycle shops" is a function that provides users with the location and contact information of the nearest recycle shop where they can drop off their items.

[1192] This invention is a system that allows users to efficiently declutter and organize their belongings using a terminal. This system operates by combining a terminal, a server, and a generative model, and suggests decluttering and tidying methods based on detailed information and usage history.

[1193] First, the user launches the application on their device. Next, they take a picture of the items they want to organize using the device's built-in camera. When the user taps the "shutter button," high-resolution image data is captured. The device then compresses the captured image data and sends it to the server.

[1194] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses deep learning and machine learning algorithms to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, publication year, etc. Based on this detailed information, the market value of the item is also estimated.

[1195] The server then accesses a usage history database to check how the user has used the item in the past. This includes frequency of use and the date and time of the last use. Based on this, the server determines whether the item is suitable for decluttering. If it is determined that the item should be decluttered, the server suggests the best way to declutter (for example, donate it, sell it at a recycle shop, or take it to a recycling center).

[1196] The server sends the generated detailed information, the market value of the items, how to organize them, and the location of nearby recycle shops to the device. The device visually displays this information to the user. The user can refer to the displayed information to actually declutter and organize their belongings. In addition, by taking the items to the recycle shop indicated, they can dispose of them effectively.

[1197] As a specific example of processing, the following flow can be considered.

[1198] 1. The user takes a photo of a book on a bookshelf.

[1199] 2. The image is sent to the server, and the generative model extracts detailed information such as "Title: 'Book Title', Author: 'Author Name', Year of Publication: 1997".

[1200] 3. The server checks the usage history and recognizes that the usage frequency is low.

[1201] 4. The server identifies the items as candidates for decluttering and suggests selling them at a recycle shop.

[1202] 5. The server sends the location and business information of the recycle shop to the terminal.

[1203] 6. The user takes the book to a thrift store based on the suggestions.

[1204] An example of a prompt for a generative AI model is as follows:

[1205] Extract detailed information about the following object. The object belongs to the category "Book". Specifically, output the title, author, and publication year.

[1206] As described above, the present invention is a system that provides powerful support for users to easily and efficiently declutter and organize their belongings.

[1207] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1208] Step 1:

[1209] The user uses the device's camera to take a picture of the item they want to organize. The user's input is the image of the item, and the output is high-resolution image data. The user launches the camera application on the device, frames the item, and taps the shutter button. This action generates image data.

[1210] Step 2:

[1211] The device compresses the captured image and sends it to the server. The input is high-resolution image data, and the output is compressed image data. The image compression algorithm in the device operates to compress the image data while maintaining a certain level of quality. The image data is then sent to the server via the Internet.

[1212] Step 3:

[1213] The server passes the received image data to a generative model and begins analysis. The input is compressed image data, and the output is detailed information about the item. The server first decompresses the compressed data and analyzes the image of the item using a generative AI model (for example, a deep learning model using TensorFlow). Specifically, it extracts detailed information such as the item's title, author, and publication year.

[1214] Step 4:

[1215] The server accesses the usage history database and checks the usage history based on the extracted detailed information. The input is the detailed information about the item, and the output is the usage history information. The server queries the database using the detailed information as a key to obtain information such as the frequency of use of the item and the date and time of its last use.

[1216] Step 5:

[1217] The server determines whether an item should be decluttered and evaluates its market value based on detailed information and usage history. The input is detailed information and usage history information, and the output is the decluttering decision result and market value assessment result. The server's algorithm evaluates the input data and, if the item is not used frequently, it is considered for decluttering and its market value is also evaluated.

[1218] Step 6:

[1219] The server selects the optimal tidying method and generates a proposal along with the location information of the recycle shop. The input is the results of the decluttering assessment and the market value assessment, and the output is the tidying method and proposal. The server takes into account the condition and market price of the items and selects the optimal tidying method (e.g., selling at a recycle shop). It also collects the location information of nearby recycle shops and includes it in the proposal.

[1220] Step 7:

[1221] The server sends the generated proposal to the terminal. The input is the proposal, and the output is data sent to the terminal. The server sends the proposal to the user's terminal and notifies them in real time.

[1222] Step 8:

[1223] The terminal displays the received proposals and information about the recycle shop to the user. The input is proposal data from the server, and the output is visual information displayed on the terminal screen. The proposals are displayed using the terminal's display function in a way that allows the user to intuitively understand them.

[1224] Step 9:

[1225] The user refers to the suggestions displayed on the device and takes action to actually dispose of the items. The input is the suggestion information displayed on the device, and the output is the actual disposal of the items. By bringing the items to a recycle shop, the user can efficiently declutter.

[1226] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1227] The present invention is a system that efficiently uses a terminal to declutter and organize items, and further adjusts tidying suggestions by recognizing the user's emotions. This system operates by combining a terminal, a server, a generative model, and an emotion engine, and suggests decluttering and tidying methods based on detailed information, usage history, and the user's emotional state.

[1228] First, the user launches the application on their device and takes a picture of the items they want to organize. The device uses its built-in camera to capture a high-resolution image. When the user taps the shutter button, the image data is captured, and then the device compresses the image and sends it to the server.

[1229] On the server side, the received image data is passed to a generative model, where analysis begins. The generative model uses algorithms based on machine learning and deep learning to analyze the image and extract detailed information about the item. For example, in the case of a book, it can recognize the title, author, and publication year. It also accesses a usage history database to check how the user has used the item in the past. The usage history includes information such as frequency of use and the date and time of last use.

[1230] Furthermore, the device uses an emotion engine to analyze the user's emotional state in real time. The emotion engine recognizes emotions based on the user's voice, facial expressions, and behavioral data to determine the user's current emotional state. For example, it can recognize whether the user is feeling stressed or relaxed.

[1231] The server determines whether an item should be decluttered based on the generated detailed information, usage history, and the user's emotional state. If it determines that decluttering is necessary, it selects the most appropriate tidying method for the user. In doing so, it refers to the emotion recognition results and adjusts the suggestions. For example, if the user is feeling stressed, it will suggest simple tidying methods or methods that are expected to have a positive effect. The suggestions include specific steps and reference images and are sent to the device.

[1232] The device displays the tidying method and reference images received from the server to the user. The user can refer to the displayed information to actually declutter and organize their belongings. For example, if a book is suggested as "sold at a recycle shop," the user can proceed with preparations to take it to the recycle shop. Advice based on emotion recognition results is also displayed, allowing the user to maintain positive emotions while tidying up.

[1233] For example, if a user takes a photo of a book on their bookshelf, the app will extract detailed information about the book, such as "Harry Potter and the Philosopher's Stone, published in 1997 by J.K. Rowling." The server will then determine that the book is infrequently used and identify it as a candidate for decluttering. Furthermore, the emotion engine will recognize that the user is relaxed and suggest selling the book at a recycle shop. The suggestion includes specific steps, the store's location, and a reference image, and is displayed on the device. The user can use this information to go and sell the book.

[1234] In this way, the present invention is a system that provides powerful support for decluttering and tidying up easily and efficiently while taking into consideration the user's feelings.

[1235] The processing flow will be explained below.

[1236] Step 1:

[1237] A user launches an app on their smartphone, taps the app icon, and the home screen appears, which includes a "shoot" button.

[1238] Step 2:

[1239] The user taps the "Photo" button in the app to activate the camera, points the camera at the object, and presses the "Shutter Button" to take a photo of the item. The device then acquires this image data.

[1240] Step 3:

[1241] The image data captured by the device is compressed. The compressed image data is sent to the server via an HTTP POST request. Error handling is also implemented to ensure communication stability.

[1242] Step 4:

[1243] The server receives the HTTP request and saves the image data contained in the request body, which is later used for analysis.

[1244] Step 5:

[1245] The server begins image analysis. The stored image data is input into a generative model to extract detailed information about the object. The generative model then uses machine learning algorithms to analyze the characters and features in the image.

[1246] Step 6:

[1247] The generative model extracts detailed information about the object and the user's usage history. For example, if it is a photo of a book, it recognizes the title, author, publication year, etc., and obtains the frequency of use and the last date and time of use from a usage history database.

[1248] Step 7:

[1249] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes emotions based on the user's voice, facial expressions, and behavioral data to determine their current emotional state. It analyzes in real time whether the user is feeling stressed or relaxed.

[1250] Step 8:

[1251] The server determines whether an item should be discarded based on detailed information, usage history, and the user's emotional state. For example, if an item has not been used for more than two years, it is a candidate for discarding.

[1252] Step 9:

[1253] If the server determines that an item should be discarded, it selects the optimal method for decluttering, including donating it or selling it at a recycle shop. It also refers to the emotion recognition results and suggests simple decluttering methods if the user is stressed, or more proactive suggestions if the user is relaxed.

[1254] Step 10:

[1255] The server sends the cleaning instructions and reference images to the device, which converts the data into an appropriate format (e.g., JSON) and returns it as an HTTP response.

[1256] Step 11:

[1257] The device analyzes the tidying method content and reference images received from the server and displays them on the user interface, allowing the user to check specific tidying steps and related images. In addition, positive advice provided by the emotion engine is also displayed.

[1258] Step 12:

[1259] The user actually declutters and organizes their belongings by referring to the tidying method displayed on the device. For example, they put into practice the recommendation made by the emotion engine in a relaxed state to take their belongings to a recycling shop.

[1260] Step 13:

[1261] If users wish to use additional services (sales sites or decluttering support services), they can access them by tapping the link displayed on their device, which will make the decluttering process even smoother.

[1262] The above are the specific processing steps of this system.

[1263] Example 2

[1264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1265] Conventional decluttering and tidying systems provide uniform suggestions without considering the user's emotional state, which can be stressful for many users. Furthermore, they do not effectively utilize detailed information about items or usage history, making it difficult to provide optimal suggestions for users. Furthermore, if the tidying methods provided by the system are not appropriate for a particular emotional state, users may find it difficult to follow the suggestions.

[1266] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1267] In this invention, the server includes means for receiving images and analyzing them using a generative AI model, means for extracting detailed information and usage history of items through analysis, means for determining items to be decluttered based on the extracted information and the user's emotional state, means for sending suggestions including tidying methods according to the user's emotional state to the terminal, and means for displaying the sent suggestions and supporting the user in carrying them out. This enables efficient and stress-free decluttering and tidying while taking the user's emotions into consideration.

[1268] 1. "Terminal" means a type of electronic device operated by a user, which has hardware such as a camera, microphone, and display, and has the ability to run applications.

[1269] 2. "Server" means a computer system that receives data sent from a terminal via a network and performs processing such as analysis and database management.

[1270] 3. A "generative AI model" is a type of artificial intelligence that uses machine learning and deep learning algorithms to analyze data and extract detailed information.

[1271] 4. "Image analysis" is the process of taking image data as input and extracting information about an item or other object from it.

[1272] 5. "Detailed information" refers to specific characteristics and attributes of the photographed item, such as the title, author, and publication year.

[1273] 6. "Usage history" refers to data about how a user has used an item in the past, including frequency of use and the date and time of last use.

[1274] 7. An "emotion engine" is a software or hardware system that analyzes a user's emotional state based on their voice, facial expression, and behavioral data.

[1275] 8. "Danshari" is the process of organizing and disposing of items that users have determined to be unnecessary.

[1276] 9. "Tidying up method" refers to the specific means and procedures for organizing and tidying up items.

[1277] 10. "Suggestions" are specific advice or steps regarding decluttering and tidying up that are sent from the server to the device.

[1278] This invention is a system that uses a device to efficiently declutter and organize items, and further recognizes the user's emotions and adjusts tidying suggestions accordingly. This system operates by combining a device, a server, a generative AI model, and an emotion engine.

[1279] First, the user launches the dedicated Danshari application using their device. The initial screen of the application is displayed, providing an interface for starting to organize items. The user then takes a photo of the item they want to organize using the device's camera. Specifically, when the user taps the "photo button," the device's camera takes a high-resolution image. The device's camera module captures the image and stores it in the device's internal memory.

[1280] Next, the device compresses the captured image data in JPEG format or similar and sends it to the server. The device's network module sends the image data to the server using an HTTP POST request. The server receives the HTTP POST request and uses web server software (e.g., NGINX or Apache) to save the image data in a storage directory.

[1281] The server then runs the generative AI model to analyze the received image. During this analysis, detailed information about the object contained in the image (e.g., title, author, publication year, etc.) is extracted. The server then runs a Python script to apply the generative AI model using a machine learning library such as TensorFlow.

[1282] The server then accesses a usage history database to see how the user has used the item in the past, and executes an SQL query to retrieve the relevant usage history from the database, including frequency of use and the last time it was used.

[1283] The device then runs an emotion engine to analyze the user's emotional state. The emotion engine recognizes the user's emotional state by collecting data on their voice, facial expressions, and behavior. Specifically, the device collects data using its microphone and camera, and passes it to an emotion engine such as Affectiva's SDK for analysis.

[1284] The server determines whether items should be decluttered based on detailed information, usage history, and emotion recognition results generated by the server. The server executes logic and evaluates the need for decluttering according to conditional branching. If it determines that decluttering is necessary, it suggests the optimal tidying method based on the user's emotional state. The server inputs prompts into the generative AI model and obtains the output results. This makes it possible to suggest simple tidying methods or methods that are expected to have a positive effect if the user is feeling stressed.

[1285] A specific example is given below.

[1286] Example: When a user takes a photo of a book on a bookshelf, the device's camera captures an image of the book. The device compresses the image and sends it to the server. The server uses a generative AI model to extract detailed information such as "Harry Potter and the Philosopher's Stone, published in 1997, by J.K. Rowling." The server determines that the book is infrequently used and makes it a candidate for decluttering. The emotion engine recognizes that the user is relaxed and suggests "selling at a recycle shop." The suggestion includes specific steps, the store's location, and a reference image. The device displays the suggestion to the user, who then begins preparing to take the book to the recycle shop.

[1287] Example prompt for a generative AI model:

[1288] "I took a picture of a book on my bookshelf. Please extract detailed information about this book and help me decide whether to get rid of it."

[1289] "Please suggest the best way for users to clean up when they are feeling stressed."

[1290] As described above, this system takes into consideration the user's emotions and enables efficient and stress-free decluttering and tidying.

[1291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] The user operates the device and launches the dedicated Danshari application. The device displays the initial screen of the application.

[1294] Input: Launch application

[1295] Output: Display of initial screen

[1296] Specific operations: The device operating system loads the application and renders the user interface.

[1297] Step 2:

[1298] The user takes a photo of the item they want to organize using their device's camera. When the user taps the "photo button," the device's camera captures a high-resolution image.

[1299] Input: Photo of item

[1300] Output: Captured image data

[1301] Specific operation: The camera module captures an image and saves it in the internal memory in JPEG format.

[1302] Step 3:

[1303] The image data captured by the terminal is compressed and sent to the server.

[1304] Input: Photographed image data

[1305] Output: Compressed image data and HTTP POST request

[1306] Specific operation: The terminal software compresses the image in JPEG format and uses the network module to send an HTTP POST request.

[1307] Step 4:

[1308] The server receives the HTTP POST request and saves the image data.

[1309] Input: Compressed image data

[1310] Output: Image data stored on the server

[1311] What happens: Web server software (e.g., NGINX or Apache) processes the request and stores the image in a database or on the file system.

[1312] Step 5:

[1313] The server runs the generative AI model to analyze the received image and extract detailed information (title, author, publication year, etc.).

[1314] Input: Image data stored on the server

[1315] Output: Extracted details

[1316] What it does: A Python script applies a generative AI model using a machine learning library such as TensorFlow to perform image analysis.

[1317] Step 6:

[1318] The server accesses the usage history database and extracts the usage history of the item.

[1319] Input: Extracted details

[1320] Output: Item usage history

[1321] Specific operation: Executes an SQL query and retrieves the relevant usage history from the database.

[1322] Step 7:

[1323] The device runs an emotion engine to analyze the user's emotional state, collecting the user's voice, facial expression, and behavioral data.

[1324] Input: User's voice, facial expression, and behavior data

[1325] Output: User's emotional state

[1326] Specific operation: Collects data using the device's microphone and camera and passes it to an emotion engine such as Affectiva's SDK.

[1327] Step 8:

[1328] The server determines whether an item should be decluttered based on detailed information, usage history, and emotion recognition results.

[1329] Input: detailed information, usage history, emotion recognition results

[1330] Output: Decluttering results

[1331] What it does: The server logic synthesizes the information and evaluates the need for decluttering based on conditional branching.

[1332] Step 9:

[1333] The server uses a generative AI model to suggest the optimal tidying method.

[1334] Input: Decision result of decluttering, user's emotional state

[1335] Output: Tidying up suggestions

[1336] Specific operation: Input a prompt into the generative AI model, obtain the output, and generate a cleaning method.

[1337] Step 10:

[1338] The server sends the generated proposal to the terminal.

[1339] Input: Tidying up suggestions, reference images

[1340] Output: Proposal as HTTP response

[1341] Specific operation: The proposal content is sent to the terminal as an HTTP response.

[1342] Step 11:

[1343] The terminal displays the received proposal to the user.

[1344] Input: Proposal content, reference image

[1345] Output: Display of proposal

[1346] Specific behavior: The suggestions are displayed as text and images on the device display, and the user interface is updated.

[1347] Through the above steps, this system is able to support efficient decluttering and tidying while taking into consideration the user's emotions.

[1348] (Application example 2)

[1349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1350] In modern society, many people find it difficult to organize and declutter. Deciding which items to discard or how to proceed with decluttering can be particularly difficult and stressful, as emotions are often involved. Conventional systems lack the technology to provide optimal decluttering recommendations while taking into account the user's emotional state, preventing efficient and positive decluttering and tidying.

[1351] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1352] In this invention, the server includes means for taking an image on the terminal, means for transmitting the image to the server, means for receiving the image on the server and analyzing it using a generative model, means for extracting detailed information and usage history through the analysis, means including an emotion recognition engine for analyzing the user's emotional state in real time, means for adjusting the content of decluttering suggestions based on the emotional state, means for transmitting suggestions including the assessment results and tidying methods to the terminal, and means for displaying the transmitted suggestions and supporting the user in implementing them. This enables efficient and positive decluttering and tidying while taking the user's emotional state into consideration.

[1353] A "terminal" is a computing device that allows a user to take images and communicate with a server.

[1354] "Image" refers to visual information captured by a user using a device, and is the data to be analyzed.

[1355] A "server" is a central computing system for receiving image data sent from a terminal and for analyzing and processing the information.

[1356] A "generative model" is an algorithmic system that uses machine learning and deep learning to analyze image data and extract detailed information.

[1357] "Detailed information" refers to the identification data of an item obtained through image analysis (e.g., book title, author, publication year, etc.).

[1358] "Usage history" refers to a record of how a user has used the item in the past (e.g., frequency of use, date of last use, etc.).

[1359] "Danshari" is the process of organizing your home by throwing away unnecessary items and keeping only what is necessary.

[1360] An "emotion recognition engine" is a system that analyzes a user's voice, facial expression, and behavioral data in real time to determine the user's emotional state.

[1361] "Suggestions" are advice on how to tidy up or declutter that the server provides to the user based on the generative model and emotion recognition results.

[1362] "Sending" is the communication process by which the server delivers image data, analysis results, and proposals to the terminal.

[1363] "Display" refers to the act of visually presenting the suggestions and information received by the terminal to the user.

[1364] "Assistance" is the guidance or help provided to help users implement the suggested tidying methods.

[1365] The system based on this invention provides support for users to declutter and tidy up efficiently and positively. Specifically, the system allows users to use a device to take pictures of items, send the image data to a server, and the server generates decluttering suggestions based on detailed information, usage history, and the user's emotional state.

[1366] The system program is configured as follows:

[1367] 1. Image capture and transmission

[1368] The device is equipped with a camera function that allows the user to take pictures of the items they want to organize. The user activates the camera, takes a picture of the items they want to organize (e.g., books, clothes, furniture, etc.), and taps the shutter button. The image data is compressed and sent to the server, which then receives high-resolution visual data.

[1369] 2. Analysis of image data

[1370] The server then passes the received image data to a generative model based on machine learning or deep learning to extract detailed information. This model uses a known database to determine the item's category and characteristics (e.g., book title, author, publication year, etc.). The server then accesses a database of the user's usage history to determine how the item has been used in the past. This usage history includes frequency of use and the date and time of last use.

[1371] 3. Recognizing emotional states

[1372] The device is equipped with an emotion engine for emotion recognition, which analyzes the user's voice, facial expressions, and behavioral data in real time. This emotion engine determines the user's current emotional state (e.g., relaxed, stressed, sad, happy). For example, the emotion recognition engine can suggest "sale at a recycle shop" for an image taken while the user is relaxing in front of a bookshelf.

[1373] 4. Proposal Generation and Notification

[1374] The server determines whether an item should be decluttered based on the analyzed details, usage history, and emotional state. Once the determination is made, it selects the optimal decluttering method (e.g., recycle, discard, or preserve) and generates appropriate suggestions for the user. These suggestions take into account the user's emotional state and include advice to reduce stress. For example, specific steps such as "It would be a good idea to sell this book at a thrift store" are presented, along with the location of the thrift store and reference images.

[1375] Specific examples

[1376] For example, if a user takes a photo of a book on a bookshelf, the device extracts detailed information about the book (such as the book title, author name, and publication year). The server determines that the book is not frequently used, and uses an emotion engine to recognize that the user is in a relaxed state. As a result, the server suggests "selling at a recycle shop" and provides directions to the recycle shop and specific steps to follow.

[1377] Prompt Sentence Examples

[1378] "Please advise how this item should be disposed of based on the user's state of relaxation."

[1379] "What is the best way to declutter if the user is stressed?"

[1380] In this way, the present invention makes decluttering suggestions while taking into account the user's emotional state, enabling more efficient and less stressful tidying up.

[1381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1382] Step 1:

[1383] The user takes a picture of an item on the device. The device's camera function is activated, and when the user presses the capture button, high-resolution image data is acquired. The image data is input and passed to the next step in a compressed form.

[1384] Step 2:

[1385] The device sends the captured image data to the server. The compressed image data is sent to the server using HTTP or HTTPS as the communication protocol. The sent image data becomes input and is passed to the analysis module on the server side.

[1386] Step 3:

[1387] The server passes the received image data to a generative model for detailed analysis. The generative model uses algorithms based on machine learning or deep learning to extract attributes of the item (e.g., book title, author, publication year, etc.). At this stage, the image data is input and detailed information about the item is output.

[1388] Step 4:

[1389] Based on the detailed information analyzed by the server, the server accesses the user's usage history database to obtain the item's usage status (e.g., frequency of use, date and time of last use, etc.). The extracted detailed information of the item is input, and usage history information is output.

[1390] Step 5:

[1391] The device uses an emotion recognition engine to analyze the user's current emotional state in real time, taking the user's voice, facial expression, and behavioral data as input. The emotion recognition engine analyzes and outputs the user's emotional state (e.g., relaxed, stressed, sad, happy).

[1392] Step 6:

[1393] Based on the detailed information, usage history, and the user's emotional state, the server determines whether an item should be decluttered. It then generates the optimal tidying method based on the user's emotional state. To achieve this, a generative AI model is used, which inputs the detailed information, usage history, and emotional state, and outputs decluttering suggestions and tidying procedures.

[1394] Step 7:

[1395] The server generates decluttering suggestions and tidying procedures, which are then sent to the device. The suggestions are adjusted based on the user's emotional state. The suggestions and tidying procedures are input, converted into a data format that can be sent to the device, and then output.

[1396] Step 8:

[1397] The device receives the decluttering suggestions and tidying procedures from the server and displays them to the user. This allows the user to actually declutter and tidy up based on the information displayed on the screen. The suggestions sent to the device are input, and the information displayed to the user is output.

[1398] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1399] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1400] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1401] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1402] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1403] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1404] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1405] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1406] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1407] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1408] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1409] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1410] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1411] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1412] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1413] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1414] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1415] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1416] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1417] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1418] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1419] The following is further disclosed regarding the above embodiment.

[1420] (Claim 1)

[1421] means for capturing an image in the terminal;

[1422] means for transmitting the image to a server;

[1423] means for receiving the image at a server and analyzing it using a generative model;

[1424] means for extracting detailed information and usage history through said analysis;

[1425] A means for determining items to be discarded based on the extracted information;

[1426] means for transmitting the determination result and a proposal including a tidying up method to a terminal;

[1427] The system includes means for displaying the submitted proposal and assisting the user in implementing it.

[1428] (Claim 2)

[1429] 2. The system according to claim 1, further comprising means for automatically selecting a tidying up method based on the usage history.

[1430] (Claim 3)

[1431] The system according to claim 1, further comprising means for providing a link to a sales site for storage products and disposal of unwanted items, and a function for introducing tidying up support services.

[1432] "Example 1"

[1433] (Claim 1)

[1434] means for capturing an image in the terminal;

[1435] means for transmitting the image to a server;

[1436] A means for receiving the image in a server and analyzing it using a generative AI model;

[1437] means for extracting detailed information of the item through said analysis;

[1438] A means for determining whether an item should be discarded based on the detailed information and usage history;

[1439] means for transmitting the determination result and a proposal including a tidying up method to a terminal;

[1440] The system includes means for displaying the submitted proposal and assisting the user in implementing it.

[1441] (Claim 2)

[1442] 2. The system according to claim 1, further comprising means for automatically selecting a tidying up method based on the usage history.

[1443] (Claim 3)

[1444] The system according to claim 1, further comprising means for providing a link to a sales site for storage products and disposal of unwanted items, and a function for introducing tidying up support services.

[1445] "Application Example 1"

[1446] (Claim 1)

[1447] means for capturing an image in the terminal;

[1448] means for transmitting the image to a server;

[1449] means for receiving the image at a server and analyzing it using a generative model;

[1450] means for extracting detailed information and usage history through said analysis;

[1451] A means for determining items to be discarded based on the extracted information;

[1452] means for transmitting the determination result and a proposal including a tidying up method to a terminal;

[1453] means for displaying the submitted suggestions and assisting the user in implementing them;

[1454] a means of assessing the market value of the goods;

[1455] A system including a means for providing location information of nearby thrift stores.

[1456] (Claim 2)

[1457] 2. The system according to claim 1, further comprising means for automatically selecting a tidying up method based on the usage history.

[1458] (Claim 3)

[1459] The system according to claim 1, further comprising means for providing a link to a sales site for storage products and disposal of unwanted items, and a function for introducing tidying up support services.

[1460] "Example 2: Combining Emotion Engines"

[1461] (Claim 1)

[1462] means for capturing an image in the terminal;

[1463] means for transmitting the image to a server;

[1464] A means for receiving the image in a server and analyzing it using a generative AI model;

[1465] means for extracting detailed information and usage history of the item through the analysis;

[1466] A means for determining items to be decluttered based on the extracted information and the user's emotional state;

[1467] means for transmitting to a terminal a suggestion including a tidying up method according to the determination result and the user's emotional state;

[1468] The system includes means for displaying the submitted proposal and assisting the user in implementing it.

[1469] (Claim 2)

[1470] The system according to claim 1, further comprising means for automatically selecting a tidying up method based on usage history and the user's emotional state.

[1471] (Claim 3)

[1472] The system according to claim 1, further comprising means for providing a link to a sales site for storage products and disposal of unwanted items, and a function for introducing tidying up support services.

[1473] "Application example 2 when combining emotion engines"

[1474] (Claim 1)

[1475] means for capturing an image in the terminal;

[1476] means for transmitting the image to a server;

[1477] means for receiving the image at a server and analyzing it using a generative model;

[1478] means for extracting detailed information and usage history through said analysis;

[1479] A means for determining items to be discarded based on the extracted information;

[1480] means including an emotion recognition engine for analyzing the user's emotional state in real time;

[1481] A means for adjusting the content of the decluttering suggestion based on the emotional state;

[1482] means for transmitting the determination result and a proposal including a tidying up method to a terminal;

[1483] means for displaying the submitted proposal and assisting the user in implementing it;

[1484] A system including:

[1485] (Claim 2)

[1486] 2. The system according to claim 1, further comprising means for automatically selecting a tidying up method based on the usage history.

[1487] (Claim 3)

[1488] The system according to claim 1, further comprising means for providing a link to a sales site for storage products and disposal of unwanted items, and a function for introducing tidying up support services. [Explanation of symbols]

[1489] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing an image in the terminal; means for transmitting the image to a server; means for receiving the image at a server and analyzing it using a generative model; means for extracting detailed information and usage history through said analysis; A means for determining items to be discarded based on the extracted information; means for transmitting the determination result and a proposal including a tidying up method to a terminal; The system includes means for displaying the submitted proposal and assisting the user in implementing it.

2. The system according to claim 1, further comprising means for automatically selecting a tidying up method based on the usage history.

3. The system according to claim 1, further comprising means for providing a link to a sales site relating to storage products and disposal of unwanted items, and a function for introducing tidying up support services.

Images