system

Abstract

To provide a system capable of efficiently promoting tidying up. [Solution] A system including a means for acquiring image data, a means for transmitting the acquired image data to a cloud server, a means for analyzing the image data and identifying objects on the cloud server, a means for evaluating the condition of the room based on the analysis results, a means for generating advice regarding the condition of the room and providing it to the user, a means for providing a user interface for labeling items, a means for transmitting label information to the cloud server, a means for tracking the frequency of use of items on the cloud server, a means for generating decluttering advice regarding less frequently used items, and a means for providing information for linking with waste disposal companies and flea market apps.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Because a person's living space is private and difficult to show or discuss with others, it is difficult to efficiently organize and declutter. Furthermore, many people are unable to declutter or declutter due to time constraints or elderly parents. This leads to concerns that progress in organizing will be hindered and the living environment will deteriorate. The purpose of this invention is to provide technology that can solve these difficulties and efficiently organize and declutter. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system including means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects on the cloud server, means for evaluating the condition of the room based on the analysis results, means for generating advice regarding the condition of the room and providing it to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of use of items on the cloud server, means for generating decluttering advice regarding infrequently used items, and means for providing information for linking with waste disposal companies and flea market apps.

[0006] "Means for acquiring image data" refers to a function that allows a user to take a photo of a room using a device such as a smartphone.

[0007] A "cloud server" refers to a remote data storage and computing resource accessible via the Internet, and is a server that has the functionality to analyze and evaluate image data.

[0008] "Means for analyzing image data" refers to a function that uses an algorithm executed on a cloud server to identify objects from the transmitted image data and determine their names and location information.

[0009] The "means for identifying objects" refers to an algorithm that detects each element in the transmitted image data and categorizes it as what it is.

[0010] The "means for evaluating the condition of a room" refers to an algorithm that quantifies and evaluates the overall tidiness and storage efficiency of a room based on analyzed object information.

[0011] "Means for generating advice" refers to a function that creates specific suggestions regarding the placement and storage methods of items based on the results of the room evaluation.

[0012] "User Interface" refers to the screens and input means through which a user interacts with the system and labels items through an application.

[0013] "Label information" refers to data regarding categories and frequency of use entered by the user for each item.

[0014] The "means for tracking frequency of use" refers to an algorithm that analyzes image data sent periodically by the cloud server and monitors the usage of each item.

[0015] "Means for generating decluttering advice" refers to a function that automatically provides suggestions on how to dispose of items that are no longer needed based on the results of usage frequency tracking.

[0016] "Means for providing information for linking with waste disposal companies and flea market apps" refers to a function in which the cloud server presents users with information on appropriate waste disposal companies and flea market apps based on the results of waste classification, thereby supporting linking. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a system that utilizes AI to support efficient organizing and storage, and decluttering. Below, we will generate a program for this system and explain the program's processing in natural language.

[0039] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[0040] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[0041] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[0042] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[0043] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[0044] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[0045] Users use the app to label magazines and other items, then take a photo of their room and send it to the cloud server, which compares the old and new images to analyze how often items are used. For items that are used less frequently, the app offers recommendations for recycling or selling them.

[0046] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0047] The processing flow will be explained below.

[0048] Step 1:

[0049] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[0050] Step 2:

[0051] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[0052] Step 3:

[0053] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[0054] Step 4:

[0055] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[0056] Step 5:

[0057] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[0058] Step 6:

[0059] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[0060] Step 7:

[0061] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[0062] Step 8:

[0063] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[0064] Step 9:

[0065] The device sends the labeled information to the server, where it is stored in an item database.

[0066] Step 10:

[0067] The server periodically sends reminders to the user to take photos of the room again, and the user who receives the reminder takes photos of the room again.

[0068] Step 11:

[0069] The user takes another photo of the room and the device sends this new image data to the server.

[0070] Step 12:

[0071] The server receives the new image data, analyzes it, and compares it with the previous image data. The image analysis engine again identifies the object's location and category, and updates the usage frequency data.

[0072] Step 13:

[0073] The server identifies items that are used less frequently based on the updated usage frequency data, and uses this identified data to generate effective decluttering advice.

[0074] Step 14:

[0075] The server then sends the generated decluttering advice to the device, including advice on recycling, referrals to waste disposal companies, and selling items on a flea market app.

[0076] Step 15:

[0077] The device displays decluttering advice to the user and, if necessary, provides link information for using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[0078] This process allows the user to efficiently organize and tidy up their room.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] In the past, organizing and decluttering a room required a lot of effort, and it was difficult to get specific advice on how to organize and properly dispose of items.It was also difficult to understand how often items were used and how to declutter efficiently.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes means for analyzing image data and identifying objects, means for providing a user interface for tagging the identified objects, and means for evaluating the degree of tidiness of a room based on the analysis results, thereby making it possible to automatically evaluate the tidiness of a room and provide the user with specific advice on tidiness and decluttering.

[0084] "Image data" refers to digital data of still images or moving images recorded as visual information.

[0085] A "cloud server" is a remote server that processes data and provides storage over a network.

[0086] "Means for identifying objects" refers to technology that analyzes image data to recognize and classify the various objects contained therein.

[0087] "User interface" refers to the operation screen and input device that allow the user to directly interact with the system.

[0088] A "means for assessing the degree of tidiness" is a technique that uses an algorithm or method for assessing the state of a room to express the level of tidiness as a number or evaluation comment.

[0089] "Label information" is a tag or identifier used to indicate additional information such as the category or frequency of use of an object.

[0090] "Means for tracking frequency of use" refers to technology that records and analyzes the usage of each object to determine frequency.

[0091] The "means for generating decluttering advice" is a technology that generates specific suggestions that instruct how to dispose of infrequently used objects.

[0092] "Information on collaboration with waste disposal companies and marketplaces" refers to data on companies and trading platforms necessary for properly disposing of or selling unwanted items.

[0093] This invention relates to a system that uses AI to efficiently support organizing and storing things. Below, we will create a program for this system and explain how it works.

[0094] This system includes hardware and software such as a user's device (e.g., a smartphone), a cloud server, and an image analysis algorithm (specifically, it uses machine learning frameworks such as TENSORFLOW (registered trademark) and PyTorch). The specific process is as follows:

[0095] First, the user takes a photo of the room using their smartphone. This is done using a dedicated application. The captured photo data is then sent from the device to a cloud server. The sending method is, for example, the HTTPS protocol. If an error occurs during transmission, the device has the ability to attempt to resend the data.

[0096] The cloud server decompresses the received image data and begins image analysis. This analysis uses machine learning models such as TensorFlow to execute object detection algorithms. The server identifies each object in the image and assigns it a category (e.g., sofa, table, book, etc.). It also obtains each object's location information (coordinate data).

[0097] The cloud server then evaluates the room's tidiness based on the identified object information. This evaluation analyzes the number of recognized objects and their category arrangement, calculating, for example, the number of scattered items and the utilization rate of storage space. The server then quantifies these evaluation results and generates a rating such as "poorly organized" or "somewhat organized."

[0098] Based on the evaluation results, the cloud server generates specific advice on tidying up the room, such as "put magazines in a storage box" or "put away items on the table." The advice is sent to the device in text or illustration format.

[0099] The user checks the advice provided through the device and attaches tags or labels to each item. These labels include information such as the item's category and frequency of use. The device then sends the label information entered by the user to the cloud server.

[0100] The cloud server stores the received label information in a database and runs an analysis algorithm to track the frequency of use of each item. Periodically, the user takes new photos of the room and sends them to the cloud server from their device. The server compares the old and new image data to identify items that have been used less frequently.

[0101] Finally, the cloud server suggests efficient ways to declutter infrequently used items. For example, it recommends recycling methods for recyclable items and selling items that are still usable on a marketplace. Based on the item classification results, the server generates link information for appropriate waste disposal companies and marketplaces and sends this information to the device.

[0102] As a concrete example, consider the case where a user takes a photo of their living room and sends it to the server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device. The user then uses the app to label the magazines and other items, takes another photo of the room, and sends it to the server. The cloud server compares the old and new images and analyzes how often the items are used. For items that are used less frequently, it provides recommendations for recycling or selling them.

[0103] An example of a prompt is, "Take a photo of your living room and send it to the server through the app. You will soon see a tidy-up rating and specific advice."

[0104] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0105] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0106] Step 1:

[0107] The user takes a photo of the room using their smartphone. They select a room that contains a wide range of visual information, such as a living room. After taking the photo, they press the "Upload" button to proceed to the next step.

[0108] Input: Photo of the room (image data)

[0109] Output: Image data saved on the device

[0110] Step 2:

[0111] The device will send the captured photo data to the cloud server. The photo data will be compressed first and then securely transmitted using the HTTPS protocol. The retry function will be enabled until the transmission is complete.

[0112] Input: Image data stored on the device

[0113] Output: Image data sent to the cloud server

[0114] Step 3:

[0115] The server decompresses the received image data and begins analyzing it using the TensorFlow model. The image analysis algorithm performs object detection and identifies objects in the image.

[0116] Input: Image data sent to the cloud server

[0117] Output: Identified object category and location information (coordinate data)

[0118] Step 4:

[0119] The server evaluates the room's tidiness based on the identified object information, using an analytical algorithm to analyze the number of objects and their category arrangement, and calculates the "number of cluttered items" and "storage space utilization rate."

[0120] Input: Identified object category and location information

[0121] Output: Organization score and evaluation results

[0122] Step 5:

[0123] Based on the evaluation results, the server generates specific advice on tidying up the room. For example, the advice includes instructions such as "put magazines in a storage box" and "put away items on the table." The advice is generated in the form of text and illustrations.

[0124] Input: Organization score and evaluation results

[0125] Output: Advice text and illustration data

[0126] Step 6:

[0127] The server sends the generated advice to the terminal, where the user can see specific instructions on how to tidy up their room.

[0128] Input: Advice text and illustration data

[0129] Output: Advice sent to terminal

[0130] Step 7:

[0131] The user tags or labels each item based on the advice provided. The label contains information such as the object's category and frequency of use. When the user presses the "Done tagging" button, the label information is sent to the server.

[0132] Input: Label information added based on the advice

[0133] Output: Label information sent to the cloud server

[0134] Step 8:

[0135] The server stores the received label information in a database and starts tracking usage frequency. It runs an analysis algorithm to accumulate data and record the frequency of use of each item.

[0136] Input: Label information sent to the cloud server

[0137] Output: Usage frequency information stored in a database

[0138] Step 9:

[0139] Periodically, the user takes another photo of the room and presses the "Reevaluate" button to send the image data from the device to the cloud server, where it is analyzed.

[0140] Input: Newly taken photo of the room (image data)

[0141] Output: New image data sent to the cloud server

[0142] Step 10:

[0143] The server compares the old and new image data to identify less frequently used items, and generates advice on how to efficiently declutter.

[0144] Input: Old and new image data

[0145] Output: A list of items that are rarely used and advice on decluttering

[0146] Step 11:

[0147] The server generates linked information about waste disposal companies and marketplaces based on the item classification results, along with decluttering advice, and sends this to the terminal.

[0148] Input: Decluttering advice and item classification results

[0149] Output: Decluttering advice and collaboration information sent to the device

[0150] (Application example 1)

[0151] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0152] Until now, support systems have been provided to help individuals efficiently organize their homes, but there are limitations to these systems in terms of improving the efficiency of inventory management and product display in retail stores and brick-and-mortar shops, as well as providing appropriate advice based on product usage frequency and sales data. In particular, there is a need for appropriate disposal of infrequently used and unsold products. It is also important to propose efficient product display methods and improve display methods. However, current systems are unable to adequately address these issues.

[0153] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0154] In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the condition of a room based on the analysis results, means for generating and providing advice regarding the condition of the room to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of use of items in the cloud server, means for generating decluttering advice regarding infrequently used items, means for providing information for linking with waste disposal companies and recycling applications, means for evaluating the tidiness of product displays and providing advice regarding improving the display method, and means for identifying infrequently used products and suggesting sales promotions or returns. This enables more efficient inventory management and product display in retail stores and brick-and-mortar stores, and the provision of appropriate advice based on the frequency of product use.

[0155] "Image data" refers to a collection of electronically acquired visual information that can be analyzed to identify an object.

[0156] A "cloud server" is a server that stores and processes data via the Internet, and is responsible for processing information in cooperation with user devices.

[0157] "Means for identifying objects" refers to the algorithms or software used to identify specific objects from image data.

[0158] "Means for evaluating the state of a room" refers to a method for quantifying and evaluating the degree of tidiness of a room and the arrangement of objects based on the results of image analysis.

[0159] "Means for generating and providing advice" refers to a method for creating specific improvement methods and suggestions based on the evaluation results and notifying the user.

[0160] "User interface" refers to the screens and operating means that allow users to interact with the system, specifically those that assist with labeling items and inputting information.

[0161] "Label information" refers to metadata such as category and frequency of use associated with a particular item.

[0162] "Tracking means" refers to a method for tracking and recording the usage and movement of a particular object or item.

[0163] "Decluttering advice" refers to specific suggestions and methods for efficiently disposing of unnecessary items.

[0164] "Waste disposal company" refers to a business that provides services to properly dispose of unwanted items.

[0165] "Reuse applications" refer to platforms and services that allow users to reuse or sell unwanted items to other users.

[0166] "Product display tidiness" refers to the state in which products in a store are properly arranged and organized, expressed based on numerical values ​​and evaluation criteria.

[0167] "Sales promotion and return proposals" refers to proposing efficient sales promotion methods and return methods for products that are used infrequently.

[0168] The present invention relates to a system for improving the efficiency of inventory management and product display in retail stores and brick-and-mortar stores, and for providing advice based on frequency of use. Next, a specific embodiment of this system will be described in detail.

[0169] System Configuration

[0170] First, a user takes a photo of the product display in the store using a device such as a smartphone. The captured image data is sent from the device via a network to a cloud server. The cloud server analyzes the received image data and identifies the objects.

[0171] Specifically, the cloud server uses an image analysis algorithm (for example, using an AI model such as TensorFlow) to automatically identify products and display shelves, and determine their location and size. Based on the results of this analysis, the cloud server quantifies and evaluates the store's tidiness.

[0172] Based on the evaluation results, the cloud server generates specific advice for improving product display, such as adjusting shelf height or placing specific products in the front. The advice is generated in text and illustration format and sent to the user's device.

[0173] The user checks the advice on their device and labels each product with information such as category and frequency of use. The label information is then sent to the cloud server, which then tracks the frequency of use of the product. After a certain period of time, the user takes another photo of the product display and sends it to the cloud server.

[0174] The cloud server compares the old and new image data to identify infrequently used or unsold products. Based on these results, the cloud server generates specific processing methods, such as proposing sales promotion campaigns or recommending product returns. For example, these suggestions include "put infrequently used products on sale" and "return unsold products."

[0175] Furthermore, the cloud server generates link information with appropriate waste disposal companies and recycling applications for disposing of unwanted products based on product information, allowing users to easily find ways to dispose of or reuse unwanted products.

[0176] Specific examples

[0177] For example, if a user takes a photo of the inside of a store and sends it to a cloud server, the server analyzes the image to identify the type of product and its display status. Based on the analysis results, advice such as "place a specific product in the front" is generated and sent to the user's device. The user then rearranges the products according to the advice and sends the results back to the server. The server compares the old and new image data and suggests how to handle less frequently used products.

[0178] Prompt Sentence Examples

[0179] "Analyze in-store photos to assess the organization of product displays and generate recommendations on efficient organization and identifying underused items."

[0180] The system configuration and specific embodiments of the present invention have been described above.

[0181] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0182] Step 1:

[0183] A user takes a photo of the product display in a store using a smartphone, thereby capturing visual information about the store as image data, which is used as input for the next processing step.

[0184] Step 2:

[0185] The device sends the captured image data to a cloud server. In this sending process, the image data is uploaded to the cloud server via a network. The input is the image data, and the output is the image data received by the cloud server.

[0186] Step 3:

[0187] The cloud server analyzes the received image data and identifies the objects. Specifically, it uses a generative AI model (such as TensorFlow) to analyze the image data and identify the type, location, and size of each product. The input is image data, and the output is object identification information.

[0188] Step 4:

[0189] The cloud server evaluates the store's tidiness based on the object identification information. The evaluation is based on criteria such as the number and arrangement of products, and how space is used. The input is object identification information, and the output is the evaluation result of the tidiness.

[0190] Step 5:

[0191] The cloud server generates advice on improving product displays based on the results of the organization evaluation. The advice includes specific suggestions such as "adjust the height of shelves" or "place specific products at the front." The input is the organization evaluation result, and the output is advice text and illustration information.

[0192] Step 6:

[0193] The device receives the advice sent from the cloud server and provides it to the user through a user interface. At this stage, the user can check the advice text. The input is the advice information, and the output is the interface display used by the user.

[0194] Step 7:

[0195] The user follows the advice and labels the product. The label contains information such as the product category and frequency of use. The user retransmits this information to the cloud server via their device. The input is the label information, and the output is the data to be sent to the cloud server.

[0196] Step 8:

[0197] The cloud server tracks product usage frequency based on label information. This tracking is done to record and analyze product usage over time and identify less frequently used products. Label information is the input, and usage frequency data is the output.

[0198] Step 9:

[0199] The user periodically takes photos of the product displays in the store and sends them to the cloud server via their device. The cloud server compares the old and new image data to identify infrequently used products and unsold products. The old and new image data are input, and the output is the identification of infrequently used products.

[0200] Step 10:

[0201] The cloud server generates promotional and return suggestions for the identified infrequently used products. Suggestions include "put infrequently used products on sale" and "return unsold products." The input is the identification of infrequently used products, and the output is the promotional and return suggestions.

[0202] Step 11:

[0203] The cloud server generates linkage information with appropriate waste disposal companies and reuse applications for disposing of unwanted products. This allows users to easily obtain a means for disposing of or reusing unwanted products. Information about unwanted products is input, and linkage information is output.

[0204] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0205] This invention combines a system that uses AI to efficiently support organizing and storage, and decluttering, with an emotion engine that recognizes the user's emotions. Below, we will generate a program for this system and explain the program's processing in natural language.

[0206] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[0207] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[0208] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. When the user inputs facial expressions and voice data using the device's camera and microphone, the device analyzes this and sends it to a cloud server as emotion data. The emotion engine uses this data to identify the user's emotions. For example, it can recognize when the user is tired, stressed, or relaxed.

[0209] The cloud server has the ability to adjust the advice it provides based on the recognized emotional data. For example, if the user is feeling stressed, it can generate advice that reduces the user's burden by breaking it down into simpler steps or adding words of encouragement. On the other hand, if the user is relaxed, it can provide more detailed advice.

[0210] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[0211] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[0212] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[0213] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[0214] When the user confirms this advice and uses the device's camera to capture their facial expression, the emotion engine analyzes it and, if it determines that the user is slightly tired, sends this information to the cloud server. Based on this emotion data, the cloud server generates gentle advice for the user, such as, "Start with one step today. If you try again tomorrow, you'll feel even better."

[0215] This system allows users to organize their rooms and promote decluttering while minimizing the emotional burden. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0216] The processing flow will be explained below.

[0217] Step 1:

[0218] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[0219] Step 2:

[0220] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[0221] Step 3:

[0222] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[0223] Step 4:

[0224] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[0225] Step 5:

[0226] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[0227] Step 6:

[0228] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[0229] Step 7:

[0230] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[0231] Step 8:

[0232] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[0233] Step 9:

[0234] The device sends the labeled information to the server, where it is stored in an item database.

[0235] Step 10:

[0236] The user inputs their facial expressions and voice using the device's interface, and the device's camera and microphone are used to collect data, which is then analyzed by the emotion engine.

[0237] Step 11:

[0238] The device sends emotional data analyzed by the emotion engine to a cloud server, which determines, for example, whether the user is feeling stressed or relaxed.

[0239] Step 12:

[0240] The server adjusts the advice content based on the emotional data it receives. For example, if the user is feeling stressed, it generates advice that includes simple tasks and encouraging words.

[0241] Step 13:

[0242] The server sends the adjusted advice to the device, and the advice optimized for the user is displayed within the app.

[0243] Step 14:

[0244] The user periodically takes new photos of the room and sends them to the cloud server from the device.

[0245] Step 15:

[0246] The server receives the new image data and performs a comparative analysis with the previous image data, again identifying changes in object location and category and updating the usage frequency data.

[0247] Step 16:

[0248] The server identifies items that are used less frequently based on usage frequency data, generates effective decluttering advice, and sends it to the device.

[0249] Step 17:

[0250] The device displays decluttering advice to the user and, if necessary, provides information on using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[0251] This process allows users to efficiently organize their rooms and declutter while minimizing the emotional burden. The cloud server automatically provides advice and collaborative information, reducing the burden on users.

[0252] Example 2

[0253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] In modern society, efficient organization and storage, as well as decluttering, are challenges faced by many people. These tasks require time and effort, especially for households with many possessions and businesspeople with limited time. Furthermore, because a user's motivation for organizing varies greatly depending on their emotional state, simply providing instructions on the task does not produce sufficient results. Furthermore, there is a lack of specific advice on the appropriate timing for decluttering, or on how to reuse and dispose of items. To address these challenges, an integrated system is needed that provides flexible advice based on the user's emotions and supports decluttering by tracking the frequency of use of items.

[0255] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating advice regarding the room condition and providing it to the user, means for acquiring user emotion data, means for transmitting the acquired emotion data to the cloud server, means for analyzing the emotion data and adjusting the advice in the cloud server, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, and means for providing information for linking with waste disposal companies and applications. This not only enables users to efficiently tidy up their rooms but also enables them to easily implement appropriate decluttering and reuse methods while reducing emotional burden.

[0256] "Image data" is data representing visual information acquired using a photographing device such as a digital camera or smartphone.

[0257] A "cloud server" refers to a distributed server system that can store and process data over the Internet.

[0258] "Object identification" is the process of identifying various items contained in image data and recognizing their categories and attributes.

[0259] "Emotion data" is information that indicates the emotional state of the user, analyzed from facial expressions, voice, and the like.

[0260] "Advice generation" is the process of creating advice on actions and methods suitable for the user based on the analysis results and evaluation data.

[0261] "User interface" refers to the screens and operating means that allow users to directly interact with the system.

[0262] "Label information" is information that includes data on the category and frequency of use assigned to each item.

[0263] "Usage frequency tracking" is the process of recording and monitoring how often each item is used.

[0264] "Danshari Advice" is a process that provides advice on organizing and disposing of unnecessary items based on data such as frequency of use.

[0265] A "waste disposal company" refers to a company or organization that specializes in disposing of unwanted materials.

[0266] "Application" means software designed to provide a specific function or service.

[0267] This invention is a system that supports efficient organizing and storage and decluttering. This system aims to reduce the physical and emotional burden on the user by utilizing AI to recognize the user's emotions and provide advice based on those emotions. The invention includes the following processing steps.

[0268] Users take photos of their rooms using devices such as smartphones. The captured photo data is then sent from the device to a cloud server. This transmission process uses a common communication protocol and applies encryption technology to ensure data security.

[0269] The cloud server analyzes the received photo data. This analysis is performed using a programming language such as Python and image analysis algorithms such as OpenCV and TensorFlow. First, the cloud server identifies objects in the photo (e.g., sofa, table, magazine, etc.). This allows the location and category of the object to be determined.

[0270] The cloud server then evaluates the room's tidiness based on the identified objects. This evaluation takes into account the number and location of objects, the utilization rate of storage space, and other factors. The evaluation results are quantified and specific advice (e.g., "Put magazines in a storage box" or "Put things away on the table") is generated. This advice is presented in text or illustration format and sent to the device.

[0271] Additionally, users can input their facial expressions and voice data using the device's camera and microphone. The input emotional data is analyzed on the device and sent to a cloud server. This analysis is performed using emotion analysis software (e.g., Affectiva). The cloud server uses this data to identify the user's emotional state and tailor the advice provided. For example, if the user is feeling stressed, the advice can be broken down into simple steps or accompanied by encouraging words to reduce the user's burden. On the other hand, if the user is feeling relaxed, the cloud server can provide more detailed advice.

[0272] Users can review the advice provided through their device and label each item. This label includes information such as the item's category and how often it is used. The label information is sent to a cloud server, which tracks how often it is used. The cloud server compares the old and new image data to identify items that are used less frequently. Based on this, the system suggests efficient ways to declutter (for example, recycling or selling items on a flea market app).

[0273] The cloud server also categorizes items and generates link information with appropriate waste disposal companies and flea market apps, allowing users to easily dispose of or reuse unwanted items.

[0274] As a concrete example, consider the case where a user takes a photo of their living room and sends it to a cloud server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device.

[0275] An example prompt is:

[0276] "Analyze a photo of a living room and identify the sofa, table, and scattered magazines in the room. Then, rate the room's tidiness and generate specific advice. Also, if the emotion engine determines that the user is a little tired, use that information to provide gentle advice."

[0277] In this way, the system supports users in efficiently organizing their rooms and promoting decluttering. It also provides flexible advice based on the user's emotional state, reducing the emotional burden on the user.

[0278] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0279] Step 1:

[0280] A user takes a photo of a room using a device such as a smartphone. When the user taps the "Take Photo" button, the device's camera is activated and a photo of the room is captured. The input is image data acquired through the device camera, and the output is saved as the captured photo.

[0281] Step 2:

[0282] The device sends the captured photo data to the cloud server. When the user taps the "Send" button, the device uploads the photo data to the cloud server via the Internet. This process uses a communication protocol (e.g., HTTPS). The input is the saved photo data, and the output is the image data sent to the cloud server.

[0283] Step 3:

[0284] The cloud server analyzes the received image data using an image analysis algorithm on the server (e.g., OpenCV, TensorFlow). This algorithm identifies objects in the photo and determines their location and category. The input is the image data sent to the cloud server, and the output is the category information and location information of the identified objects.

[0285] Step 4:

[0286] The cloud server evaluates the room's tidiness based on the analysis results. For example, it calculates the number of objects, their locations, and the utilization rate of storage space. A numerical score is assigned to the evaluation, and the room's condition is evaluated based on the results. The input is the identified object information, and the output is a numerical evaluation of the tidiness.

[0287] Step 5:

[0288] The cloud server generates advice about the state of the room based on the evaluation results. The advice includes specific instructions such as "put the magazines in a storage box" or "put away the items on the table." This advice is generated in the form of text or illustrations and sent to the device. The input is the evaluation result of the degree of tidiness, and the output is the generated advice.

[0289] Step 6:

[0290] The user inputs facial and voice data using the device's camera and microphone. When the user taps the "Emotion Analysis" button, the device collects the user's emotional data. The input is the user's facial and voice data, and the output is composed of emotional data.

[0291] Step 7:

[0292] The device sends the collected emotion data to the cloud server. When the user taps the "Send Emotion Data" button, the device uploads the emotion data to the cloud server. The input is the composed emotion data, and the output is the emotion data sent to the cloud server.

[0293] Step 8:

[0294] The cloud server analyzes the emotional data and identifies the user's emotional state. It uses an emotion analysis engine (e.g., Affectiva) to determine whether the user is stressed or relaxed. The input is the emotional data sent to the cloud server, and the output is the analyzed emotional state.

[0295] Step 9:

[0296] The cloud server adjusts advice according to the analyzed emotional state. For example, if the user is feeling stressed, it will break down the advice into simple steps. The input is the analyzed emotional state, and the output is the adjusted advice.

[0297] Step 10:

[0298] The user checks the advice and labels each item. Through the user interface, the user inputs label information (such as category and frequency of use). The input is the label information entered by the user, and the output is the labeled item information.

[0299] Step 11:

[0300] The device sends the label information to the cloud server. When the user taps the "Send Label Information" button, the device uploads the label information to the cloud server. The input is the labeled item information, and the output is the label information sent to the cloud server.

[0301] Step 12:

[0302] The cloud server tracks the frequency of use of each item based on the label information sent to it. It analyzes the usage frequency data and records information such as "used once a month" in a database. The input is the label information sent to the cloud server, and the output is the recorded usage frequency data.

[0303] Step 13:

[0304] The cloud server periodically compares old and new image data received and identifies items that are used less frequently. Based on this, it proposes efficient ways to declutter. For example, it generates advice such as "recyclable" or "sellable on a flea market app." The input is the tracked usage frequency data and old and new image data, and the output is advice on decluttering.

[0305] Step 14:

[0306] The cloud server classifies the items and generates linkage information for appropriate waste disposal companies and flea market apps. For example, it generates specific information such as "recycle this item" or "sell this on a specific flea market app." The input is data on infrequently used items, and the output is the linkage information provided.

[0307] (Application example 2)

[0308] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] Previous systems that support organizing and decluttering provided uniform advice without considering the user's emotional state. As a result, they lacked appropriate support even when users were feeling tired or stressed, hindering efficient organizing. Furthermore, there was no comprehensive approach for specific inventory management or display optimization, making it impossible to provide advice tailored to individual situations.

[0310] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating and providing advice regarding the room condition to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, means for providing information for linking with waste disposal companies and electronic trading platforms, means for recognizing and analyzing the user's emotions, and means for adjusting and providing advice based on the emotion data. This enables flexible and efficient support for organizing and decluttering that takes the user's emotional state into consideration.

[0311] A "means for acquiring image data" is a device for capturing images of the physical environment, such as by using a camera on a device.

[0312] The "means for transmitting the acquired image data to the cloud server" is a mechanism for uploading the captured image data to a server in a remote location via the Internet.

[0313] The "means for analyzing image data and identifying objects on a cloud server" is a technology for recognizing and classifying objects in an image using an image analysis algorithm on a cloud server.

[0314] "Means for evaluating the state of a room based on analysis results" refers to a method for evaluating the tidiness of a specified physical space based on the results of image analysis and expressing it as a numerical value or status.

[0315] The "means for generating and providing advice to the user regarding the condition of the room" refers to a system that, based on the evaluation results, creates specific instructions and suggestions for tidying up that the user should carry out and notifies them to the user.

[0316] The "means for providing a user interface for labeling items" is an interface that allows a user to digitally or physically assign information such as a name or category to an item.

[0317] The "means for transmitting label information to a cloud server" refers to a method for uploading label information assigned by a user to a cloud server and storing or analyzing the data.

[0318] The "means for tracking the frequency of use of items on a cloud server" is a system that monitors and records the frequency of use of each item based on label information and other data stored on a cloud server.

[0319] The "means for generating decluttering advice for infrequently used items" is a method for identifying infrequently used items and generating instructions or suggestions for efficiently disposing of them.

[0320] The "means for providing information for linking with waste disposal companies and electronic trading platforms" is a system that provides users with link information with appropriate companies and platforms for disposing of or reusing items.

[0321] "Means for recognizing and analyzing user emotions" refers to technology that uses a camera or microphone to obtain emotional data from the user's facial expressions and voice, and then analyzes this data.

[0322] "Means for adjusting and providing advice based on emotional data" refers to a method for flexibly adjusting the content and format of the advice provided according to the recognized emotional state of the user and providing feedback to the user.

[0323] This invention uses a system that combines AI technology and an emotion engine to support inventory management and display optimization in physical stores. The system sends image data taken by store staff using smart glasses to a cloud server, and provides efficient advice through image and emotion analysis.

[0324] 1. System Program

[0325] The server includes the following means:

[0326] Means for acquiring image data

[0327] A means of sending image data to a cloud server

[0328] A means for analyzing image data and identifying objects on a cloud server

[0329] A means of evaluating the status of the store based on analysis results

[0330] A means of generating and providing state advice to the user

[0331] Means for providing a user interface for labeling items

[0332] A means of sending label information to the cloud server

[0333] A method for tracking item usage frequency on a cloud server

[0334] A means of generating decluttering advice for less frequently used items

[0335] A means of providing information for working with waste disposal companies and electronic trading platforms

[0336] A means of recognizing and analyzing user emotions

[0337] A means to tailor and provide advice based on sentiment data

[0338] 2. A natural language description of the program's operation

[0339] The system uses the following hardware and software:

[0340] Hardware: Smart glasses (Google® Glass®, Vuzix, etc.), camera, microphone

[0341] Software: AWS (registered trademark) (or Azure (registered trademark)) cloud services, OpenCV (image analysis), NVIDIA Emotion AI (emotion analysis)

[0342] First, a user (store staff member) uses smart glasses to take a photo of inventory shelves or display shelves. This image data is sent from the smart glasses to a cloud server. The cloud server uses image analysis software such as OpenCV to identify objects in the image and evaluate the shelf status. Based on the evaluation results, the cloud server generates specific advice regarding display and inventory and provides it to the user via the smart glasses.

[0343] The smart glasses' cameras and microphones are then used to capture the user's facial expressions and voice data. This data is then used with emotion analysis software such as NVIDIA Emotion AI to recognize the user's emotional state. The cloud server then adjusts the content of the advice provided to the user based on the recognized emotional data.

[0344] For example, if the user is tired, the system suggests simple tidying tasks, while if the user is relaxed, it provides more detailed display optimization advice. The user can follow the advice provided and use the system to receive new advice whenever the situation changes.

[0345] Using this system, store staff can manage inventory and optimize display efficiently and with minimal emotional burden.

[0346] 3. Examples of concrete examples and prompts

[0347] Example 1: During the night shift, staff use the system to organize stock shelves. Tired staff are given the advice, "Just move the items on the easy list today and be done."

[0348] Example 2: This system is used to review the overall store display before the peak season. Staff who have spare time are given detailed advice such as "Review the current display and make space to add new products."

[0349] Example prompt sentence:

[0350] input:

[0351] Take a photo of your stocked shelves today and ask for advice. Also, assess the emotional state of your staff.

[0352] output:

[0353] We've analyzed your current inventory shelves. You seem a little tired, so let's start today with a quick cleanup. First, check the leftmost column. Then, check the rightmost column and combine any duplicate items onto a single shelf. Good luck!

[0354] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0355] Step 1:

[0356] A user wears smart glasses and takes a photo of an inventory or display shelf. The camera in the smart glasses captures the image data, which is then stored on the device.

[0357] Input: A photo of a shelf in a store

[0358] Output: Image data

[0359] Step 2:

[0360] The acquired image data is transmitted from the smart glasses to a cloud server, and the image data is uploaded to the cloud server via the Internet using the network connection function of the smart glasses.

[0361] Input: Image data

[0362] Output: Image data stored on a cloud server

[0363] Step 3:

[0364] The cloud server analyzes the image data using an image analysis algorithm (e.g., using OpenCV) to identify objects in the image, and then determines the object's category and location.

[0365] Input: Image data stored on a cloud server

[0366] Output: Object category and location

[0367] Step 4:

[0368] The cloud server evaluates the store's condition based on the analysis results, and the condition is quantified based on indicators such as tidiness and inventory status.

[0369] Input: Object category and location

[0370] Output: Evaluation result (number and status)

[0371] Step 5:

[0372] The cloud server generates specific advice based on the evaluation results, including specific instructions and suggestions for tidying up the user.

[0373] Input: Evaluation result

[0374] Output: Text data of advice

[0375] Step 6:

[0376] The generated advice is sent to the smart glasses, and the user can view the advice on the display of the smart glasses.

[0377] Input: Text data of advice

[0378] Output: Advice displayed on the smart glasses display

[0379] Step 7:

[0380] When a user inputs facial expressions and voice data through the smart glasses, the data is sent to a cloud server, where the user's emotional data is acquired using the smart glasses' camera and microphone.

[0381] Input: User facial and voice data

[0382] Output: Emotion data sent to the cloud server

[0383] Step 8:

[0384] The cloud server analyzes the emotional data using an emotion analysis algorithm (e.g., using NVIDIA Emotion AI) to identify the user's emotional state.

[0385] Input: Emotion data

[0386] Output: User's emotional state (fatigue level, stress level, etc.)

[0387] Step 9:

[0388] The cloud server tailors the advice based on the user's emotional state: if the user is tired, it suggests simple tasks, and if the user is relaxed, it offers detailed organization tips.

[0389] Input: User's emotional state

[0390] Output: Text data of the adjusted advice

[0391] Step 10:

[0392] The adjusted advice is then sent back to the smart glasses, where the user can review the new advice, allowing them to efficiently manage inventory and optimize display.

[0393] Input: Text data of the adjusted advice

[0394] Output: New advice displayed on the smart glasses display

[0395] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0396] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0397] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0398] [Second embodiment]

[0399] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0400] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0401] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0402] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0403] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0404] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0405] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0406] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0407] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0408] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0409] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0410] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0411] This invention relates to a system that utilizes AI to support efficient organizing and storage, and decluttering. Below, we will generate a program for this system and explain the program's processing in natural language.

[0412] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[0413] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[0414] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[0415] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[0416] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[0417] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[0418] Users use the app to label magazines and other items, then take a photo of their room and send it to the cloud server, which compares the old and new images to analyze how often items are used. For items that are used less frequently, the app offers recommendations for recycling or selling them.

[0419] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0420] The processing flow will be explained below.

[0421] Step 1:

[0422] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[0423] Step 2:

[0424] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[0425] Step 3:

[0426] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[0427] Step 4:

[0428] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[0429] Step 5:

[0430] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[0431] Step 6:

[0432] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[0433] Step 7:

[0434] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[0435] Step 8:

[0436] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[0437] Step 9:

[0438] The device sends the labeled information to the server, where it is stored in an item database.

[0439] Step 10:

[0440] The server periodically sends reminders to the user to take photos of the room again, and the user who receives the reminder takes photos of the room again.

[0441] Step 11:

[0442] The user takes another photo of the room and the device sends this new image data to the server.

[0443] Step 12:

[0444] The server receives the new image data, analyzes it, and compares it with the previous image data. The image analysis engine again identifies the object's location and category, and updates the usage frequency data.

[0445] Step 13:

[0446] The server identifies items that are used less frequently based on the updated usage frequency data, and uses this identified data to generate effective decluttering advice.

[0447] Step 14:

[0448] The server then sends the generated decluttering advice to the device, including advice on recycling, referrals to waste disposal companies, and selling items on a flea market app.

[0449] Step 15:

[0450] The device displays decluttering advice to the user and, if necessary, provides link information for using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[0451] This process allows the user to efficiently organize and tidy up their room.

[0452] Example 1

[0453] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0454] In the past, organizing and decluttering a room required a lot of effort, and it was difficult to get specific advice on how to organize and properly dispose of items.It was also difficult to understand how often items were used and how to declutter efficiently.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0456] In this invention, the server includes means for analyzing image data and identifying objects, means for providing a user interface for tagging the identified objects, and means for evaluating the degree of tidiness of a room based on the analysis results, thereby making it possible to automatically evaluate the tidiness of a room and provide the user with specific advice on tidiness and decluttering.

[0457] "Image data" refers to digital data of still images or moving images recorded as visual information.

[0458] A "cloud server" is a remote server that processes data and provides storage over a network.

[0459] "Means for identifying objects" refers to technology that analyzes image data to recognize and classify the various objects contained therein.

[0460] "User interface" refers to the operation screen and input device that allow the user to directly interact with the system.

[0461] A "means for assessing the degree of tidiness" is a technique that uses an algorithm or method for assessing the state of a room to express the level of tidiness as a number or evaluation comment.

[0462] "Label information" is a tag or identifier used to indicate additional information such as the category or frequency of use of an object.

[0463] "Means for tracking frequency of use" refers to technology that records and analyzes the usage of each object to determine frequency.

[0464] The "means for generating decluttering advice" is a technology that generates specific suggestions that instruct how to dispose of infrequently used objects.

[0465] "Information on collaboration with waste disposal companies and marketplaces" refers to data on companies and trading platforms necessary for properly disposing of or selling unwanted items.

[0466] This invention relates to a system that uses AI to efficiently support organizing and storing things. Below, we will create a program for this system and explain how it works.

[0467] This system includes hardware and software such as a user's device (e.g., a smartphone), a cloud server, and an image analysis algorithm (specifically, using machine learning frameworks such as TensorFlow and PyTorch). The specific process is as follows:

[0468] First, the user takes a photo of the room using their smartphone. This is done using a dedicated application. The captured photo data is then sent from the device to a cloud server. The sending method is, for example, the HTTPS protocol. If an error occurs during transmission, the device has the ability to attempt to resend the data.

[0469] The cloud server decompresses the received image data and begins image analysis. This analysis uses machine learning models such as TensorFlow to execute object detection algorithms. The server identifies each object in the image and assigns it a category (e.g., sofa, table, book, etc.). It also obtains each object's location information (coordinate data).

[0470] The cloud server then evaluates the room's tidiness based on the identified object information. This evaluation analyzes the number of recognized objects and their category arrangement, calculating, for example, the number of scattered items and the utilization rate of storage space. The server then quantifies these evaluation results and generates a rating such as "poorly organized" or "somewhat organized."

[0471] Based on the evaluation results, the cloud server generates specific advice on tidying up the room, such as "put magazines in a storage box" or "put away items on the table." The advice is sent to the device in text or illustration format.

[0472] The user checks the advice provided through the device and attaches tags or labels to each item. These labels include information such as the item's category and frequency of use. The device then sends the label information entered by the user to the cloud server.

[0473] The cloud server stores the received label information in a database and runs an analysis algorithm to track the frequency of use of each item. Periodically, the user takes new photos of the room and sends them to the cloud server from their device. The server compares the old and new image data to identify items that have been used less frequently.

[0474] Finally, the cloud server suggests efficient ways to declutter infrequently used items. For example, it recommends recycling methods for recyclable items and selling items that are still usable on a marketplace. Based on the item classification results, the server generates link information for appropriate waste disposal companies and marketplaces and sends this information to the device.

[0475] As a concrete example, consider the case where a user takes a photo of their living room and sends it to the server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device. The user then uses the app to label the magazines and other items, takes another photo of the room, and sends it to the server. The cloud server compares the old and new images and analyzes how often the items are used. For items that are used less frequently, it provides recommendations for recycling or selling them.

[0476] An example of a prompt is, "Take a photo of your living room and send it to the server through the app. You will soon see a tidy-up rating and specific advice."

[0477] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0478] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0479] Step 1:

[0480] The user takes a photo of the room using their smartphone. They select a room that contains a wide range of visual information, such as a living room. After taking the photo, they press the "Upload" button to proceed to the next step.

[0481] Input: Photo of the room (image data)

[0482] Output: Image data saved on the device

[0483] Step 2:

[0484] The device will send the captured photo data to the cloud server. The photo data will be compressed first and then securely transmitted using the HTTPS protocol. The retry function will be enabled until the transmission is complete.

[0485] Input: Image data stored on the device

[0486] Output: Image data sent to the cloud server

[0487] Step 3:

[0488] The server decompresses the received image data and begins analyzing it using the TensorFlow model. The image analysis algorithm performs object detection and identifies objects in the image.

[0489] Input: Image data sent to the cloud server

[0490] Output: Identified object category and location information (coordinate data)

[0491] Step 4:

[0492] The server evaluates the room's tidiness based on the identified object information, using an analytical algorithm to analyze the number of objects and their category arrangement, and calculates the "number of cluttered items" and "storage space utilization rate."

[0493] Input: Identified object category and location information

[0494] Output: Organization score and evaluation results

[0495] Step 5:

[0496] Based on the evaluation results, the server generates specific advice on tidying up the room. For example, the advice includes instructions such as "put magazines in a storage box" and "put away items on the table." The advice is generated in the form of text and illustrations.

[0497] Input: Organization score and evaluation results

[0498] Output: Advice text and illustration data

[0499] Step 6:

[0500] The server sends the generated advice to the terminal, where the user can see specific instructions on how to tidy up their room.

[0501] Input: Advice text and illustration data

[0502] Output: Advice sent to terminal

[0503] Step 7:

[0504] The user tags or labels each item based on the advice provided. The label contains information such as the object's category and frequency of use. When the user presses the "Done tagging" button, the label information is sent to the server.

[0505] Input: Label information added based on the advice

[0506] Output: Label information sent to the cloud server

[0507] Step 8:

[0508] The server stores the received label information in a database and starts tracking usage frequency. It runs an analysis algorithm to accumulate data and record the frequency of use of each item.

[0509] Input: Label information sent to the cloud server

[0510] Output: Usage frequency information stored in a database

[0511] Step 9:

[0512] Periodically, the user takes another photo of the room and presses the "Reevaluate" button to send the image data from the device to the cloud server, where it is analyzed.

[0513] Input: Newly taken photo of the room (image data)

[0514] Output: New image data sent to the cloud server

[0515] Step 10:

[0516] The server compares the old and new image data to identify less frequently used items, and generates advice on how to efficiently declutter.

[0517] Input: Old and new image data

[0518] Output: A list of items that are rarely used and advice on decluttering

[0519] Step 11:

[0520] The server generates linked information about waste disposal companies and marketplaces based on the item classification results, along with decluttering advice, and sends this to the terminal.

[0521] Input: Decluttering advice and item classification results

[0522] Output: Decluttering advice and collaboration information sent to the device

[0523] (Application example 1)

[0524] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0525] Until now, support systems have been provided to help individuals efficiently organize their homes, but there are limitations to these systems in terms of improving the efficiency of inventory management and product display in retail stores and brick-and-mortar shops, as well as providing appropriate advice based on product usage frequency and sales data. In particular, there is a need for appropriate disposal of infrequently used and unsold products. It is also important to propose efficient product display methods and improve display methods. However, current systems are unable to adequately address these issues.

[0526] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0527] In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the condition of a room based on the analysis results, means for generating and providing advice regarding the condition of the room to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of use of items in the cloud server, means for generating decluttering advice regarding infrequently used items, means for providing information for linking with waste disposal companies and recycling applications, means for evaluating the tidiness of product displays and providing advice regarding improving the display method, and means for identifying infrequently used products and suggesting sales promotions or returns. This enables more efficient inventory management and product display in retail stores and brick-and-mortar stores, and the provision of appropriate advice based on the frequency of product use.

[0528] "Image data" refers to a collection of electronically acquired visual information that can be analyzed to identify an object.

[0529] A "cloud server" is a server that stores and processes data via the Internet, and is responsible for processing information in cooperation with user devices.

[0530] "Means for identifying objects" refers to the algorithms or software used to identify specific objects from image data.

[0531] "Means for evaluating the state of a room" refers to a method for quantifying and evaluating the degree of tidiness of a room and the arrangement of objects based on the results of image analysis.

[0532] "Means for generating and providing advice" refers to a method for creating specific improvement methods and suggestions based on the evaluation results and notifying the user.

[0533] "User interface" refers to the screens and operating means that allow users to interact with the system, specifically those that assist with labeling items and inputting information.

[0534] "Label information" refers to metadata such as category and frequency of use associated with a particular item.

[0535] "Tracking means" refers to a method for tracking and recording the usage and movement of a particular object or item.

[0536] "Decluttering advice" refers to specific suggestions and methods for efficiently disposing of unnecessary items.

[0537] "Waste disposal company" refers to a business that provides services to properly dispose of unwanted items.

[0538] "Reuse applications" refer to platforms and services that allow users to reuse or sell unwanted items to other users.

[0539] "Product display tidiness" refers to the state in which products in a store are properly arranged and organized, expressed based on numerical values ​​and evaluation criteria.

[0540] "Sales promotion and return proposals" refers to proposing efficient sales promotion methods and return methods for products that are used infrequently.

[0541] The present invention relates to a system for improving the efficiency of inventory management and product display in retail stores and brick-and-mortar stores, and for providing advice based on frequency of use. Next, a specific embodiment of this system will be described in detail.

[0542] System Configuration

[0543] First, a user takes a photo of the product display in the store using a device such as a smartphone. The captured image data is sent from the device via a network to a cloud server. The cloud server analyzes the received image data and identifies the objects.

[0544] Specifically, the cloud server uses an image analysis algorithm (for example, using an AI model such as TensorFlow) to automatically identify products and display shelves, and determine their location and size. Based on the results of this analysis, the cloud server quantifies and evaluates the store's tidiness.

[0545] Based on the evaluation results, the cloud server generates specific advice for improving product display, such as adjusting shelf height or placing specific products in the front. The advice is generated in text and illustration format and sent to the user's device.

[0546] The user checks the advice on their device and labels each product with information such as category and frequency of use. The label information is then sent to the cloud server, which then tracks the frequency of use of the product. After a certain period of time, the user takes another photo of the product display and sends it to the cloud server.

[0547] The cloud server compares the old and new image data to identify infrequently used or unsold products. Based on these results, the cloud server generates specific processing methods, such as proposing sales promotion campaigns or recommending product returns. For example, these suggestions include "put infrequently used products on sale" and "return unsold products."

[0548] Furthermore, the cloud server generates link information with appropriate waste disposal companies and recycling applications for disposing of unwanted products based on product information, allowing users to easily find ways to dispose of or reuse unwanted products.

[0549] Specific examples

[0550] For example, if a user takes a photo of the inside of a store and sends it to a cloud server, the server analyzes the image to identify the type of product and its display status. Based on the analysis results, advice such as "place a specific product in the front" is generated and sent to the user's device. The user then rearranges the products according to the advice and sends the results back to the server. The server compares the old and new image data and suggests how to handle less frequently used products.

[0551] Prompt Sentence Examples

[0552] "Analyze in-store photos to assess the organization of product displays and generate recommendations on efficient organization and identifying underused items."

[0553] The system configuration and specific embodiments of the present invention have been described above.

[0554] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0555] Step 1:

[0556] A user takes a photo of the product display in a store using a smartphone, thereby capturing visual information about the store as image data, which is used as input for the next processing step.

[0557] Step 2:

[0558] The device sends the captured image data to a cloud server. In this sending process, the image data is uploaded to the cloud server via a network. The input is the image data, and the output is the image data received by the cloud server.

[0559] Step 3:

[0560] The cloud server analyzes the received image data and identifies the objects. Specifically, it uses a generative AI model (such as TensorFlow) to analyze the image data and identify the type, location, and size of each product. The input is image data, and the output is object identification information.

[0561] Step 4:

[0562] The cloud server evaluates the store's tidiness based on the object identification information. The evaluation is based on criteria such as the number and arrangement of products, and how space is used. The input is object identification information, and the output is the evaluation result of the tidiness.

[0563] Step 5:

[0564] The cloud server generates advice on improving product displays based on the results of the organization evaluation. The advice includes specific suggestions such as "adjust the height of shelves" or "place specific products at the front." The input is the organization evaluation result, and the output is advice text and illustration information.

[0565] Step 6:

[0566] The device receives the advice sent from the cloud server and provides it to the user through a user interface. At this stage, the user can check the advice text. The input is the advice information, and the output is the interface display used by the user.

[0567] Step 7:

[0568] The user follows the advice and labels the product. The label contains information such as the product category and frequency of use. The user retransmits this information to the cloud server via their device. The input is the label information, and the output is the data to be sent to the cloud server.

[0569] Step 8:

[0570] The cloud server tracks product usage frequency based on label information. This tracking is done to record and analyze product usage over time and identify less frequently used products. Label information is the input, and usage frequency data is the output.

[0571] Step 9:

[0572] The user periodically takes photos of the product displays in the store and sends them to the cloud server via their device. The cloud server compares the old and new image data to identify infrequently used products and unsold products. The old and new image data are input, and the output is the identification of infrequently used products.

[0573] Step 10:

[0574] The cloud server generates promotional and return suggestions for the identified infrequently used products. Suggestions include "put infrequently used products on sale" and "return unsold products." The input is the identification of infrequently used products, and the output is the promotional and return suggestions.

[0575] Step 11:

[0576] The cloud server generates linkage information with appropriate waste disposal companies and reuse applications for disposing of unwanted products. This allows users to easily obtain a means for disposing of or reusing unwanted products. Information about unwanted products is input, and linkage information is output.

[0577] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0578] This invention combines a system that uses AI to efficiently support organizing and storage, and decluttering, with an emotion engine that recognizes the user's emotions. Below, we will generate a program for this system and explain the program's processing in natural language.

[0579] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[0580] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[0581] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. When the user inputs facial expressions and voice data using the device's camera and microphone, the device analyzes this and sends it to a cloud server as emotion data. The emotion engine uses this data to identify the user's emotions. For example, it can recognize when the user is tired, stressed, or relaxed.

[0582] The cloud server has the ability to adjust the advice it provides based on the recognized emotional data. For example, if the user is feeling stressed, it can generate advice that reduces the user's burden by breaking it down into simpler steps or adding words of encouragement. On the other hand, if the user is relaxed, it can provide more detailed advice.

[0583] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[0584] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[0585] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[0586] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[0587] When the user confirms this advice and uses the device's camera to capture their facial expression, the emotion engine analyzes it and, if it determines that the user is slightly tired, sends this information to the cloud server. Based on this emotion data, the cloud server generates gentle advice for the user, such as, "Start with one step today. If you try again tomorrow, you'll feel even better."

[0588] This system allows users to organize their rooms and promote decluttering while minimizing the emotional burden. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0589] The processing flow will be explained below.

[0590] Step 1:

[0591] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[0592] Step 2:

[0593] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[0594] Step 3:

[0595] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[0596] Step 4:

[0597] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[0598] Step 5:

[0599] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[0600] Step 6:

[0601] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[0602] Step 7:

[0603] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[0604] Step 8:

[0605] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[0606] Step 9:

[0607] The device sends the labeled information to the server, where it is stored in an item database.

[0608] Step 10:

[0609] The user inputs their facial expressions and voice using the device's interface, and the device's camera and microphone are used to collect data, which is then analyzed by the emotion engine.

[0610] Step 11:

[0611] The device sends emotional data analyzed by the emotion engine to a cloud server, which determines, for example, whether the user is feeling stressed or relaxed.

[0612] Step 12:

[0613] The server adjusts the advice content based on the emotional data it receives. For example, if the user is feeling stressed, it generates advice that includes simple tasks and encouraging words.

[0614] Step 13:

[0615] The server sends the adjusted advice to the device, and the advice optimized for the user is displayed within the app.

[0616] Step 14:

[0617] The user periodically takes new photos of the room and sends them to the cloud server from the device.

[0618] Step 15:

[0619] The server receives the new image data and performs a comparative analysis with the previous image data, again identifying changes in object location and category and updating the usage frequency data.

[0620] Step 16:

[0621] The server identifies items that are used less frequently based on usage frequency data, generates effective decluttering advice, and sends it to the device.

[0622] Step 17:

[0623] The device displays decluttering advice to the user and, if necessary, provides information on using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[0624] This process allows users to efficiently organize their rooms and declutter while minimizing the emotional burden. The cloud server automatically provides advice and collaborative information, reducing the burden on users.

[0625] Example 2

[0626] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] In modern society, efficient organization and storage, as well as decluttering, are challenges faced by many people. These tasks require time and effort, especially for households with many possessions and businesspeople with limited time. Furthermore, because a user's motivation for organizing varies greatly depending on their emotional state, simply providing instructions on the task does not produce sufficient results. Furthermore, there is a lack of specific advice on the appropriate timing for decluttering, or on how to reuse and dispose of items. To address these challenges, an integrated system is needed that provides flexible advice based on the user's emotions and supports decluttering by tracking the frequency of use of items.

[0628] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating advice regarding the room condition and providing it to the user, means for acquiring user emotion data, means for transmitting the acquired emotion data to the cloud server, means for analyzing the emotion data and adjusting the advice in the cloud server, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, and means for providing information for linking with waste disposal companies and applications. This not only enables users to efficiently tidy up their rooms but also enables them to easily implement appropriate decluttering and reuse methods while reducing emotional burden.

[0629] "Image data" is data representing visual information acquired using a photographing device such as a digital camera or smartphone.

[0630] A "cloud server" refers to a distributed server system that can store and process data over the Internet.

[0631] "Object identification" is the process of identifying various items contained in image data and recognizing their categories and attributes.

[0632] "Emotion data" is information that indicates the emotional state of the user, analyzed from facial expressions, voice, and the like.

[0633] "Advice generation" is the process of creating advice on actions and methods suitable for the user based on the analysis results and evaluation data.

[0634] "User interface" refers to the screens and operating means that allow users to directly interact with the system.

[0635] "Label information" is information that includes data on the category and frequency of use assigned to each item.

[0636] "Usage frequency tracking" is the process of recording and monitoring how often each item is used.

[0637] "Danshari Advice" is a process that provides advice on organizing and disposing of unnecessary items based on data such as frequency of use.

[0638] A "waste disposal company" refers to a company or organization that specializes in disposing of unwanted materials.

[0639] "Application" means software designed to provide a specific function or service.

[0640] This invention is a system that supports efficient organizing and storage and decluttering. This system aims to reduce the physical and emotional burden on the user by utilizing AI to recognize the user's emotions and provide advice based on those emotions. The invention includes the following processing steps.

[0641] Users take photos of their rooms using devices such as smartphones. The captured photo data is then sent from the device to a cloud server. This transmission process uses a common communication protocol and applies encryption technology to ensure data security.

[0642] The cloud server analyzes the received photo data. This analysis is performed using a programming language such as Python and image analysis algorithms such as OpenCV and TensorFlow. First, the cloud server identifies objects in the photo (e.g., sofa, table, magazine, etc.). This allows the location and category of the object to be determined.

[0643] The cloud server then evaluates the room's tidiness based on the identified objects. This evaluation takes into account the number and location of objects, the utilization rate of storage space, and other factors. The evaluation results are quantified and specific advice (e.g., "Put magazines in a storage box" or "Put things away on the table") is generated. This advice is presented in text or illustration format and sent to the device.

[0644] Additionally, users can input their facial expressions and voice data using the device's camera and microphone. The input emotional data is analyzed on the device and sent to a cloud server. This analysis is performed using emotion analysis software (e.g., Affectiva). The cloud server uses this data to identify the user's emotional state and tailor the advice provided. For example, if the user is feeling stressed, the advice can be broken down into simple steps or accompanied by encouraging words to reduce the user's burden. On the other hand, if the user is feeling relaxed, the cloud server can provide more detailed advice.

[0645] Users can review the advice provided through their device and label each item. This label includes information such as the item's category and how often it is used. The label information is sent to a cloud server, which tracks how often it is used. The cloud server compares the old and new image data to identify items that are used less frequently. Based on this, the system suggests efficient ways to declutter (for example, recycling or selling items on a flea market app).

[0646] The cloud server also categorizes items and generates link information with appropriate waste disposal companies and flea market apps, allowing users to easily dispose of or reuse unwanted items.

[0647] As a concrete example, consider the case where a user takes a photo of their living room and sends it to a cloud server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device.

[0648] An example prompt is:

[0649] "Analyze a photo of a living room and identify the sofa, table, and scattered magazines in the room. Then, rate the room's tidiness and generate specific advice. Also, if the emotion engine determines that the user is a little tired, use that information to provide gentle advice."

[0650] In this way, the system supports users in efficiently organizing their rooms and promoting decluttering. It also provides flexible advice based on the user's emotional state, reducing the emotional burden on the user.

[0651] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0652] Step 1:

[0653] A user takes a photo of a room using a device such as a smartphone. When the user taps the "Take Photo" button, the device's camera is activated and a photo of the room is captured. The input is image data acquired through the device camera, and the output is saved as the captured photo.

[0654] Step 2:

[0655] The device sends the captured photo data to the cloud server. When the user taps the "Send" button, the device uploads the photo data to the cloud server via the Internet. This process uses a communication protocol (e.g., HTTPS). The input is the saved photo data, and the output is the image data sent to the cloud server.

[0656] Step 3:

[0657] The cloud server analyzes the received image data using an image analysis algorithm on the server (e.g., OpenCV, TensorFlow). This algorithm identifies objects in the photo and determines their location and category. The input is the image data sent to the cloud server, and the output is the category information and location information of the identified objects.

[0658] Step 4:

[0659] The cloud server evaluates the room's tidiness based on the analysis results. For example, it calculates the number of objects, their locations, and the utilization rate of storage space. A numerical score is assigned to the evaluation, and the room's condition is evaluated based on the results. The input is the identified object information, and the output is a numerical evaluation of the tidiness.

[0660] Step 5:

[0661] The cloud server generates advice about the state of the room based on the evaluation results. The advice includes specific instructions such as "put the magazines in a storage box" or "put away the items on the table." This advice is generated in the form of text or illustrations and sent to the device. The input is the evaluation result of the degree of tidiness, and the output is the generated advice.

[0662] Step 6:

[0663] The user inputs facial and voice data using the device's camera and microphone. When the user taps the "Emotion Analysis" button, the device collects the user's emotional data. The input is the user's facial and voice data, and the output is composed of emotional data.

[0664] Step 7:

[0665] The device sends the collected emotion data to the cloud server. When the user taps the "Send Emotion Data" button, the device uploads the emotion data to the cloud server. The input is the composed emotion data, and the output is the emotion data sent to the cloud server.

[0666] Step 8:

[0667] The cloud server analyzes the emotional data and identifies the user's emotional state. It uses an emotion analysis engine (e.g., Affectiva) to determine whether the user is stressed or relaxed. The input is the emotional data sent to the cloud server, and the output is the analyzed emotional state.

[0668] Step 9:

[0669] The cloud server adjusts advice according to the analyzed emotional state. For example, if the user is feeling stressed, it will break down the advice into simple steps. The input is the analyzed emotional state, and the output is the adjusted advice.

[0670] Step 10:

[0671] The user checks the advice and labels each item. Through the user interface, the user inputs label information (such as category and frequency of use). The input is the label information entered by the user, and the output is the labeled item information.

[0672] Step 11:

[0673] The device sends the label information to the cloud server. When the user taps the "Send Label Information" button, the device uploads the label information to the cloud server. The input is the labeled item information, and the output is the label information sent to the cloud server.

[0674] Step 12:

[0675] The cloud server tracks the frequency of use of each item based on the label information sent to it. It analyzes the usage frequency data and records information such as "used once a month" in a database. The input is the label information sent to the cloud server, and the output is the recorded usage frequency data.

[0676] Step 13:

[0677] The cloud server periodically compares old and new image data received and identifies items that are used less frequently. Based on this, it proposes efficient ways to declutter. For example, it generates advice such as "recyclable" or "sellable on a flea market app." The input is the tracked usage frequency data and old and new image data, and the output is advice on decluttering.

[0678] Step 14:

[0679] The cloud server classifies the items and generates linkage information for appropriate waste disposal companies and flea market apps. For example, it generates specific information such as "recycle this item" or "sell this on a specific flea market app." The input is data on infrequently used items, and the output is the linkage information provided.

[0680] (Application example 2)

[0681] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0682] Previous systems that support organizing and decluttering provided uniform advice without considering the user's emotional state. As a result, they lacked appropriate support even when users were feeling tired or stressed, hindering efficient organizing. Furthermore, there was no comprehensive approach for specific inventory management or display optimization, making it impossible to provide advice tailored to individual situations.

[0683] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating and providing advice regarding the room condition to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, means for providing information for linking with waste disposal companies and electronic trading platforms, means for recognizing and analyzing the user's emotions, and means for adjusting and providing advice based on the emotion data. This enables flexible and efficient support for organizing and decluttering that takes the user's emotional state into consideration.

[0684] A "means for acquiring image data" is a device for capturing images of the physical environment, such as by using a camera on a device.

[0685] The "means for transmitting the acquired image data to the cloud server" is a mechanism for uploading the captured image data to a server in a remote location via the Internet.

[0686] The "means for analyzing image data and identifying objects on a cloud server" is a technology for recognizing and classifying objects in an image using an image analysis algorithm on a cloud server.

[0687] "Means for evaluating the state of a room based on analysis results" refers to a method for evaluating the tidiness of a specified physical space based on the results of image analysis and expressing it as a numerical value or status.

[0688] The "means for generating and providing advice to the user regarding the condition of the room" refers to a system that, based on the evaluation results, creates specific instructions and suggestions for tidying up that the user should carry out and notifies them to the user.

[0689] The "means for providing a user interface for labeling items" is an interface that allows a user to digitally or physically assign information such as a name or category to an item.

[0690] The "means for transmitting label information to a cloud server" refers to a method for uploading label information assigned by a user to a cloud server and storing or analyzing the data.

[0691] The "means for tracking the frequency of use of items on a cloud server" is a system that monitors and records the frequency of use of each item based on label information and other data stored on a cloud server.

[0692] The "means for generating decluttering advice for infrequently used items" is a method for identifying infrequently used items and generating instructions or suggestions for efficiently disposing of them.

[0693] The "means for providing information for linking with waste disposal companies and electronic trading platforms" is a system that provides users with link information with appropriate companies and platforms for disposing of or reusing items.

[0694] "Means for recognizing and analyzing user emotions" refers to technology that uses a camera or microphone to obtain emotional data from the user's facial expressions and voice, and then analyzes this data.

[0695] "Means for adjusting and providing advice based on emotional data" refers to a method for flexibly adjusting the content and format of the advice provided according to the recognized emotional state of the user and providing feedback to the user.

[0696] This invention uses a system that combines AI technology and an emotion engine to support inventory management and display optimization in physical stores. The system sends image data taken by store staff using smart glasses to a cloud server, and provides efficient advice through image and emotion analysis.

[0697] 1. System Program

[0698] The server includes the following means:

[0699] Means for acquiring image data

[0700] A means of sending image data to a cloud server

[0701] A means for analyzing image data and identifying objects on a cloud server

[0702] A means of evaluating the status of the store based on analysis results

[0703] A means of generating and providing state advice to the user

[0704] Means for providing a user interface for labeling items

[0705] A means of sending label information to the cloud server

[0706] A method for tracking item usage frequency on a cloud server

[0707] A means of generating decluttering advice for less frequently used items

[0708] A means of providing information for working with waste disposal companies and electronic trading platforms

[0709] A means of recognizing and analyzing user emotions

[0710] A means to tailor and provide advice based on sentiment data

[0711] 2. A natural language description of the program's operation

[0712] The system uses the following hardware and software:

[0713] Hardware: Smart glasses (Google Glass, Vuzix, etc.), camera, microphone

[0714] Software: AWS (or Azure) cloud services, OpenCV (image analysis), NVIDIA Emotion AI (emotion analysis)

[0715] First, a user (store staff member) uses smart glasses to take a photo of inventory shelves or display shelves. This image data is sent from the smart glasses to a cloud server. The cloud server uses image analysis software such as OpenCV to identify objects in the image and evaluate the shelf status. Based on the evaluation results, the cloud server generates specific advice regarding display and inventory and provides it to the user via the smart glasses.

[0716] The smart glasses' cameras and microphones are then used to capture the user's facial expressions and voice data. This data is then used with emotion analysis software such as NVIDIA Emotion AI to recognize the user's emotional state. The cloud server then adjusts the content of the advice provided to the user based on the recognized emotional data.

[0717] For example, if the user is tired, the system suggests simple tidying tasks, while if the user is relaxed, it provides more detailed display optimization advice. The user can follow the advice provided and use the system to receive new advice whenever the situation changes.

[0718] Using this system, store staff can manage inventory and optimize display efficiently and with minimal emotional burden.

[0719] 3. Examples of concrete examples and prompts

[0720] Example 1: During the night shift, staff use the system to organize stock shelves. Tired staff are given the advice, "Just move the items on the easy list today and be done."

[0721] Example 2: This system is used to review the overall store display before the peak season. Staff who have spare time are given detailed advice such as "Review the current display and make space to add new products."

[0722] Example prompt sentence:

[0723] input:

[0724] Take a photo of your stocked shelves today and ask for advice. Also, assess the emotional state of your staff.

[0725] output:

[0726] We've analyzed your current inventory shelves. You seem a little tired, so let's start today with a quick cleanup. First, check the leftmost column. Then, check the rightmost column and combine any duplicate items onto a single shelf. Good luck!

[0727] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0728] Step 1:

[0729] A user wears smart glasses and takes a photo of an inventory or display shelf. The camera in the smart glasses captures the image data, which is then stored on the device.

[0730] Input: A photo of a shelf in a store

[0731] Output: Image data

[0732] Step 2:

[0733] The acquired image data is transmitted from the smart glasses to a cloud server, and the image data is uploaded to the cloud server via the Internet using the network connection function of the smart glasses.

[0734] Input: Image data

[0735] Output: Image data stored on a cloud server

[0736] Step 3:

[0737] The cloud server analyzes the image data using an image analysis algorithm (e.g., using OpenCV) to identify objects in the image, and then determines the object's category and location.

[0738] Input: Image data stored on a cloud server

[0739] Output: Object category and location

[0740] Step 4:

[0741] The cloud server evaluates the store's condition based on the analysis results, and the condition is quantified based on indicators such as tidiness and inventory status.

[0742] Input: Object category and location

[0743] Output: Evaluation result (number and status)

[0744] Step 5:

[0745] The cloud server generates specific advice based on the evaluation results, including specific instructions and suggestions for tidying up the user.

[0746] Input: Evaluation result

[0747] Output: Text data of advice

[0748] Step 6:

[0749] The generated advice is sent to the smart glasses, and the user can view the advice on the display of the smart glasses.

[0750] Input: Text data of advice

[0751] Output: Advice displayed on the smart glasses display

[0752] Step 7:

[0753] When a user inputs facial expressions and voice data through the smart glasses, the data is sent to a cloud server, where the user's emotional data is acquired using the smart glasses' camera and microphone.

[0754] Input: User facial and voice data

[0755] Output: Emotion data sent to the cloud server

[0756] Step 8:

[0757] The cloud server analyzes the emotional data using an emotion analysis algorithm (e.g., using NVIDIA Emotion AI) to identify the user's emotional state.

[0758] Input: Emotion data

[0759] Output: User's emotional state (fatigue level, stress level, etc.)

[0760] Step 9:

[0761] The cloud server tailors the advice based on the user's emotional state: if the user is tired, it suggests simple tasks, and if the user is relaxed, it offers detailed organization tips.

[0762] Input: User's emotional state

[0763] Output: Text data of the adjusted advice

[0764] Step 10:

[0765] The adjusted advice is then sent back to the smart glasses, where the user can review the new advice, allowing them to efficiently manage inventory and optimize display.

[0766] Input: Text data of the adjusted advice

[0767] Output: New advice displayed on the smart glasses display

[0768] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0769] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0770] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0771] [Third embodiment]

[0772] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0773] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0774] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0775] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0776] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0777] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0778] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0779] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0780] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0781] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0782] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0783] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0784] This invention relates to a system that utilizes AI to support efficient organizing and storage, and decluttering. Below, we will generate a program for this system and explain the program's processing in natural language.

[0785] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[0786] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[0787] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[0788] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[0789] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[0790] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[0791] Users use the app to label magazines and other items, then take a photo of their room and send it to the cloud server, which compares the old and new images to analyze how often items are used. For items that are used less frequently, the app offers recommendations for recycling or selling them.

[0792] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0793] The processing flow will be explained below.

[0794] Step 1:

[0795] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[0796] Step 2:

[0797] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[0798] Step 3:

[0799] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[0800] Step 4:

[0801] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[0802] Step 5:

[0803] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[0804] Step 6:

[0805] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[0806] Step 7:

[0807] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[0808] Step 8:

[0809] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[0810] Step 9:

[0811] The device sends the labeled information to the server, where it is stored in an item database.

[0812] Step 10:

[0813] The server periodically sends reminders to the user to take photos of the room again, and the user who receives the reminder takes photos of the room again.

[0814] Step 11:

[0815] The user takes another photo of the room and the device sends this new image data to the server.

[0816] Step 12:

[0817] The server receives the new image data, analyzes it, and compares it with the previous image data. The image analysis engine again identifies the object's location and category, and updates the usage frequency data.

[0818] Step 13:

[0819] The server identifies items that are used less frequently based on the updated usage frequency data, and uses this identified data to generate effective decluttering advice.

[0820] Step 14:

[0821] The server then sends the generated decluttering advice to the device, including advice on recycling, referrals to waste disposal companies, and selling items on a flea market app.

[0822] Step 15:

[0823] The device displays decluttering advice to the user and, if necessary, provides link information for using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[0824] This process allows the user to efficiently organize and tidy up their room.

[0825] Example 1

[0826] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0827] In the past, organizing and decluttering a room required a lot of effort, and it was difficult to get specific advice on how to organize and properly dispose of items.It was also difficult to understand how often items were used and how to declutter efficiently.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0829] In this invention, the server includes means for analyzing image data and identifying objects, means for providing a user interface for tagging the identified objects, and means for evaluating the degree of tidiness of a room based on the analysis results, thereby making it possible to automatically evaluate the tidiness of a room and provide the user with specific advice on tidiness and decluttering.

[0830] "Image data" refers to digital data of still images or moving images recorded as visual information.

[0831] A "cloud server" is a remote server that processes data and provides storage over a network.

[0832] "Means for identifying objects" refers to technology that analyzes image data to recognize and classify the various objects contained therein.

[0833] "User interface" refers to the operation screen and input device that allow the user to directly interact with the system.

[0834] A "means for assessing the degree of tidiness" is a technique that uses an algorithm or method for assessing the state of a room to express the level of tidiness as a number or evaluation comment.

[0835] "Label information" is a tag or identifier used to indicate additional information such as the category or frequency of use of an object.

[0836] "Means for tracking frequency of use" refers to technology that records and analyzes the usage of each object to determine frequency.

[0837] The "means for generating decluttering advice" is a technology that generates specific suggestions that instruct how to dispose of infrequently used objects.

[0838] "Information on collaboration with waste disposal companies and marketplaces" refers to data on companies and trading platforms necessary for properly disposing of or selling unwanted items.

[0839] This invention relates to a system that uses AI to efficiently support organizing and storing things. Below, we will create a program for this system and explain how it works.

[0840] This system includes hardware and software such as a user's device (e.g., a smartphone), a cloud server, and an image analysis algorithm (specifically, using machine learning frameworks such as TensorFlow and PyTorch). The specific process is as follows:

[0841] First, the user takes a photo of the room using their smartphone. This is done using a dedicated application. The captured photo data is then sent from the device to a cloud server. The sending method is, for example, the HTTPS protocol. If an error occurs during transmission, the device has the ability to attempt to resend the data.

[0842] The cloud server decompresses the received image data and begins image analysis. This analysis uses machine learning models such as TensorFlow to execute object detection algorithms. The server identifies each object in the image and assigns it a category (e.g., sofa, table, book, etc.). It also obtains each object's location information (coordinate data).

[0843] The cloud server then evaluates the room's tidiness based on the identified object information. This evaluation analyzes the number of recognized objects and their category arrangement, calculating, for example, the number of scattered items and the utilization rate of storage space. The server then quantifies these evaluation results and generates a rating such as "poorly organized" or "somewhat organized."

[0844] Based on the evaluation results, the cloud server generates specific advice on tidying up the room, such as "put magazines in a storage box" or "put away items on the table." The advice is sent to the device in text or illustration format.

[0845] The user checks the advice provided through the device and attaches tags or labels to each item. These labels include information such as the item's category and frequency of use. The device then sends the label information entered by the user to the cloud server.

[0846] The cloud server stores the received label information in a database and runs an analysis algorithm to track the frequency of use of each item. Periodically, the user takes new photos of the room and sends them to the cloud server from their device. The server compares the old and new image data to identify items that have been used less frequently.

[0847] Finally, the cloud server suggests efficient ways to declutter infrequently used items. For example, it recommends recycling methods for recyclable items and selling items that are still usable on a marketplace. Based on the item classification results, the server generates link information for appropriate waste disposal companies and marketplaces and sends this information to the device.

[0848] As a concrete example, consider the case where a user takes a photo of their living room and sends it to the server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device. The user then uses the app to label the magazines and other items, takes another photo of the room, and sends it to the server. The cloud server compares the old and new images and analyzes how often the items are used. For items that are used less frequently, it provides recommendations for recycling or selling them.

[0849] An example of a prompt is, "Take a photo of your living room and send it to the server through the app. You will soon see a tidy-up rating and specific advice."

[0850] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0851] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0852] Step 1:

[0853] The user takes a photo of the room using their smartphone. They select a room that contains a wide range of visual information, such as a living room. After taking the photo, they press the "Upload" button to proceed to the next step.

[0854] Input: Photo of the room (image data)

[0855] Output: Image data saved on the device

[0856] Step 2:

[0857] The device will send the captured photo data to the cloud server. The photo data will be compressed first and then securely transmitted using the HTTPS protocol. The retry function will be enabled until the transmission is complete.

[0858] Input: Image data stored on the device

[0859] Output: Image data sent to the cloud server

[0860] Step 3:

[0861] The server decompresses the received image data and begins analyzing it using the TensorFlow model. The image analysis algorithm performs object detection and identifies objects in the image.

[0862] Input: Image data sent to the cloud server

[0863] Output: Identified object category and location information (coordinate data)

[0864] Step 4:

[0865] The server evaluates the room's tidiness based on the identified object information, using an analytical algorithm to analyze the number of objects and their category arrangement, and calculates the "number of cluttered items" and "storage space utilization rate."

[0866] Input: Identified object category and location information

[0867] Output: Organization score and evaluation results

[0868] Step 5:

[0869] Based on the evaluation results, the server generates specific advice on tidying up the room. For example, the advice includes instructions such as "put magazines in a storage box" and "put away items on the table." The advice is generated in the form of text and illustrations.

[0870] Input: Organization score and evaluation results

[0871] Output: Advice text and illustration data

[0872] Step 6:

[0873] The server sends the generated advice to the terminal, where the user can see specific instructions on how to tidy up their room.

[0874] Input: Advice text and illustration data

[0875] Output: Advice sent to terminal

[0876] Step 7:

[0877] The user tags or labels each item based on the advice provided. The label contains information such as the object's category and frequency of use. When the user presses the "Done tagging" button, the label information is sent to the server.

[0878] Input: Label information added based on the advice

[0879] Output: Label information sent to the cloud server

[0880] Step 8:

[0881] The server stores the received label information in a database and starts tracking usage frequency. It runs an analysis algorithm to accumulate data and record the frequency of use of each item.

[0882] Input: Label information sent to the cloud server

[0883] Output: Usage frequency information stored in a database

[0884] Step 9:

[0885] Periodically, the user takes another photo of the room and presses the "Reevaluate" button to send the image data from the device to the cloud server, where it is analyzed.

[0886] Input: Newly taken photo of the room (image data)

[0887] Output: New image data sent to the cloud server

[0888] Step 10:

[0889] The server compares the old and new image data to identify less frequently used items, and generates advice on how to efficiently declutter.

[0890] Input: Old and new image data

[0891] Output: A list of items that are rarely used and advice on decluttering

[0892] Step 11:

[0893] The server generates linked information about waste disposal companies and marketplaces based on the item classification results, along with decluttering advice, and sends this to the terminal.

[0894] Input: Decluttering advice and item classification results

[0895] Output: Decluttering advice and collaboration information sent to the device

[0896] (Application example 1)

[0897] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0898] Until now, support systems have been provided to help individuals efficiently organize their homes, but there are limitations to these systems in terms of improving the efficiency of inventory management and product display in retail stores and brick-and-mortar shops, as well as providing appropriate advice based on product usage frequency and sales data. In particular, there is a need for appropriate disposal of infrequently used and unsold products. It is also important to propose efficient product display methods and improve display methods. However, current systems are unable to adequately address these issues.

[0899] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0900] In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the condition of a room based on the analysis results, means for generating and providing advice regarding the condition of the room to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of use of items in the cloud server, means for generating decluttering advice regarding infrequently used items, means for providing information for linking with waste disposal companies and recycling applications, means for evaluating the tidiness of product displays and providing advice regarding improving the display method, and means for identifying infrequently used products and suggesting sales promotions or returns. This enables more efficient inventory management and product display in retail stores and brick-and-mortar stores, and the provision of appropriate advice based on the frequency of product use.

[0901] "Image data" refers to a collection of electronically acquired visual information that can be analyzed to identify an object.

[0902] A "cloud server" is a server that stores and processes data via the Internet, and is responsible for processing information in cooperation with user devices.

[0903] "Means for identifying objects" refers to the algorithms or software used to identify specific objects from image data.

[0904] "Means for evaluating the state of a room" refers to a method for quantifying and evaluating the degree of tidiness of a room and the arrangement of objects based on the results of image analysis.

[0905] "Means for generating and providing advice" refers to a method for creating specific improvement methods and suggestions based on the evaluation results and notifying the user.

[0906] "User interface" refers to the screens and operating means that allow users to interact with the system, specifically those that assist with labeling items and inputting information.

[0907] "Label information" refers to metadata such as category and frequency of use associated with a particular item.

[0908] "Tracking means" refers to a method for tracking and recording the usage and movement of a particular object or item.

[0909] "Decluttering advice" refers to specific suggestions and methods for efficiently disposing of unnecessary items.

[0910] "Waste disposal company" refers to a business that provides services to properly dispose of unwanted items.

[0911] "Reuse applications" refer to platforms and services that allow users to reuse or sell unwanted items to other users.

[0912] "Product display tidiness" refers to the state in which products in a store are properly arranged and organized, expressed based on numerical values ​​and evaluation criteria.

[0913] "Sales promotion and return proposals" refers to proposing efficient sales promotion methods and return methods for products that are used infrequently.

[0914] The present invention relates to a system for improving the efficiency of inventory management and product display in retail stores and brick-and-mortar stores, and for providing advice based on frequency of use. Next, a specific embodiment of this system will be described in detail.

[0915] System Configuration

[0916] First, a user takes a photo of the product display in the store using a device such as a smartphone. The captured image data is sent from the device via a network to a cloud server. The cloud server analyzes the received image data and identifies the objects.

[0917] Specifically, the cloud server uses an image analysis algorithm (for example, using an AI model such as TensorFlow) to automatically identify products and display shelves, and determine their location and size. Based on the results of this analysis, the cloud server quantifies and evaluates the store's tidiness.

[0918] Based on the evaluation results, the cloud server generates specific advice for improving product display, such as adjusting shelf height or placing specific products in the front. The advice is generated in text and illustration format and sent to the user's device.

[0919] The user checks the advice on their device and labels each product with information such as category and frequency of use. The label information is then sent to the cloud server, which then tracks the frequency of use of the product. After a certain period of time, the user takes another photo of the product display and sends it to the cloud server.

[0920] The cloud server compares the old and new image data to identify infrequently used or unsold products. Based on these results, the cloud server generates specific processing methods, such as proposing sales promotion campaigns or recommending product returns. For example, these suggestions include "put infrequently used products on sale" and "return unsold products."

[0921] Furthermore, the cloud server generates link information with appropriate waste disposal companies and recycling applications for disposing of unwanted products based on product information, allowing users to easily find ways to dispose of or reuse unwanted products.

[0922] Specific examples

[0923] For example, if a user takes a photo of the inside of a store and sends it to a cloud server, the server analyzes the image to identify the type of product and its display status. Based on the analysis results, advice such as "place a specific product in the front" is generated and sent to the user's device. The user then rearranges the products according to the advice and sends the results back to the server. The server compares the old and new image data and suggests how to handle less frequently used products.

[0924] Prompt Sentence Examples

[0925] "Analyze in-store photos to assess the organization of product displays and generate recommendations on efficient organization and identifying underused items."

[0926] The system configuration and specific embodiments of the present invention have been described above.

[0927] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0928] Step 1:

[0929] A user takes a photo of the product display in a store using a smartphone, thereby capturing visual information about the store as image data, which is used as input for the next processing step.

[0930] Step 2:

[0931] The device sends the captured image data to a cloud server. In this sending process, the image data is uploaded to the cloud server via a network. The input is the image data, and the output is the image data received by the cloud server.

[0932] Step 3:

[0933] The cloud server analyzes the received image data and identifies the objects. Specifically, it uses a generative AI model (such as TensorFlow) to analyze the image data and identify the type, location, and size of each product. The input is image data, and the output is object identification information.

[0934] Step 4:

[0935] The cloud server evaluates the store's tidiness based on the object identification information. The evaluation is based on criteria such as the number and arrangement of products, and how space is used. The input is object identification information, and the output is the evaluation result of the tidiness.

[0936] Step 5:

[0937] The cloud server generates advice on improving product displays based on the results of the organization evaluation. The advice includes specific suggestions such as "adjust the height of shelves" or "place specific products at the front." The input is the organization evaluation result, and the output is advice text and illustration information.

[0938] Step 6:

[0939] The device receives the advice sent from the cloud server and provides it to the user through a user interface. At this stage, the user can check the advice text. The input is the advice information, and the output is the interface display used by the user.

[0940] Step 7:

[0941] The user follows the advice and labels the product. The label contains information such as the product category and frequency of use. The user retransmits this information to the cloud server via their device. The input is the label information, and the output is the data to be sent to the cloud server.

[0942] Step 8:

[0943] The cloud server tracks product usage frequency based on label information. This tracking is done to record and analyze product usage over time and identify less frequently used products. Label information is the input, and usage frequency data is the output.

[0944] Step 9:

[0945] The user periodically takes photos of the product displays in the store and sends them to the cloud server via their device. The cloud server compares the old and new image data to identify infrequently used products and unsold products. The old and new image data are input, and the output is the identification of infrequently used products.

[0946] Step 10:

[0947] The cloud server generates promotional and return suggestions for the identified infrequently used products. Suggestions include "put infrequently used products on sale" and "return unsold products." The input is the identification of infrequently used products, and the output is the promotional and return suggestions.

[0948] Step 11:

[0949] The cloud server generates linkage information with appropriate waste disposal companies and reuse applications for disposing of unwanted products. This allows users to easily obtain a means for disposing of or reusing unwanted products. Information about unwanted products is input, and linkage information is output.

[0950] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0951] This invention combines a system that uses AI to efficiently support organizing and storage, and decluttering, with an emotion engine that recognizes the user's emotions. Below, we will generate a program for this system and explain the program's processing in natural language.

[0952] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[0953] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[0954] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. When the user inputs facial expressions and voice data using the device's camera and microphone, the device analyzes this and sends it to a cloud server as emotion data. The emotion engine uses this data to identify the user's emotions. For example, it can recognize when the user is tired, stressed, or relaxed.

[0955] The cloud server has the ability to adjust the advice it provides based on the recognized emotional data. For example, if the user is feeling stressed, it can generate advice that reduces the user's burden by breaking it down into simpler steps or adding words of encouragement. On the other hand, if the user is relaxed, it can provide more detailed advice.

[0956] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[0957] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[0958] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[0959] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[0960] When the user confirms this advice and uses the device's camera to capture their facial expression, the emotion engine analyzes it and, if it determines that the user is slightly tired, sends this information to the cloud server. Based on this emotion data, the cloud server generates gentle advice for the user, such as, "Start with one step today. If you try again tomorrow, you'll feel even better."

[0961] This system allows users to organize their rooms and promote decluttering while minimizing the emotional burden. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[0962] The processing flow will be explained below.

[0963] Step 1:

[0964] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[0965] Step 2:

[0966] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[0967] Step 3:

[0968] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[0969] Step 4:

[0970] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[0971] Step 5:

[0972] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[0973] Step 6:

[0974] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[0975] Step 7:

[0976] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[0977] Step 8:

[0978] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[0979] Step 9:

[0980] The device sends the labeled information to the server, where it is stored in an item database.

[0981] Step 10:

[0982] The user inputs their facial expressions and voice using the device's interface, and the device's camera and microphone are used to collect data, which is then analyzed by the emotion engine.

[0983] Step 11:

[0984] The device sends emotional data analyzed by the emotion engine to a cloud server, which determines, for example, whether the user is feeling stressed or relaxed.

[0985] Step 12:

[0986] The server adjusts the advice content based on the emotional data it receives. For example, if the user is feeling stressed, it generates advice that includes simple tasks and encouraging words.

[0987] Step 13:

[0988] The server sends the adjusted advice to the device, and the advice optimized for the user is displayed within the app.

[0989] Step 14:

[0990] The user periodically takes new photos of the room and sends them to the cloud server from the device.

[0991] Step 15:

[0992] The server receives the new image data and performs a comparative analysis with the previous image data, again identifying changes in object location and category and updating the usage frequency data.

[0993] Step 16:

[0994] The server identifies items that are used less frequently based on usage frequency data, generates effective decluttering advice, and sends it to the device.

[0995] Step 17:

[0996] The device displays decluttering advice to the user and, if necessary, provides information on using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[0997] This process allows users to efficiently organize their rooms and declutter while minimizing the emotional burden. The cloud server automatically provides advice and collaborative information, reducing the burden on users.

[0998] Example 2

[0999] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1000] In modern society, efficient organization and storage, as well as decluttering, are challenges faced by many people. These tasks require time and effort, especially for households with many possessions and businesspeople with limited time. Furthermore, because a user's motivation for organizing varies greatly depending on their emotional state, simply providing instructions on the task does not produce sufficient results. Furthermore, there is a lack of specific advice on the appropriate timing for decluttering, or on how to reuse and dispose of items. To address these challenges, an integrated system is needed that provides flexible advice based on the user's emotions and supports decluttering by tracking the frequency of use of items.

[1001] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating advice regarding the room condition and providing it to the user, means for acquiring user emotion data, means for transmitting the acquired emotion data to the cloud server, means for analyzing the emotion data and adjusting the advice in the cloud server, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, and means for providing information for linking with waste disposal companies and applications. This not only enables users to efficiently tidy up their rooms but also enables them to easily implement appropriate decluttering and reuse methods while reducing emotional burden.

[1002] "Image data" is data representing visual information acquired using a photographing device such as a digital camera or smartphone.

[1003] A "cloud server" refers to a distributed server system that can store and process data over the Internet.

[1004] "Object identification" is the process of identifying various items contained in image data and recognizing their categories and attributes.

[1005] "Emotion data" is information that indicates the emotional state of the user, analyzed from facial expressions, voice, and the like.

[1006] "Advice generation" is the process of creating advice on actions and methods suitable for the user based on the analysis results and evaluation data.

[1007] "User interface" refers to the screens and operating means that allow users to directly interact with the system.

[1008] "Label information" is information that includes data on the category and frequency of use assigned to each item.

[1009] "Usage frequency tracking" is the process of recording and monitoring how often each item is used.

[1010] "Danshari Advice" is a process that provides advice on organizing and disposing of unnecessary items based on data such as frequency of use.

[1011] A "waste disposal company" refers to a company or organization that specializes in disposing of unwanted materials.

[1012] "Application" means software designed to provide a specific function or service.

[1013] This invention is a system that supports efficient organizing and storage and decluttering. This system aims to reduce the physical and emotional burden on the user by utilizing AI to recognize the user's emotions and provide advice based on those emotions. The invention includes the following processing steps.

[1014] Users take photos of their rooms using devices such as smartphones. The captured photo data is then sent from the device to a cloud server. This transmission process uses a common communication protocol and applies encryption technology to ensure data security.

[1015] The cloud server analyzes the received photo data. This analysis is performed using a programming language such as Python and image analysis algorithms such as OpenCV and TensorFlow. First, the cloud server identifies objects in the photo (e.g., sofa, table, magazine, etc.). This allows the location and category of the object to be determined.

[1016] The cloud server then evaluates the room's tidiness based on the identified objects. This evaluation takes into account the number and location of objects, the utilization rate of storage space, and other factors. The evaluation results are quantified and specific advice (e.g., "Put magazines in a storage box" or "Put things away on the table") is generated. This advice is presented in text or illustration format and sent to the device.

[1017] Additionally, users can input their facial expressions and voice data using the device's camera and microphone. The input emotional data is analyzed on the device and sent to a cloud server. This analysis is performed using emotion analysis software (e.g., Affectiva). The cloud server uses this data to identify the user's emotional state and tailor the advice provided. For example, if the user is feeling stressed, the advice can be broken down into simple steps or accompanied by encouraging words to reduce the user's burden. On the other hand, if the user is feeling relaxed, the cloud server can provide more detailed advice.

[1018] Users can review the advice provided through their device and label each item. This label includes information such as the item's category and how often it is used. The label information is sent to a cloud server, which tracks how often it is used. The cloud server compares the old and new image data to identify items that are used less frequently. Based on this, the system suggests efficient ways to declutter (for example, recycling or selling items on a flea market app).

[1019] The cloud server also categorizes items and generates link information with appropriate waste disposal companies and flea market apps, allowing users to easily dispose of or reuse unwanted items.

[1020] As a concrete example, consider the case where a user takes a photo of their living room and sends it to a cloud server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device.

[1021] An example prompt is:

[1022] "Analyze a photo of a living room and identify the sofa, table, and scattered magazines in the room. Then, rate the room's tidiness and generate specific advice. Also, if the emotion engine determines that the user is a little tired, use that information to provide gentle advice."

[1023] In this way, the system supports users in efficiently organizing their rooms and promoting decluttering. It also provides flexible advice based on the user's emotional state, reducing the emotional burden on the user.

[1024] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1025] Step 1:

[1026] A user takes a photo of a room using a device such as a smartphone. When the user taps the "Take Photo" button, the device's camera is activated and a photo of the room is captured. The input is image data acquired through the device camera, and the output is saved as the captured photo.

[1027] Step 2:

[1028] The device sends the captured photo data to the cloud server. When the user taps the "Send" button, the device uploads the photo data to the cloud server via the Internet. This process uses a communication protocol (e.g., HTTPS). The input is the saved photo data, and the output is the image data sent to the cloud server.

[1029] Step 3:

[1030] The cloud server analyzes the received image data using an image analysis algorithm on the server (e.g., OpenCV, TensorFlow). This algorithm identifies objects in the photo and determines their location and category. The input is the image data sent to the cloud server, and the output is the category information and location information of the identified objects.

[1031] Step 4:

[1032] The cloud server evaluates the room's tidiness based on the analysis results. For example, it calculates the number of objects, their locations, and the utilization rate of storage space. A numerical score is assigned to the evaluation, and the room's condition is evaluated based on the results. The input is the identified object information, and the output is a numerical evaluation of the tidiness.

[1033] Step 5:

[1034] The cloud server generates advice about the state of the room based on the evaluation results. The advice includes specific instructions such as "put the magazines in a storage box" or "put away the items on the table." This advice is generated in the form of text or illustrations and sent to the device. The input is the evaluation result of the degree of tidiness, and the output is the generated advice.

[1035] Step 6:

[1036] The user inputs facial and voice data using the device's camera and microphone. When the user taps the "Emotion Analysis" button, the device collects the user's emotional data. The input is the user's facial and voice data, and the output is composed of emotional data.

[1037] Step 7:

[1038] The device sends the collected emotion data to the cloud server. When the user taps the "Send Emotion Data" button, the device uploads the emotion data to the cloud server. The input is the composed emotion data, and the output is the emotion data sent to the cloud server.

[1039] Step 8:

[1040] The cloud server analyzes the emotional data and identifies the user's emotional state. It uses an emotion analysis engine (e.g., Affectiva) to determine whether the user is stressed or relaxed. The input is the emotional data sent to the cloud server, and the output is the analyzed emotional state.

[1041] Step 9:

[1042] The cloud server adjusts advice according to the analyzed emotional state. For example, if the user is feeling stressed, it will break down the advice into simple steps. The input is the analyzed emotional state, and the output is the adjusted advice.

[1043] Step 10:

[1044] The user checks the advice and labels each item. Through the user interface, the user inputs label information (such as category and frequency of use). The input is the label information entered by the user, and the output is the labeled item information.

[1045] Step 11:

[1046] The device sends the label information to the cloud server. When the user taps the "Send Label Information" button, the device uploads the label information to the cloud server. The input is the labeled item information, and the output is the label information sent to the cloud server.

[1047] Step 12:

[1048] The cloud server tracks the frequency of use of each item based on the label information sent to it. It analyzes the usage frequency data and records information such as "used once a month" in a database. The input is the label information sent to the cloud server, and the output is the recorded usage frequency data.

[1049] Step 13:

[1050] The cloud server periodically compares old and new image data received and identifies items that are used less frequently. Based on this, it proposes efficient ways to declutter. For example, it generates advice such as "recyclable" or "sellable on a flea market app." The input is the tracked usage frequency data and old and new image data, and the output is advice on decluttering.

[1051] Step 14:

[1052] The cloud server classifies the items and generates linkage information for appropriate waste disposal companies and flea market apps. For example, it generates specific information such as "recycle this item" or "sell this on a specific flea market app." The input is data on infrequently used items, and the output is the linkage information provided.

[1053] (Application example 2)

[1054] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1055] Previous systems that support organizing and decluttering provided uniform advice without considering the user's emotional state. As a result, they lacked appropriate support even when users were feeling tired or stressed, hindering efficient organizing. Furthermore, there was no comprehensive approach for specific inventory management or display optimization, making it impossible to provide advice tailored to individual situations.

[1056] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating and providing advice regarding the room condition to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, means for providing information for linking with waste disposal companies and electronic trading platforms, means for recognizing and analyzing the user's emotions, and means for adjusting and providing advice based on the emotion data. This enables flexible and efficient support for organizing and decluttering that takes the user's emotional state into consideration.

[1057] A "means for acquiring image data" is a device for capturing images of the physical environment, such as by using a camera on a device.

[1058] The "means for transmitting the acquired image data to the cloud server" is a mechanism for uploading the captured image data to a server in a remote location via the Internet.

[1059] The "means for analyzing image data and identifying objects on a cloud server" is a technology for recognizing and classifying objects in an image using an image analysis algorithm on a cloud server.

[1060] "Means for evaluating the state of a room based on analysis results" refers to a method for evaluating the tidiness of a specified physical space based on the results of image analysis and expressing it as a numerical value or status.

[1061] The "means for generating and providing advice to the user regarding the condition of the room" refers to a system that, based on the evaluation results, creates specific instructions and suggestions for tidying up that the user should carry out and notifies them to the user.

[1062] The "means for providing a user interface for labeling items" is an interface that allows a user to digitally or physically assign information such as a name or category to an item.

[1063] The "means for transmitting label information to a cloud server" refers to a method for uploading label information assigned by a user to a cloud server and storing or analyzing the data.

[1064] The "means for tracking the frequency of use of items on a cloud server" is a system that monitors and records the frequency of use of each item based on label information and other data stored on a cloud server.

[1065] The "means for generating decluttering advice for infrequently used items" is a method for identifying infrequently used items and generating instructions or suggestions for efficiently disposing of them.

[1066] The "means for providing information for linking with waste disposal companies and electronic trading platforms" is a system that provides users with link information with appropriate companies and platforms for disposing of or reusing items.

[1067] "Means for recognizing and analyzing user emotions" refers to technology that uses a camera or microphone to obtain emotional data from the user's facial expressions and voice, and then analyzes this data.

[1068] "Means for adjusting and providing advice based on emotional data" refers to a method for flexibly adjusting the content and format of the advice provided according to the recognized emotional state of the user and providing feedback to the user.

[1069] This invention uses a system that combines AI technology and an emotion engine to support inventory management and display optimization in physical stores. The system sends image data taken by store staff using smart glasses to a cloud server, and provides efficient advice through image and emotion analysis.

[1070] 1. System Program

[1071] The server includes the following means:

[1072] Means for acquiring image data

[1073] A means of sending image data to a cloud server

[1074] A means for analyzing image data and identifying objects on a cloud server

[1075] A means of evaluating the status of the store based on analysis results

[1076] A means of generating and providing state advice to the user

[1077] Means for providing a user interface for labeling items

[1078] A means of sending label information to the cloud server

[1079] A method for tracking item usage frequency on a cloud server

[1080] A means of generating decluttering advice for less frequently used items

[1081] A means of providing information for working with waste disposal companies and electronic trading platforms

[1082] A means of recognizing and analyzing user emotions

[1083] A means to tailor and provide advice based on sentiment data

[1084] 2. A natural language description of the program's operation

[1085] The system uses the following hardware and software:

[1086] Hardware: Smart glasses (Google Glass, Vuzix, etc.), camera, microphone

[1087] Software: AWS (or Azure) cloud services, OpenCV (image analysis), NVIDIA Emotion AI (emotion analysis)

[1088] First, a user (store staff member) uses smart glasses to take a photo of inventory shelves or display shelves. This image data is sent from the smart glasses to a cloud server. The cloud server uses image analysis software such as OpenCV to identify objects in the image and evaluate the shelf status. Based on the evaluation results, the cloud server generates specific advice regarding display and inventory and provides it to the user via the smart glasses.

[1089] The smart glasses' cameras and microphones are then used to capture the user's facial expressions and voice data. This data is then used with emotion analysis software such as NVIDIA Emotion AI to recognize the user's emotional state. The cloud server then adjusts the content of the advice provided to the user based on the recognized emotional data.

[1090] For example, if the user is tired, the system suggests simple tidying tasks, while if the user is relaxed, it provides more detailed display optimization advice. The user can follow the advice provided and use the system to receive new advice whenever the situation changes.

[1091] Using this system, store staff can manage inventory and optimize display efficiently and with minimal emotional burden.

[1092] 3. Examples of concrete examples and prompts

[1093] Example 1: During the night shift, staff use the system to organize stock shelves. Tired staff are given the advice, "Just move the items on the easy list today and be done."

[1094] Example 2: This system is used to review the overall store display before the peak season. Staff who have spare time are given detailed advice such as "Review the current display and make space to add new products."

[1095] Example prompt sentence:

[1096] input:

[1097] Take a photo of your stocked shelves today and ask for advice. Also, assess the emotional state of your staff.

[1098] output:

[1099] We've analyzed your current inventory shelves. You seem a little tired, so let's start today with a quick cleanup. First, check the leftmost column. Then, check the rightmost column and combine any duplicate items onto a single shelf. Good luck!

[1100] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1101] Step 1:

[1102] A user wears smart glasses and takes a photo of an inventory or display shelf. The camera in the smart glasses captures the image data, which is then stored on the device.

[1103] Input: A photo of a shelf in a store

[1104] Output: Image data

[1105] Step 2:

[1106] The acquired image data is transmitted from the smart glasses to a cloud server, and the image data is uploaded to the cloud server via the Internet using the network connection function of the smart glasses.

[1107] Input: Image data

[1108] Output: Image data stored on a cloud server

[1109] Step 3:

[1110] The cloud server analyzes the image data using an image analysis algorithm (e.g., using OpenCV) to identify objects in the image, and then determines the object's category and location.

[1111] Input: Image data stored on a cloud server

[1112] Output: Object category and location

[1113] Step 4:

[1114] The cloud server evaluates the store's condition based on the analysis results, and the condition is quantified based on indicators such as tidiness and inventory status.

[1115] Input: Object category and location

[1116] Output: Evaluation result (number and status)

[1117] Step 5:

[1118] The cloud server generates specific advice based on the evaluation results, including specific instructions and suggestions for tidying up the user.

[1119] Input: Evaluation result

[1120] Output: Text data of advice

[1121] Step 6:

[1122] The generated advice is sent to the smart glasses, and the user can view the advice on the display of the smart glasses.

[1123] Input: Text data of advice

[1124] Output: Advice displayed on the smart glasses display

[1125] Step 7:

[1126] When a user inputs facial expressions and voice data through the smart glasses, the data is sent to a cloud server, where the user's emotional data is acquired using the smart glasses' camera and microphone.

[1127] Input: User facial and voice data

[1128] Output: Emotion data sent to the cloud server

[1129] Step 8:

[1130] The cloud server analyzes the emotional data using an emotion analysis algorithm (e.g., using NVIDIA Emotion AI) to identify the user's emotional state.

[1131] Input: Emotion data

[1132] Output: User's emotional state (fatigue level, stress level, etc.)

[1133] Step 9:

[1134] The cloud server tailors the advice based on the user's emotional state: if the user is tired, it suggests simple tasks, and if the user is relaxed, it offers detailed organization tips.

[1135] Input: User's emotional state

[1136] Output: Text data of the adjusted advice

[1137] Step 10:

[1138] The adjusted advice is then sent back to the smart glasses, where the user can review the new advice, allowing them to efficiently manage inventory and optimize display.

[1139] Input: Text data of the adjusted advice

[1140] Output: New advice displayed on the smart glasses display

[1141] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1142] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1143] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1144] [Fourth embodiment]

[1145] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1146] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1147] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1148] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1149] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1150] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1151] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1152] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1153] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1154] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1155] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1156] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1157] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1158] This invention relates to a system that utilizes AI to support efficient organizing and storage, and decluttering. Below, we will generate a program for this system and explain the program's processing in natural language.

[1159] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[1160] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[1161] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[1162] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[1163] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[1164] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[1165] Users use the app to label magazines and other items, then take a photo of their room and send it to the cloud server, which compares the old and new images to analyze how often items are used. For items that are used less frequently, the app offers recommendations for recycling or selling them.

[1166] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[1167] The processing flow will be explained below.

[1168] Step 1:

[1169] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[1170] Step 2:

[1171] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[1172] Step 3:

[1173] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[1174] Step 4:

[1175] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[1176] Step 5:

[1177] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[1178] Step 6:

[1179] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[1180] Step 7:

[1181] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[1182] Step 8:

[1183] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[1184] Step 9:

[1185] The device sends the labeled information to the server, where it is stored in an item database.

[1186] Step 10:

[1187] The server periodically sends reminders to the user to take photos of the room again, and the user who receives the reminder takes photos of the room again.

[1188] Step 11:

[1189] The user takes another photo of the room and the device sends this new image data to the server.

[1190] Step 12:

[1191] The server receives the new image data, analyzes it, and compares it with the previous image data. The image analysis engine again identifies the object's location and category, and updates the usage frequency data.

[1192] Step 13:

[1193] The server identifies items that are used less frequently based on the updated usage frequency data, and uses this identified data to generate effective decluttering advice.

[1194] Step 14:

[1195] The server then sends the generated decluttering advice to the device, including advice on recycling, referrals to waste disposal companies, and selling items on a flea market app.

[1196] Step 15:

[1197] The device displays decluttering advice to the user and, if necessary, provides link information for using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[1198] This process allows the user to efficiently organize and tidy up their room.

[1199] Example 1

[1200] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1201] In the past, organizing and decluttering a room required a lot of effort, and it was difficult to get specific advice on how to organize and properly dispose of items.It was also difficult to understand how often items were used and how to declutter efficiently.

[1202] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1203] In this invention, the server includes means for analyzing image data and identifying objects, means for providing a user interface for tagging the identified objects, and means for evaluating the degree of tidiness of a room based on the analysis results, thereby making it possible to automatically evaluate the tidiness of a room and provide the user with specific advice on tidiness and decluttering.

[1204] "Image data" refers to digital data of still images or moving images recorded as visual information.

[1205] A "cloud server" is a remote server that processes data and provides storage over a network.

[1206] "Means for identifying objects" refers to technology that analyzes image data to recognize and classify the various objects contained therein.

[1207] "User interface" refers to the operation screen and input device that allow the user to directly interact with the system.

[1208] A "means for assessing the degree of tidiness" is a technique that uses an algorithm or method for assessing the state of a room to express the level of tidiness as a number or evaluation comment.

[1209] "Label information" is a tag or identifier used to indicate additional information such as the category or frequency of use of an object.

[1210] "Means for tracking frequency of use" refers to technology that records and analyzes the usage of each object to determine frequency.

[1211] The "means for generating decluttering advice" is a technology that generates specific suggestions that instruct how to dispose of infrequently used objects.

[1212] "Information on collaboration with waste disposal companies and marketplaces" refers to data on companies and trading platforms necessary for properly disposing of or selling unwanted items.

[1213] This invention relates to a system that uses AI to efficiently support organizing and storing things. Below, we will create a program for this system and explain how it works.

[1214] This system includes hardware and software such as a user's device (e.g., a smartphone), a cloud server, and an image analysis algorithm (specifically, using machine learning frameworks such as TensorFlow and PyTorch). The specific process is as follows:

[1215] First, the user takes a photo of the room using their smartphone. This is done using a dedicated application. The captured photo data is then sent from the device to a cloud server. The sending method is, for example, the HTTPS protocol. If an error occurs during transmission, the device has the ability to attempt to resend the data.

[1216] The cloud server decompresses the received image data and begins image analysis. This analysis uses machine learning models such as TensorFlow to execute object detection algorithms. The server identifies each object in the image and assigns it a category (e.g., sofa, table, book, etc.). It also obtains each object's location information (coordinate data).

[1217] The cloud server then evaluates the room's tidiness based on the identified object information. This evaluation analyzes the number of recognized objects and their category arrangement, calculating, for example, the number of scattered items and the utilization rate of storage space. The server then quantifies these evaluation results and generates a rating such as "poorly organized" or "somewhat organized."

[1218] Based on the evaluation results, the cloud server generates specific advice on tidying up the room, such as "put magazines in a storage box" or "put away items on the table." The advice is sent to the device in text or illustration format.

[1219] The user checks the advice provided through the device and attaches tags or labels to each item. These labels include information such as the item's category and frequency of use. The device then sends the label information entered by the user to the cloud server.

[1220] The cloud server stores the received label information in a database and runs an analysis algorithm to track the frequency of use of each item. Periodically, the user takes new photos of the room and sends them to the cloud server from their device. The server compares the old and new image data to identify items that have been used less frequently.

[1221] Finally, the cloud server suggests efficient ways to declutter infrequently used items. For example, it recommends recycling methods for recyclable items and selling items that are still usable on a marketplace. Based on the item classification results, the server generates link information for appropriate waste disposal companies and marketplaces and sends this information to the device.

[1222] As a concrete example, consider the case where a user takes a photo of their living room and sends it to the server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device. The user then uses the app to label the magazines and other items, takes another photo of the room, and sends it to the server. The cloud server compares the old and new images and analyzes how often the items are used. For items that are used less frequently, it provides recommendations for recycling or selling them.

[1223] An example of a prompt is, "Take a photo of your living room and send it to the server through the app. You will soon see a tidy-up rating and specific advice."

[1224] This system allows users to efficiently organize their rooms and promote decluttering. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[1225] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1226] Step 1:

[1227] The user takes a photo of the room using their smartphone. They select a room that contains a wide range of visual information, such as a living room. After taking the photo, they press the "Upload" button to proceed to the next step.

[1228] Input: Photo of the room (image data)

[1229] Output: Image data saved on the device

[1230] Step 2:

[1231] The device will send the captured photo data to the cloud server. The photo data will be compressed first and then securely transmitted using the HTTPS protocol. The retry function will be enabled until the transmission is complete.

[1232] Input: Image data stored on the device

[1233] Output: Image data sent to the cloud server

[1234] Step 3:

[1235] The server decompresses the received image data and begins analyzing it using the TensorFlow model. The image analysis algorithm performs object detection and identifies objects in the image.

[1236] Input: Image data sent to the cloud server

[1237] Output: Identified object category and location information (coordinate data)

[1238] Step 4:

[1239] The server evaluates the room's tidiness based on the identified object information, using an analytical algorithm to analyze the number of objects and their category arrangement, and calculates the "number of cluttered items" and "storage space utilization rate."

[1240] Input: Identified object category and location information

[1241] Output: Organization score and evaluation results

[1242] Step 5:

[1243] Based on the evaluation results, the server generates specific advice on tidying up the room. For example, the advice includes instructions such as "put magazines in a storage box" and "put away items on the table." The advice is generated in the form of text and illustrations.

[1244] Input: Organization score and evaluation results

[1245] Output: Advice text and illustration data

[1246] Step 6:

[1247] The server sends the generated advice to the terminal, where the user can see specific instructions on how to tidy up their room.

[1248] Input: Advice text and illustration data

[1249] Output: Advice sent to terminal

[1250] Step 7:

[1251] The user tags or labels each item based on the advice provided. The label contains information such as the object's category and frequency of use. When the user presses the "Done tagging" button, the label information is sent to the server.

[1252] Input: Label information added based on the advice

[1253] Output: Label information sent to the cloud server

[1254] Step 8:

[1255] The server stores the received label information in a database and starts tracking usage frequency. It runs an analysis algorithm to accumulate data and record the frequency of use of each item.

[1256] Input: Label information sent to the cloud server

[1257] Output: Usage frequency information stored in a database

[1258] Step 9:

[1259] Periodically, the user takes another photo of the room and presses the "Reevaluate" button to send the image data from the device to the cloud server, where it is analyzed.

[1260] Input: Newly taken photo of the room (image data)

[1261] Output: New image data sent to the cloud server

[1262] Step 10:

[1263] The server compares the old and new image data to identify less frequently used items, and generates advice on how to efficiently declutter.

[1264] Input: Old and new image data

[1265] Output: A list of items that are rarely used and advice on decluttering

[1266] Step 11:

[1267] The server generates linked information about waste disposal companies and marketplaces based on the item classification results, along with decluttering advice, and sends this to the terminal.

[1268] Input: Decluttering advice and item classification results

[1269] Output: Decluttering advice and collaboration information sent to the device

[1270] (Application example 1)

[1271] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1272] Until now, support systems have been provided to help individuals efficiently organize their homes, but there are limitations to these systems in terms of improving the efficiency of inventory management and product display in retail stores and brick-and-mortar shops, as well as providing appropriate advice based on product usage frequency and sales data. In particular, there is a need for appropriate disposal of infrequently used and unsold products. It is also important to propose efficient product display methods and improve display methods. However, current systems are unable to adequately address these issues.

[1273] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1274] In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the condition of a room based on the analysis results, means for generating and providing advice regarding the condition of the room to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of use of items in the cloud server, means for generating decluttering advice regarding infrequently used items, means for providing information for linking with waste disposal companies and recycling applications, means for evaluating the tidiness of product displays and providing advice regarding improving the display method, and means for identifying infrequently used products and suggesting sales promotions or returns. This enables more efficient inventory management and product display in retail stores and brick-and-mortar stores, and the provision of appropriate advice based on the frequency of product use.

[1275] "Image data" refers to a collection of electronically acquired visual information that can be analyzed to identify an object.

[1276] A "cloud server" is a server that stores and processes data via the Internet, and is responsible for processing information in cooperation with user devices.

[1277] "Means for identifying objects" refers to the algorithms or software used to identify specific objects from image data.

[1278] "Means for evaluating the state of a room" refers to a method for quantifying and evaluating the degree of tidiness of a room and the arrangement of objects based on the results of image analysis.

[1279] "Means for generating and providing advice" refers to a method for creating specific improvement methods and suggestions based on the evaluation results and notifying the user.

[1280] "User interface" refers to the screens and operating means that allow users to interact with the system, specifically those that assist with labeling items and inputting information.

[1281] "Label information" refers to metadata such as category and frequency of use associated with a particular item.

[1282] "Tracking means" refers to a method for tracking and recording the usage and movement of a particular object or item.

[1283] "Decluttering advice" refers to specific suggestions and methods for efficiently disposing of unnecessary items.

[1284] "Waste disposal company" refers to a business that provides services to properly dispose of unwanted items.

[1285] "Reuse applications" refer to platforms and services that allow users to reuse or sell unwanted items to other users.

[1286] "Product display tidiness" refers to the state in which products in a store are properly arranged and organized, expressed based on numerical values ​​and evaluation criteria.

[1287] "Sales promotion and return proposals" refers to proposing efficient sales promotion methods and return methods for products that are used infrequently.

[1288] The present invention relates to a system for improving the efficiency of inventory management and product display in retail stores and brick-and-mortar stores, and for providing advice based on frequency of use. Next, a specific embodiment of this system will be described in detail.

[1289] System Configuration

[1290] First, a user takes a photo of the product display in the store using a device such as a smartphone. The captured image data is sent from the device via a network to a cloud server. The cloud server analyzes the received image data and identifies the objects.

[1291] Specifically, the cloud server uses an image analysis algorithm (for example, using an AI model such as TensorFlow) to automatically identify products and display shelves, and determine their location and size. Based on the results of this analysis, the cloud server quantifies and evaluates the store's tidiness.

[1292] Based on the evaluation results, the cloud server generates specific advice for improving product display, such as adjusting shelf height or placing specific products in the front. The advice is generated in text and illustration format and sent to the user's device.

[1293] The user checks the advice on their device and labels each product with information such as category and frequency of use. The label information is then sent to the cloud server, which then tracks the frequency of use of the product. After a certain period of time, the user takes another photo of the product display and sends it to the cloud server.

[1294] The cloud server compares the old and new image data to identify infrequently used or unsold products. Based on these results, the cloud server generates specific processing methods, such as proposing sales promotion campaigns or recommending product returns. For example, these suggestions include "put infrequently used products on sale" and "return unsold products."

[1295] Furthermore, the cloud server generates link information with appropriate waste disposal companies and recycling applications for disposing of unwanted products based on product information, allowing users to easily find ways to dispose of or reuse unwanted products.

[1296] Specific examples

[1297] For example, if a user takes a photo of the inside of a store and sends it to a cloud server, the server analyzes the image to identify the type of product and its display status. Based on the analysis results, advice such as "place a specific product in the front" is generated and sent to the user's device. The user then rearranges the products according to the advice and sends the results back to the server. The server compares the old and new image data and suggests how to handle less frequently used products.

[1298] Prompt Sentence Examples

[1299] "Analyze in-store photos to assess the organization of product displays and generate recommendations on efficient organization and identifying underused items."

[1300] The system configuration and specific embodiments of the present invention have been described above.

[1301] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1302] Step 1:

[1303] A user takes a photo of the product display in a store using a smartphone, thereby capturing visual information about the store as image data, which is used as input for the next processing step.

[1304] Step 2:

[1305] The device sends the captured image data to a cloud server. In this sending process, the image data is uploaded to the cloud server via a network. The input is the image data, and the output is the image data received by the cloud server.

[1306] Step 3:

[1307] The cloud server analyzes the received image data and identifies the objects. Specifically, it uses a generative AI model (such as TensorFlow) to analyze the image data and identify the type, location, and size of each product. The input is image data, and the output is object identification information.

[1308] Step 4:

[1309] The cloud server evaluates the store's tidiness based on the object identification information. The evaluation is based on criteria such as the number and arrangement of products, and how space is used. The input is object identification information, and the output is the evaluation result of the tidiness.

[1310] Step 5:

[1311] The cloud server generates advice on improving product displays based on the results of the organization evaluation. The advice includes specific suggestions such as "adjust the height of shelves" or "place specific products at the front." The input is the organization evaluation result, and the output is advice text and illustration information.

[1312] Step 6:

[1313] The device receives the advice sent from the cloud server and provides it to the user through a user interface. At this stage, the user can check the advice text. The input is the advice information, and the output is the interface display used by the user.

[1314] Step 7:

[1315] The user follows the advice and labels the product. The label contains information such as the product category and frequency of use. The user retransmits this information to the cloud server via their device. The input is the label information, and the output is the data to be sent to the cloud server.

[1316] Step 8:

[1317] The cloud server tracks product usage frequency based on label information. This tracking is done to record and analyze product usage over time and identify less frequently used products. Label information is the input, and usage frequency data is the output.

[1318] Step 9:

[1319] The user periodically takes photos of the product displays in the store and sends them to the cloud server via their device. The cloud server compares the old and new image data to identify infrequently used products and unsold products. The old and new image data are input, and the output is the identification of infrequently used products.

[1320] Step 10:

[1321] The cloud server generates promotional and return suggestions for the identified infrequently used products. Suggestions include "put infrequently used products on sale" and "return unsold products." The input is the identification of infrequently used products, and the output is the promotional and return suggestions.

[1322] Step 11:

[1323] The cloud server generates linkage information with appropriate waste disposal companies and reuse applications for disposing of unwanted products. This allows users to easily obtain a means for disposing of or reusing unwanted products. Information about unwanted products is input, and linkage information is output.

[1324] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1325] This invention combines a system that uses AI to efficiently support organizing and storage, and decluttering, with an emotion engine that recognizes the user's emotions. Below, we will generate a program for this system and explain the program's processing in natural language.

[1326] The system begins when a user takes a photo of a room using a device such as a smartphone. This photo data is then sent from the device to a cloud server. The cloud server then analyzes the image data and automatically identifies objects in the room. Specifically, it uses an image analysis algorithm to determine the object's category and location.

[1327] The cloud server also evaluates the tidiness of the room based on the identified object information. For example, it considers the number of scattered objects and the utilization rate of storage space to quantify the room's condition. Based on this evaluation result, the cloud server generates specific advice on how to arrange and store items. The advice is provided in text or illustration format and sent to the device.

[1328] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. When the user inputs facial expressions and voice data using the device's camera and microphone, the device analyzes this and sends it to a cloud server as emotion data. The emotion engine uses this data to identify the user's emotions. For example, it can recognize when the user is tired, stressed, or relaxed.

[1329] The cloud server has the ability to adjust the advice it provides based on the recognized emotional data. For example, if the user is feeling stressed, it can generate advice that reduces the user's burden by breaking it down into simpler steps or adding words of encouragement. On the other hand, if the user is relaxed, it can provide more detailed advice.

[1330] Users can review the advice provided through their device and label each item. The label contains information such as the item's category and frequency of use. This label information is then sent back to the cloud server, which then tracks the frequency of use of the item.

[1331] Periodically, the user takes new photos of their room and sends them to the cloud server from their device. The cloud server compares the old and new image data and identifies items that are used less frequently. Based on these results, the cloud server suggests efficient ways to declutter. For example, it may recommend recycling methods for recyclable items, or selling items that are still usable on a flea market app.

[1332] Furthermore, the cloud server classifies items and generates link information with appropriate waste disposal companies and flea market apps, providing users with an easy way to dispose of or reuse unwanted items.

[1333] As a concrete example, let's consider the case where a user takes a photo of their living room and sends it to the server via the app. The cloud server analyzes this image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of tidiness. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put the items on the table away" and sends it to the device.

[1334] When the user confirms this advice and uses the device's camera to capture their facial expression, the emotion engine analyzes it and, if it determines that the user is slightly tired, sends this information to the cloud server. Based on this emotion data, the cloud server generates gentle advice for the user, such as, "Start with one step today. If you try again tomorrow, you'll feel even better."

[1335] This system allows users to organize their rooms and promote decluttering while minimizing the emotional burden. In addition, the cloud server automatically provides advice and collaborative information, allowing users to organize their rooms on a daily basis without feeling burdened.

[1336] The processing flow will be explained below.

[1337] Step 1:

[1338] The user takes a photo of the entire room using the smartphone camera. The user launches the app and taps the camera button to activate the camera function.

[1339] Step 2:

[1340] The device sends the photo data it has taken to a cloud server. After the photo is taken, an API is automatically called and the image data is uploaded to the cloud server.

[1341] Step 3:

[1342] The server temporarily stores the received image data in storage, and the uploaded image data is added to a queue waiting for analysis.

[1343] Step 4:

[1344] The server launches an image analysis engine to analyze the received image data. Specifically, an object detection algorithm (e.g., YOLO, RCNN) identifies objects in the image in real time and determines the location and size information of each object.

[1345] Step 5:

[1346] The server then runs a script based on the analysis results to evaluate the overall condition of the room, for example, calculating the number of cluttered items or the efficiency of storage space utilization according to specific evaluation criteria.

[1347] Step 6:

[1348] Based on the results of the room evaluation, the server automatically generates specific advice on how to arrange and store items, and the advice is saved in text and illustration format.

[1349] Step 7:

[1350] The server sends the generated advice to the device, and the device displays the received advice data within the app.

[1351] Step 8:

[1352] Users use the app's interface to label each item, a process in which they enter information such as category and frequency of use for items that are relevant to their recommendations.

[1353] Step 9:

[1354] The device sends the labeled information to the server, where it is stored in an item database.

[1355] Step 10:

[1356] The user inputs their facial expressions and voice using the device's interface, and the device's camera and microphone are used to collect data, which is then analyzed by the emotion engine.

[1357] Step 11:

[1358] The device sends emotional data analyzed by the emotion engine to a cloud server, which determines, for example, whether the user is feeling stressed or relaxed.

[1359] Step 12:

[1360] The server adjusts the advice content based on the emotional data it receives. For example, if the user is feeling stressed, it generates advice that includes simple tasks and encouraging words.

[1361] Step 13:

[1362] The server sends the adjusted advice to the device, and the advice optimized for the user is displayed within the app.

[1363] Step 14:

[1364] The user periodically takes new photos of the room and sends them to the cloud server from the device.

[1365] Step 15:

[1366] The server receives the new image data and performs a comparative analysis with the previous image data, again identifying changes in object location and category and updating the usage frequency data.

[1367] Step 16:

[1368] The server identifies items that are used less frequently based on usage frequency data, generates effective decluttering advice, and sends it to the device.

[1369] Step 17:

[1370] The device displays decluttering advice to the user and, if necessary, provides information on using waste disposal companies or flea market apps. The user follows the advice to dispose of or reuse unwanted items.

[1371] This process allows users to efficiently organize their rooms and declutter while minimizing the emotional burden. The cloud server automatically provides advice and collaborative information, reducing the burden on users.

[1372] Example 2

[1373] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1374] In modern society, efficient organization and storage, as well as decluttering, are challenges faced by many people. These tasks require time and effort, especially for households with many possessions and businesspeople with limited time. Furthermore, because a user's motivation for organizing varies greatly depending on their emotional state, simply providing instructions on the task does not produce sufficient results. Furthermore, there is a lack of specific advice on the appropriate timing for decluttering, or on how to reuse and dispose of items. To address these challenges, an integrated system is needed that provides flexible advice based on the user's emotions and supports decluttering by tracking the frequency of use of items.

[1375] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating advice regarding the room condition and providing it to the user, means for acquiring user emotion data, means for transmitting the acquired emotion data to the cloud server, means for analyzing the emotion data and adjusting the advice in the cloud server, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, and means for providing information for linking with waste disposal companies and applications. This not only enables users to efficiently tidy up their rooms but also enables them to easily implement appropriate decluttering and reuse methods while reducing emotional burden.

[1376] "Image data" is data representing visual information acquired using a photographing device such as a digital camera or smartphone.

[1377] A "cloud server" refers to a distributed server system that can store and process data over the Internet.

[1378] "Object identification" is the process of identifying various items contained in image data and recognizing their categories and attributes.

[1379] "Emotion data" is information that indicates the emotional state of the user, analyzed from facial expressions, voice, and the like.

[1380] "Advice generation" is the process of creating advice on actions and methods suitable for the user based on the analysis results and evaluation data.

[1381] "User interface" refers to the screens and operating means that allow users to directly interact with the system.

[1382] "Label information" is information that includes data on the category and frequency of use assigned to each item.

[1383] "Usage frequency tracking" is the process of recording and monitoring how often each item is used.

[1384] "Danshari Advice" is a process that provides advice on organizing and disposing of unnecessary items based on data such as frequency of use.

[1385] A "waste disposal company" refers to a company or organization that specializes in disposing of unwanted materials.

[1386] "Application" means software designed to provide a specific function or service.

[1387] This invention is a system that supports efficient organizing and storage and decluttering. This system aims to reduce the physical and emotional burden on the user by utilizing AI to recognize the user's emotions and provide advice based on those emotions. The invention includes the following processing steps.

[1388] Users take photos of their rooms using devices such as smartphones. The captured photo data is then sent from the device to a cloud server. This transmission process uses a common communication protocol and applies encryption technology to ensure data security.

[1389] The cloud server analyzes the received photo data. This analysis is performed using a programming language such as Python and image analysis algorithms such as OpenCV and TensorFlow. First, the cloud server identifies objects in the photo (e.g., sofa, table, magazine, etc.). This allows the location and category of the object to be determined.

[1390] The cloud server then evaluates the room's tidiness based on the identified objects. This evaluation takes into account the number and location of objects, the utilization rate of storage space, and other factors. The evaluation results are quantified and specific advice (e.g., "Put magazines in a storage box" or "Put things away on the table") is generated. This advice is presented in text or illustration format and sent to the device.

[1391] Additionally, users can input their facial expressions and voice data using the device's camera and microphone. The input emotional data is analyzed on the device and sent to a cloud server. This analysis is performed using emotion analysis software (e.g., Affectiva). The cloud server uses this data to identify the user's emotional state and tailor the advice provided. For example, if the user is feeling stressed, the advice can be broken down into simple steps or accompanied by encouraging words to reduce the user's burden. On the other hand, if the user is feeling relaxed, the cloud server can provide more detailed advice.

[1392] Users can review the advice provided through their device and label each item. This label includes information such as the item's category and how often it is used. The label information is sent to a cloud server, which tracks how often it is used. The cloud server compares the old and new image data to identify items that are used less frequently. Based on this, the system suggests efficient ways to declutter (for example, recycling or selling items on a flea market app).

[1393] The cloud server also categorizes items and generates link information with appropriate waste disposal companies and flea market apps, allowing users to easily dispose of or reuse unwanted items.

[1394] As a concrete example, consider the case where a user takes a photo of their living room and sends it to a cloud server via an app. The cloud server analyzes the image and identifies the sofa, table, scattered magazines, etc. The cloud server then evaluates the state of the room as "not organized" and quantifies the degree of organization. Based on this, it generates specific advice such as "put the magazines in a storage box" or "put away the items on the table" and sends it to the device.

[1395] An example prompt is:

[1396] "Analyze a photo of a living room and identify the sofa, table, and scattered magazines in the room. Then, rate the room's tidiness and generate specific advice. Also, if the emotion engine determines that the user is a little tired, use that information to provide gentle advice."

[1397] In this way, the system supports users in efficiently organizing their rooms and promoting decluttering. It also provides flexible advice based on the user's emotional state, reducing the emotional burden on the user.

[1398] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1399] Step 1:

[1400] A user takes a photo of a room using a device such as a smartphone. When the user taps the "Take Photo" button, the device's camera is activated and a photo of the room is captured. The input is image data acquired through the device camera, and the output is saved as the captured photo.

[1401] Step 2:

[1402] The device sends the captured photo data to the cloud server. When the user taps the "Send" button, the device uploads the photo data to the cloud server via the Internet. This process uses a communication protocol (e.g., HTTPS). The input is the saved photo data, and the output is the image data sent to the cloud server.

[1403] Step 3:

[1404] The cloud server analyzes the received image data using an image analysis algorithm on the server (e.g., OpenCV, TensorFlow). This algorithm identifies objects in the photo and determines their location and category. The input is the image data sent to the cloud server, and the output is the category information and location information of the identified objects.

[1405] Step 4:

[1406] The cloud server evaluates the room's tidiness based on the analysis results. For example, it calculates the number of objects, their locations, and the utilization rate of storage space. A numerical score is assigned to the evaluation, and the room's condition is evaluated based on the results. The input is the identified object information, and the output is a numerical evaluation of the tidiness.

[1407] Step 5:

[1408] The cloud server generates advice about the state of the room based on the evaluation results. The advice includes specific instructions such as "put the magazines in a storage box" or "put away the items on the table." This advice is generated in the form of text or illustrations and sent to the device. The input is the evaluation result of the degree of tidiness, and the output is the generated advice.

[1409] Step 6:

[1410] The user inputs facial and voice data using the device's camera and microphone. When the user taps the "Emotion Analysis" button, the device collects the user's emotional data. The input is the user's facial and voice data, and the output is composed of emotional data.

[1411] Step 7:

[1412] The device sends the collected emotion data to the cloud server. When the user taps the "Send Emotion Data" button, the device uploads the emotion data to the cloud server. The input is the composed emotion data, and the output is the emotion data sent to the cloud server.

[1413] Step 8:

[1414] The cloud server analyzes the emotional data and identifies the user's emotional state. It uses an emotion analysis engine (e.g., Affectiva) to determine whether the user is stressed or relaxed. The input is the emotional data sent to the cloud server, and the output is the analyzed emotional state.

[1415] Step 9:

[1416] The cloud server adjusts advice according to the analyzed emotional state. For example, if the user is feeling stressed, it will break down the advice into simple steps. The input is the analyzed emotional state, and the output is the adjusted advice.

[1417] Step 10:

[1418] The user checks the advice and labels each item. Through the user interface, the user inputs label information (such as category and frequency of use). The input is the label information entered by the user, and the output is the labeled item information.

[1419] Step 11:

[1420] The device sends the label information to the cloud server. When the user taps the "Send Label Information" button, the device uploads the label information to the cloud server. The input is the labeled item information, and the output is the label information sent to the cloud server.

[1421] Step 12:

[1422] The cloud server tracks the frequency of use of each item based on the label information sent to it. It analyzes the usage frequency data and records information such as "used once a month" in a database. The input is the label information sent to the cloud server, and the output is the recorded usage frequency data.

[1423] Step 13:

[1424] The cloud server periodically compares old and new image data received and identifies items that are used less frequently. Based on this, it proposes efficient ways to declutter. For example, it generates advice such as "recyclable" or "sellable on a flea market app." The input is the tracked usage frequency data and old and new image data, and the output is advice on decluttering.

[1425] Step 14:

[1426] The cloud server classifies the items and generates linkage information for appropriate waste disposal companies and flea market apps. For example, it generates specific information such as "recycle this item" or "sell this on a specific flea market app." The input is data on infrequently used items, and the output is the linkage information provided.

[1427] (Application example 2)

[1428] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1429] Previous systems that support organizing and decluttering provided uniform advice without considering the user's emotional state. As a result, they lacked appropriate support even when users were feeling tired or stressed, hindering efficient organizing. Furthermore, there was no comprehensive approach for specific inventory management or display optimization, making it impossible to provide advice tailored to individual situations.

[1430] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and identifying objects in the cloud server, means for evaluating the room condition based on the analysis results, means for generating and providing advice regarding the room condition to the user, means for providing a user interface for labeling items, means for transmitting label information to the cloud server, means for tracking the frequency of item use in the cloud server, means for generating decluttering advice for infrequently used items, means for providing information for linking with waste disposal companies and electronic trading platforms, means for recognizing and analyzing the user's emotions, and means for adjusting and providing advice based on the emotion data. This enables flexible and efficient support for organizing and decluttering that takes the user's emotional state into consideration.

[1431] A "means for acquiring image data" is a device for capturing images of the physical environment, such as by using a camera on a device.

[1432] The "means for transmitting the acquired image data to the cloud server" is a mechanism for uploading the captured image data to a server in a remote location via the Internet.

[1433] The "means for analyzing image data and identifying objects on a cloud server" is a technology for recognizing and classifying objects in an image using an image analysis algorithm on a cloud server.

[1434] "Means for evaluating the state of a room based on analysis results" refers to a method for evaluating the tidiness of a specified physical space based on the results of image analysis and expressing it as a numerical value or status.

[1435] The "means for generating and providing advice to the user regarding the condition of the room" refers to a system that, based on the evaluation results, creates specific instructions and suggestions for tidying up that the user should carry out and notifies them to the user.

[1436] The "means for providing a user interface for labeling items" is an interface that allows a user to digitally or physically assign information such as a name or category to an item.

[1437] The "means for transmitting label information to a cloud server" refers to a method for uploading label information assigned by a user to a cloud server and storing or analyzing the data.

[1438] The "means for tracking the frequency of use of items on a cloud server" is a system that monitors and records the frequency of use of each item based on label information and other data stored on a cloud server.

[1439] The "means for generating decluttering advice for infrequently used items" is a method for identifying infrequently used items and generating instructions or suggestions for efficiently disposing of them.

[1440] The "means for providing information for linking with waste disposal companies and electronic trading platforms" is a system that provides users with link information with appropriate companies and platforms for disposing of or reusing items.

[1441] "Means for recognizing and analyzing user emotions" refers to technology that uses a camera or microphone to obtain emotional data from the user's facial expressions and voice, and then analyzes this data.

[1442] "Means for adjusting and providing advice based on emotional data" refers to a method for flexibly adjusting the content and format of the advice provided according to the recognized emotional state of the user and providing feedback to the user.

[1443] This invention uses a system that combines AI technology and an emotion engine to support inventory management and display optimization in physical stores. The system sends image data taken by store staff using smart glasses to a cloud server, and provides efficient advice through image and emotion analysis.

[1444] 1. System Program

[1445] The server includes the following means:

[1446] Means for acquiring image data

[1447] A means of sending image data to a cloud server

[1448] A means for analyzing image data and identifying objects on a cloud server

[1449] A means of evaluating the status of the store based on analysis results

[1450] A means of generating and providing state advice to the user

[1451] Means for providing a user interface for labeling items

[1452] A means of sending label information to the cloud server

[1453] A method for tracking item usage frequency on a cloud server

[1454] A means of generating decluttering advice for less frequently used items

[1455] A means of providing information for working with waste disposal companies and electronic trading platforms

[1456] A means of recognizing and analyzing user emotions

[1457] A means to tailor and provide advice based on sentiment data

[1458] 2. A natural language description of the program's operation

[1459] The system uses the following hardware and software:

[1460] Hardware: Smart glasses (Google Glass, Vuzix, etc.), camera, microphone

[1461] Software: AWS (or Azure) cloud services, OpenCV (image analysis), NVIDIA Emotion AI (emotion analysis)

[1462] First, a user (store staff member) uses smart glasses to take a photo of inventory shelves or display shelves. This image data is sent from the smart glasses to a cloud server. The cloud server uses image analysis software such as OpenCV to identify objects in the image and evaluate the shelf status. Based on the evaluation results, the cloud server generates specific advice regarding display and inventory and provides it to the user via the smart glasses.

[1463] The smart glasses' cameras and microphones are then used to capture the user's facial expressions and voice data. This data is then used with emotion analysis software such as NVIDIA Emotion AI to recognize the user's emotional state. The cloud server then adjusts the content of the advice provided to the user based on the recognized emotional data.

[1464] For example, if the user is tired, the system suggests simple tidying tasks, while if the user is relaxed, it provides more detailed display optimization advice. The user can follow the advice provided and use the system to receive new advice whenever the situation changes.

[1465] Using this system, store staff can manage inventory and optimize display efficiently and with minimal emotional burden.

[1466] 3. Examples of concrete examples and prompts

[1467] Example 1: During the night shift, staff use the system to organize stock shelves. Tired staff are given the advice, "Just move the items on the easy list today and be done."

[1468] Example 2: This system is used to review the overall store display before the peak season. Staff who have spare time are given detailed advice such as "Review the current display and make space to add new products."

[1469] Example prompt sentence:

[1470] input:

[1471] Take a photo of your stocked shelves today and ask for advice. Also, assess the emotional state of your staff.

[1472] output:

[1473] We've analyzed your current inventory shelves. You seem a little tired, so let's start today with a quick cleanup. First, check the leftmost column. Then, check the rightmost column and combine any duplicate items onto a single shelf. Good luck!

[1474] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1475] Step 1:

[1476] A user wears smart glasses and takes a photo of an inventory or display shelf. The camera in the smart glasses captures the image data, which is then stored on the device.

[1477] Input: A photo of a shelf in a store

[1478] Output: Image data

[1479] Step 2:

[1480] The acquired image data is transmitted from the smart glasses to a cloud server, and the image data is uploaded to the cloud server via the Internet using the network connection function of the smart glasses.

[1481] Input: Image data

[1482] Output: Image data stored on a cloud server

[1483] Step 3:

[1484] The cloud server analyzes the image data using an image analysis algorithm (e.g., using OpenCV) to identify objects in the image, and then determines the object's category and location.

[1485] Input: Image data stored on a cloud server

[1486] Output: Object category and location

[1487] Step 4:

[1488] The cloud server evaluates the store's condition based on the analysis results, and the condition is quantified based on indicators such as tidiness and inventory status.

[1489] Input: Object category and location

[1490] Output: Evaluation result (number and status)

[1491] Step 5:

[1492] The cloud server generates specific advice based on the evaluation results, including specific instructions and suggestions for tidying up the user.

[1493] Input: Evaluation result

[1494] Output: Text data of advice

[1495] Step 6:

[1496] The generated advice is sent to the smart glasses, and the user can view the advice on the display of the smart glasses.

[1497] Input: Text data of advice

[1498] Output: Advice displayed on the smart glasses display

[1499] Step 7:

[1500] When a user inputs facial expressions and voice data through the smart glasses, the data is sent to a cloud server, where the user's emotional data is acquired using the smart glasses' camera and microphone.

[1501] Input: User facial and voice data

[1502] Output: Emotion data sent to the cloud server

[1503] Step 8:

[1504] The cloud server analyzes the emotional data using an emotion analysis algorithm (e.g., using NVIDIA Emotion AI) to identify the user's emotional state.

[1505] Input: Emotion data

[1506] Output: User's emotional state (fatigue level, stress level, etc.)

[1507] Step 9:

[1508] The cloud server tailors the advice based on the user's emotional state: if the user is tired, it suggests simple tasks, and if the user is relaxed, it offers detailed organization tips.

[1509] Input: User's emotional state

[1510] Output: Text data of the adjusted advice

[1511] Step 10:

[1512] The adjusted advice is then sent back to the smart glasses, where the user can review the new advice, allowing them to efficiently manage inventory and optimize display.

[1513] Input: Text data of the adjusted advice

[1514] Output: New advice displayed on the smart glasses display

[1515] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1516] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1517] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1518] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1519] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1520] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1521] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1522] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1523] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1524] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1525] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1526] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1527] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1528] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1529] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1530] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1531] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1532] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1533] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1534] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1535] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1536] The following is further disclosed regarding the above embodiment.

[1537] (Claim 1)

[1538] means for acquiring image data;

[1539] means for transmitting the acquired image data to a cloud server;

[1540] means for analyzing image data and identifying objects in a cloud server;

[1541] A means of evaluating the room condition based on the analysis results,

[1542] means for generating and providing advice to a user regarding the state of the room;

[1543] means for providing a user interface for labeling items;

[1544] means for transmitting label information to a cloud server;

[1545] means for tracking the frequency of use of the item in a cloud server;

[1546] a means for generating decluttering advice for less frequently used items;

[1547] A means of providing information to link with waste disposal companies and flea market apps,

[1548] A system including:

[1549] (Claim 2)

[1550] The cloud server further includes means for classifying the items into garbage and providing information for linking with waste disposal companies and flea market apps based on the classification results.

[1551] 10. The system of claim 1.

[1552] (Claim 3)

[1553] The means for analyzing the acquired image data includes means for identifying object position and size information and assigning a category.

[1554] 10. The system of claim 1.

[1555] "Example 1"

[1556] (Claim 1)

[1557] means for acquiring image data;

[1558] means for transmitting the acquired image data to a cloud server;

[1559] means for analyzing image data and identifying objects in a cloud server;

[1560] means for providing a user interface for tagging identified objects;

[1561] A method for evaluating the tidiness of a room based on the analysis results, and

[1562] A means for generating advice on tidying up the room based on the evaluation result and providing the advice to the user;

[1563] means for transmitting label information to a cloud server for tracking frequency of use of the identified object;

[1564] means for tracking the frequency of use of the item in a cloud server;

[1565] A means for generating advice suggesting an efficient method for decluttering items that are rarely used;

[1566] A means of providing information on linking with waste disposal companies and marketplaces,

[1567] A system including:

[1568] (Claim 2)

[1569] The system according to claim 1, further comprising means for classifying items in the cloud server and providing information on linkage with waste disposal companies and marketplaces based on the classification results.

[1570] (Claim 3)

[1571] 10. The system of claim 1, wherein the means for analyzing the acquired image data includes means for determining object location and size information and assigning a category.

[1572] "Application Example 1"

[1573] (Claim 1)

[1574] means for acquiring image data;

[1575] means for transmitting the acquired image data to a cloud server;

[1576] means for analyzing image data and identifying objects in a cloud server;

[1577] A means of evaluating the room condition based on the analysis results,

[1578] means for generating and providing advice to a user regarding the state of the room;

[1579] means for providing a user interface for labeling items;

[1580] means for transmitting label information to a cloud server;

[1581] means for tracking the frequency of use of the item in a cloud server;

[1582] a means for generating decluttering advice for less frequently used items;

[1583] A means of providing information to work with waste disposal companies and reuse applications;

[1584] A means of assessing the tidiness of product displays and providing advice on how to improve them;

[1585] A means to identify infrequently used products and offer promotions or returns;

[1586] A system including:

[1587] (Claim 2)

[1588] The cloud server further includes means for classifying the items as garbage and providing information for linking with a waste disposal company or a recycling application based on the classification result.

[1589] 10. The system of claim 1.

[1590] (Claim 3)

[1591] The means for analyzing the acquired image data includes means for identifying object position and size information and assigning a category.

[1592] 10. The system of claim 1.

[1593] "Example 2: Combining Emotion Engines"

[1594] (Claim 1)

[1595] means for acquiring image data;

[1596] means for transmitting the acquired image data to a cloud server;

[1597] means for analyzing image data and identifying objects in a cloud server;

[1598] A means of evaluating the room condition based on the analysis results,

[1599] means for generating and providing advice to a user regarding the state of the room;

[1600] A means for acquiring user emotion data;

[1601] means for transmitting the acquired emotion data to a cloud server;

[1602] means for analyzing the emotion data and adjusting the advice in the cloud server;

[1603] means for providing a user interface for labeling items;

[1604] means for transmitting label information to a cloud server;

[1605] means for tracking the frequency of use of the item in a cloud server;

[1606] a means for generating decluttering advice for less frequently used items;

[1607] A means of providing information to work with waste disposal companies and applications;

[1608] A system including:

[1609] (Claim 2)

[1610] The cloud server further includes a means for classifying items and providing information for linking with a waste disposal company or an application based on the classification result.

[1611] 10. The system of claim 1.

[1612] (Claim 3)

[1613] The means for analyzing the acquired image data includes means for identifying object position and size information and assigning a category.

[1614] 10. The system of claim 1.

[1615] "Application example 2 when combining emotion engines"

[1616] (Claim 1)

[1617] means for acquiring image data;

[1618] means for transmitting the acquired image data to a cloud server;

[1619] means for analyzing image data and identifying objects in a cloud server;

[1620] A means of evaluating the room condition based on the analysis results,

[1621] means for generating and providing advice to a user regarding the state of the room;

[1622] means for providing a user interface for labeling items;

[1623] means for transmitting label information to a cloud server;

[1624] means for tracking the frequency of use of the item in a cloud server;

[1625] a means for generating decluttering advice for less frequently used items;

[1626] A means of providing information to link with waste disposal companies and electronic trading platforms;

[1627] means for recognizing and analyzing user emotions;

[1628] a means of providing tailored advice based on sentiment data; and

[1629] A system including:

[1630] (Claim 2)

[1631] The cloud server further includes a means for classifying the waste items and providing information for linking with a waste disposal company or an electronic trading platform based on the classification result.

[1632] 10. The system of claim 1.

[1633] (Claim 3)

[1634] The means for analyzing the acquired image data includes means for identifying object position and size information and assigning a category.

[1635] 10. The system of claim 1. [Explanation of symbols]

[1636] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring image data; means for transmitting the acquired image data to a cloud server; means for analyzing image data and identifying objects in a cloud server; A means of evaluating the room condition based on the analysis results, means for generating and providing advice to a user regarding the state of the room; means for providing a user interface for labeling items; means for transmitting label information to a cloud server; means for tracking the frequency of use of the item in a cloud server; a means for generating decluttering advice for less frequently used items; A means of providing information to link with waste disposal companies and flea market apps, A system including:

2. The cloud server further includes means for classifying the items into garbage and providing information for linking with waste disposal companies and flea market apps based on the classification results. The system of claim 1 .

3. The means for analyzing the acquired image data includes means for identifying object position and size information and assigning a category. The system of claim 1 .

Images