system

Abstract

Provide a system. [Solution] A means for a user to take an image of the home appliance or storage location; means for transmitting the image to a server; A server analyzes the image and identifies the type and number of items; means for transmitting the analysis result to a terminal; means for displaying the analysis results on a terminal screen; A means for a user to set the desired inventory quantity; means for comparing the desired inventory quantity with the current inventory quantity and listing out any items that are in short supply; A means for searching for the same or similar products of the listed items on various shopping sites; means for displaying the search results on a terminal; A means for a user to purchase an item through the search results; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, it is important to efficiently manage the inventory of home appliances and daily necessities in storage spaces. However, manually checking inventory, listing needed items, and purchasing them is time-consuming and labor-intensive. In addition, waste often occurs due to inventory shortages and overpurchases. To solve this problem, a system is needed that can automatically grasp inventory status, list items that are in short supply, and efficiently purchase them. [Means for solving the problem]

[0005] The present invention provides a system including means for a user to take images of home appliances and storage locations, means for transmitting the images to a server, means for the server to analyze the images and identify the types and quantities of items, means for transmitting the analysis results to a terminal, means for displaying the analysis results on a terminal screen, means for a user to set a desired inventory quantity, means for comparing the desired inventory quantity with the current inventory quantity and listing items that are in short supply, means for searching various shopping sites for identical or similar products to the listed items, means for displaying the search results on a terminal, and means for a user to purchase items through the search results. This allows users to efficiently manage their inventory and smoothly purchase necessary items.

[0006] "User" refers to an individual or organization that uses the system to manage inventory and purchase goods.

[0007] "Home appliances and storage spaces" refers to places in the home where daily necessities and food are stored, such as refrigerators, shelves, and closets.

[0008] "Images" refer to photographs and still images taken by the user.

[0009] "Server" refers to a computer system for receiving, analyzing, and processing data sent from a user's terminal.

[0010] "Terminal" refers to an electronic device such as a smartphone or tablet that is directly operated by the user and has input and display functions.

[0011] "Goods" refers to specific items such as home appliances and food and daily necessities stored in storage areas.

[0012] "AI object detection algorithm" refers to an artificial intelligence technology that identifies each object in an image and determines its type and location.

[0013] "Analysis results" refers to information regarding the type and number of items obtained after the server analyzes the image.

[0014] "Inventory" refers to the quantity of a particular item currently stored in an appliance or storage location.

[0015] "Desired inventory quantity" refers to the target quantity of a particular item that a user wishes to have on hand on a daily basis.

[0016] "Shortage items" refer to items that need to be purchased in addition because the current inventory does not reach the desired inventory.

[0017] "Listing" refers to listing items that are in short supply based on the analysis results and desired inventory levels.

[0018] "Shopping Site" refers to an online store where you can purchase products over the Internet.

[0019] "Search results" refers to information on identical or similar products searched for on a shopping site.

[0020] "Purchase means" refers to a method or process for a user to purchase missing items through a shopping site. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention is a system that allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items, thereby managing inventory and automatically replenishing items.

[0043] System Overview

[0044] The system includes the following main elements:

[0045] 1. Filming Method

[0046] 2. Image transmission method

[0047] 3. Image analysis methods

[0048] 4. Data transmission and reception means

[0049] 5. Inventory Display Method

[0050] 6. Inventory Setting Method

[0051] 7. How to create a list of items that are in short supply

[0052] 8. Product Search Methods

[0053] 9. Purchasing Method

[0054] Program processing

[0055] User operations

[0056] Launching the app

[0057] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[0058] photo shoot

[0059] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[0060] Inventory Settings

[0061] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[0062] Image analysis and data processing

[0063] Image transmission

[0064] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[0065] Image analysis

[0066] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[0067] Counting items

[0068] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them to the terminal.

[0069] Display inventory information and check shortages

[0070] View inventory information

[0071] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[0072] Listing shortages

[0073] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[0074] View the missing items list

[0075] The terminal displays a list of the missing items to the user. A list of missing items is generated so that the user can check the missing items.

[0076] Product search and purchase

[0077] Shopping site search

[0078] The server searches for similar products on each shopping site based on the missing items. The terminal sends the list of missing items to the server, which then searches for the products using the shopping site's API.

[0079] Sending product information

[0080] The server searches for the relevant product and sends the information to the terminal. The product information is received from the search results and displayed on the terminal.

[0081] Purchase procedure

[0082] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[0083] Specific examples

[0084] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[0085] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[0086] This system allows users to efficiently manage inventory and smoothly replenish necessary items.

[0087] The processing flow will be explained below.

[0088] Step 1: Launch the app

[0089] The user launches the app on their smartphone.

[0090] Step 2: Take a photo

[0091] The user takes a photo of the inside of the refrigerator or shelves.

[0092] The device will launch the camera function and display the capture button.

[0093] The user presses the capture button and an image is captured.

[0094] Step 3: Send image

[0095] The terminal compresses the captured image data and prepares it for transmission.

[0096] The device sends image data to the server's API endpoint.

[0097] Step 4: Image analysis

[0098] The server receives the image data.

[0099] The server uses AI object detection algorithms to identify items in the image.

[0100] The server extracts the type and location information of the identified item.

[0101] Step 5: Counting items

[0102] The server counts the number of each item.

[0103] The server structures the counting results in JSON format and sends them to the terminal.

[0104] Step 6: View inventory information

[0105] The device deserializes the data received from the server.

[0106] The device will display a list of items and their quantities on the app.

[0107] Step 7: Inventory Setup

[0108] The user enters the desired stock quantity into the app.

[0109] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[0110] Step 8: List the missing items

[0111] The terminal compares the current stock quantity with the desired stock quantity.

[0112] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[0113] Step 9: View the Missing Items List

[0114] The terminal generates a list of the missing items and displays it to the user.

[0115] Step 10: Search shopping sites

[0116] The terminal transmits the list of missing items to the server.

[0117] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[0118] Step 11: Submit your product information

[0119] The server extracts related product information from search results on the shopping site.

[0120] The server sends product information (product name, price, link, etc.) to the terminal.

[0121] Step 12: View product information and checkout

[0122] The product information received by the device is displayed on the app.

[0123] The user selects the product they want to purchase and presses the purchase button.

[0124] The device will redirect you to the shopping site's purchase page.

[0125] By following the steps above, the user can efficiently manage inventory and smoothly purchase the items they need.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] In today's modern lifestyle, inventory management of household devices and storage spaces is extremely cumbersome, and managing consumables is particularly difficult. Users must regularly check the inventory of consumables and manually purchase the necessary items, which is time-consuming. Furthermore, inventory surpluses and shortages can lead to unnecessary expenses and the inconvenience of not having the items available when needed. An efficient system that solves these problems is needed.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes a means for a user to take an image of a home appliance or storage location, a means for transmitting the image to a processing device, and a means for the processing device to analyze the image and identify the type and number of items, thereby enabling the user to efficiently manage inventory in the home and automatically replenish necessary items.

[0131] "User" refers to the individual who operates the system and manages the inventory of home devices and storage locations.

[0132] "Household appliances" refers to devices used in the home to store items, such as refrigerators and shelves.

[0133] "Image" refers to still image data taken by a user of a home device or storage location.

[0134] "Processing device" refers to a computer device used to analyze image data and identify the type and number of items.

[0135] "Terminal" refers to a digital device that a user operates at hand, and specifically includes smartphones and tablets.

[0136] "AI object detection algorithm" refers to a machine learning model or deep learning algorithm that identifies objects in an image and determines their type and number.

[0137] "Inventory" refers to the quantity of a particular item in a household.

[0138] "Desired inventory quantity" refers to the target quantity of an item that a user wishes to keep in stock.

[0139] "Shortage items" refer to items whose current inventory is lower than the desired inventory.

[0140] An "e-commerce platform" refers to a website or app that allows people to purchase goods online.

[0141] "Search Results" refers to the product information searched for based on the missing items on the e-commerce platform.

[0142] This invention is a system for managing inventory and automatically replenishing items in the home. The system allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items and manage inventory information.

[0143] First, the user launches a dedicated app on a device such as a smartphone or tablet. The user selects the inventory management function within the app and activates the camera. The user then takes pictures of storage areas in the home, such as refrigerators and shelves. The device compresses the captured image data and sends it to a server via the Internet.

[0144] The server analyzes the received image data. Specifically, it uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image and determine their type and number. The analysis results are structured in JSON format as data including the type and number of objects, and sent to the device. This analysis is often performed by a server equipped with a high-performance GPU.

[0145] Next, the device deserializes the data received from the server and displays it as a list to the user. The user checks this list and, if necessary, enters the desired stock quantity into the app. The app saves the desired stock quantity entered by the user and compares it with the current stock quantity to calculate the shortage of items. The calculation results are also displayed as a list to the user.

[0146] Furthermore, the terminal sends a list of missing items to the server, and the server uses the API of the e-commerce platform to search for the relevant items. The search results are sent to the terminal as specific product information and displayed for the user to review. This allows the user to easily purchase the missing items. This function allows users to efficiently manage inventory and streamline purchasing procedures.

[0147] Examples:

[0148] For example, if a user notices that they are low on milk in the refrigerator while preparing breakfast, they can launch the app and take a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm identifies the type and quantity of each item (e.g., milk, eggs, butter, etc.) in the refrigerator.

[0149] The analysis reveals that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system would automatically list the missing bottle of milk. This list would be displayed on the terminal for the user to review. The server would then use the API of the e-commerce platform to search for available products of the missing milk and send the results to the terminal. The user could then select from the list and easily complete the purchase process.

[0150] The system allows users to efficiently manage their home inventory and automatically replenish items as needed.

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Step 1:

[0153] The user launches the app on their smartphone. They select the inventory management function from the app's main screen and move to the image capture screen. At this time, the device's camera function is activated and a capture button is displayed.

[0154] Input: User actions

[0155] Output: Activating the camera function and displaying the shooting screen

[0156] Step 2:

[0157] When the user takes a photo of a home device or storage location, the device acquires the image data and temporarily stores it in its internal memory.

[0158] Input: An image taken by the user

[0159] Output: Image data stored in the device's memory

[0160] Step 3:

[0161] The device compresses the captured image data and sends it to the server's API endpoint, where it compresses the image data into JPEG format and sends it to the server via the Internet.

[0162] Input: Image data stored in memory

[0163] Output: Compressed image data sent to the server

[0164] Step 4:

[0165] The server analyzes the received image data. It uses an AI object detection algorithm (e.g., YOLO, SSD) to identify the objects in the image. The server determines the type and number of objects and structures the results in JSON format.

[0166] Input: Compressed image data

[0167] Output: Parsed results in JSON format (e.g., {"Milk": 2, "Egg": 8})

[0168] Step 5:

[0169] The server sends the parsed results in JSON format to the device, which receives them and stores them as internal data.

[0170] Input: Analysis results sent from the server

[0171] Output: Analysis results stored on the device

[0172] Step 6:

[0173] The device deserializes the analysis results and displays them to the user as a list, showing the type and quantity of each item.

[0174] Input: Parsed result in JSON format

[0175] Output: Inventory information displayed to the user (e.g., Milk: 2 bottles, Eggs: 8)

[0176] Step 7:

[0177] The user inputs the desired stock quantity into the app. The device displays the stock setting screen, and when the user inputs the desired stock quantity for each item, it is saved in the app.

[0178] Input: The desired stock quantity entered by the user

[0179] Output: Desired stock quantity saved in the app

[0180] Step 8:

[0181] The terminal compares the current stock with the stock desired by the user and lists the items that are in short supply. As a result of the calculation, a list of items that are in short supply is generated.

[0182] Input: Current stock quantity and desired stock quantity

[0183] Output: List of items in short supply (e.g. Milk: 1 bottle missing)

[0184] Step 9:

[0185] The terminal sends a list of missing items to the server, which then searches for the relevant items on the e-commerce platform and retrieves product information using the API of each platform.

[0186] Input: List of missing items

[0187] Output: Product information obtained from the e-commerce platform

[0188] Step 10:

[0189] The server sends the acquired product information to the terminal, which then displays it to the user, who can then check the displayed product information.

[0190] Input: Product information obtained from the e-commerce platform

[0191] Output: Product information displayed to the user

[0192] Step 11:

[0193] The user completes the purchase process. When the user presses the purchase button, the terminal redirects them to the purchase page of the e-commerce platform to complete the purchase process.

[0194] Input: User purchase operation

[0195] Output: Purchase procedure completed

[0196] (Application example 1)

[0197] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0198] In modern brick-and-mortar stores, inventory management is a time-consuming and labor-intensive task. Accurately tracking inventory levels on shelves and display areas and replenishing them in a timely manner are particularly important and directly affect customer satisfaction. However, traditional manual inventory checks are inefficient and carry a high risk of errors. Therefore, a more efficient and accurate method of inventory management is needed.

[0199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0200] In this invention, the server includes means for a user to take images of home appliances, storage locations, shelves, and display corners in a store, means for transmitting the images to the server, means for the server to analyze the images and identify the type and number of items, and means for transmitting the analysis results to a terminal. This enables efficient and accurate inventory management in a physical store and enables a quick listing and notification of products that need to be replenished.

[0201] "Home appliances" is a general term for electrical and electronic devices used in the home.

[0202] "Storage area" refers to a space or container for storing items.

[0203] A "server" is a computer system that stores data and provides services to clients on a network.

[0204] "Image analysis" is the process of computer-based analysis of digital images to identify objects and features within the images.

[0205] "Goods" is a general term that refers to tangible goods or merchandise.

[0206] A "species" is a taxonomic group with different characteristics or properties.

[0207] "Quantity" indicates the quantity of an item.

[0208] A "terminal" is a computer device or smart device that a user operates.

[0209] "Inventory" means the total quantity of items currently stored or on display.

[0210] "Listing" is the act of displaying a list of items based on specific criteria.

[0211] A "shopping site" is a website where you can purchase goods and services over the Internet.

[0212] A "shelf" is a structure for displaying and storing items.

[0213] A "display corner" is an area within a store where items are displayed and sold.

[0214] An "AI object detection algorithm" is a computational method for identifying objects in an image using artificial intelligence technology.

[0215] "Management" means organizing and operating things according to a purpose.

[0216] "Replenishing" is the act of adding items that are lacking.

[0217] This invention relates to a system for streamlining store inventory management. This system uses image analysis technology and AI object detection algorithms to grasp the inventory status of items on shelves and display corners in a store and provides users with a list of items that are in short supply.

[0218] System configuration

[0219] 1. Photography Method:

[0220] The user takes an image of the shelf or display area using the camera on their smartphone. There are no particular restrictions on the smartphone as long as it has a general camera function.

[0221] 2. Image transmission method:

[0222] The image is sent to the server. The smartphone sends the captured image data to the server via an Internet connection. An HTTP request library (e.g., requests) is used.

[0223] 3. Image analysis methods:

[0224] The server analyzes the images using an AI object detection algorithm to identify the type and number of items, preferably with advanced GPU capabilities to run specific object detection models (e.g., YOLO, SSD).

[0225] 4. Data transmission and reception means:

[0226] The server sends the analysis results to the smartphone, which receives them. The analyzed product information is sent and received using JSON format data.

[0227] 5. Inventory Display Method:

[0228] The smartphone displays the analysis results on its screen. The user can visualize the inventory status of the items through the application. A common front-end framework (e.g., React Native, Flutter (registered trademark)) is used to provide the user interface (UI).

[0229] 6. Inventory setting method:

[0230] The user sets the desired stock quantity, which is stored in the application and used as a basis for comparison.

[0231] 7. Listing Methods:

[0232] The smartphone compares the desired inventory quantity with the current inventory quantity and creates a list of items that are in short supply based on the inventory data received from the server.

[0233] 8. Search Methods:

[0234] The smartphone sends the entire list of missing items to a server, which then searches for the items via the API of the shopping site.

[0235] 9. Purchasing Method:

[0236] The user purchases an item through the search results, and the smartphone redirects the user to a purchase page on the shopping site.

[0237] Hardware and software used

[0238] Hardware:

[0239] Smartphone (camera function and internet connection)

[0240] Server (with GPU functionality)

[0241] software:

[0242] Camera API

[0243] HTTP request library (e.g., requests)

[0244] AI object detection models (e.g., YOLO, SSD)

[0245] JSON Data Format

[0246] User interface frameworks (e.g., React Native, Flutter)

[0247] Shopping site API

[0248] Specific examples

[0249] For example, consider a situation where a store clerk uses a smartphone app to check the inventory in the candy section. When the app takes a photo of the shelves, the AI ​​object detection algorithm automatically recognizes the type and quantity of each item on the shelves and lists any items that are missing. Based on the analysis results, the server searches for identical or similar products on the shopping site and presents the search results to the user. This allows the clerk to efficiently manage inventory and quickly replenish products as needed.

[0250] Prompt Sentence Examples

[0251] A store inventory management app is used to check the stock in the snack section. When a store associate takes a photo of the shelf with the smartphone app, the app automatically recognizes the type and quantity of each item and lists any items that are low in stock. Specifically, if it finds that potato chips are out of stock, the app notifies the associate. Please describe in detail a situation in which the associate can efficiently check the stock based on this prompt.

[0252] The above is a specific embodiment for carrying out the present invention.

[0253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0254] Step 1:

[0255] The user takes pictures of home appliances, storage areas, shelves, and display corners in a store.

[0256] Users use the camera function of their smartphone to take photos of home appliances, storage areas, or shelves and display areas in stores. These images are used as visual data for the items to be managed in inventory.

[0257] Input: Image taken with a smartphone camera

[0258] Output: Digital image data

[0259] Step 2:

[0260] The terminal transmits the image to the server.

[0261] The device uses an HTTP request over its internet connection to send the captured image data to the server, where it is compressed and encoded as appropriate and sent to the server's API.

[0262] Input: Digital image data

[0263] Output: Image data sent to the server

[0264] Step 3:

[0265] The server analyzes the image and identifies the type and number of items.

[0266] The server analyzes the received image data using an AI object detection algorithm (e.g., YOLO, SSD), which identifies the type and number of each item in the image.

[0267] Input: Image data sent to the server

[0268] Output: Analysis results regarding the type and number of items

[0269] Step 4:

[0270] The server sends the analysis results to the terminal.

[0271] The server structures the analysis results as JSON data and sends it to the terminal via an HTTP request. This JSON data includes the type and quantity of each item.

[0272] Input: Analysis results regarding type and number of items

[0273] Output: JSON data of the analysis results sent to the terminal

[0274] Step 5:

[0275] The terminal displays the analysis results on the screen.

[0276] The terminal deserializes the JSON data received from the server and displays the inventory status of the items on the screen through a user interface, where the user can check the type and quantity of each item.

[0277] Input: JSON data of analysis results

[0278] Output: Inventory information displayed on a smartphone screen

[0279] Step 6:

[0280] Set the stock quantity desired by the user

[0281] The user enters the desired inventory quantity in the application, for example, 12 eggs or 3 bottles of milk, and this information becomes the baseline data.

[0282] Input: The number of items in stock desired by the user

[0283] Output: The desired stock quantity stored in the application

[0284] Step 7:

[0285] The terminal compares the desired inventory with the current inventory and lists the items that are in short supply.

[0286] The terminal compares the current inventory data with the desired inventory quantity set by the user, calculates the shortage, and lists it. The list of shortage items is used in the next step.

[0287] Input: Desired stock quantity and current stock quantity

[0288] Output: List of missing items

[0289] Step 8:

[0290] The server searches various shopping sites for the same or similar products of the listed items.

[0291] The server uses the API of each shopping site to search for the same or similar products based on the list of missing items, and the search results are sent to the terminal.

[0292] Input: List of missing items

[0293] Output: Search results from a shopping site

[0294] Step 9:

[0295] The terminal displays the search results on the screen.

[0296] The terminal displays the search results received from the server, and the user can use them as a reference to purchase the necessary items.

[0297] Input: Search results sent from the server

[0298] Output: Search results displayed on a smartphone screen

[0299] Step 10:

[0300] The user purchases an item through the search result.

[0301] The user can then proceed to purchase the selected item within the application, and the device will redirect the user to the shopping site's purchase page.

[0302] Input: Select a user based on search results

[0303] Output: Purchase completed

[0304] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0305] The present invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchase suggestions.

[0306] System Overview

[0307] The system includes the following main elements:

[0308] 1. Filming Method

[0309] 2. Image transmission method

[0310] 3. Image analysis methods

[0311] 4. Data transmission and reception means

[0312] 5. Inventory Display Method

[0313] 6. Inventory Setting Method

[0314] 7. How to create a list of items that are in short supply

[0315] 8. Product Search Methods

[0316] 9. Purchasing Method

[0317] 10. Emotion Engine

[0318] Program processing

[0319] User operations

[0320] Launching the app

[0321] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[0322] photo shoot

[0323] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[0324] Inventory Settings

[0325] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[0326] Emotion input

[0327] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. Emotional data is collected by the user's voice input or facial expressions displayed through the camera. This data is used for inventory management and purchasing suggestions.

[0328] Image analysis and data processing

[0329] Image transmission

[0330] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[0331] Image analysis

[0332] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[0333] Counting items

[0334] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them.

[0335] Display inventory information and check shortages

[0336] View inventory information

[0337] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[0338] Listing shortages

[0339] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[0340] View the missing items list

[0341] The terminal generates a list of the missing items and displays it to the user.

[0342] Product search and purchase

[0343] Shopping site search

[0344] The terminal sends a list of missing items to the server, which then uses the APIs of various shopping sites (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[0345] Sending product information

[0346] The server extracts related product information from search results on the shopping site, and sends the product information (product name, price, link, etc.) to the device.

[0347] Purchase procedure

[0348] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[0349] Use of emotion engine

[0350] Emotional Data Analysis and Applications

[0351] The emotion engine analyzes the user's voice and facial expression data in real time to recognize their emotions. This emotion data is then applied to inventory management and purchasing suggestions. For example, if the user is feeling stressed, it will suggest products that will help them relax.

[0352] Recording frequency of use and emotions

[0353] The emotion engine accumulates and records the user's emotional data and predicts their long-term consumption trends, enabling optimal product recommendations based on the user's preferences and tendencies.

[0354] Specific examples

[0355] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[0356] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[0357] Also, if a user is feeling stressed during busy morning hours, the emotion engine will recognize this and suggest products that will help reduce stress (such as relaxing herbal tea). In this way, appropriate product suggestions are made based on the user's emotions.

[0358] This system not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows them to receive optimal product suggestions based on their emotions.

[0359] The processing flow will be explained below.

[0360] Step 1: Launch the app

[0361] The user launches the app on their smartphone.

[0362] Step 2: Take a photo

[0363] The user takes a photo of the inside of the refrigerator or shelves.

[0364] The device will launch the camera function and display the capture button.

[0365] The user presses the capture button and an image is captured.

[0366] Step 3: Send image

[0367] The terminal compresses the captured image data and prepares it for transmission.

[0368] The device sends image data to the server's API endpoint.

[0369] Step 4: Image analysis

[0370] The server receives the image data.

[0371] The server uses AI object detection algorithms to identify items in the image.

[0372] The server extracts the type and location information of the identified item.

[0373] Step 5: Counting items

[0374] The server counts the number of each item.

[0375] The server structures the counting results in JSON format and sends them to the terminal.

[0376] Step 6: View inventory information

[0377] The device deserializes the data received from the server.

[0378] The device will display a list of items and their quantities on the app.

[0379] Step 7: Inventory Setup

[0380] The user enters the desired stock quantity into the app.

[0381] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[0382] Step 8: Enter emotions

[0383] The user displays facial expressions through voice input or a camera.

[0384] The device sends voice data and facial expression images to the emotion engine.

[0385] Step 9: Sentiment Analysis

[0386] The server's emotion engine analyzes voice data and facial expression images in real time to recognize the user's emotions.

[0387] The server stores the recognized emotion data and uses it for subsequent inventory management and purchasing suggestions.

[0388] Step 10: List the missing items

[0389] The terminal compares the current stock quantity with the desired stock quantity.

[0390] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[0391] Step 11: View the Shortage List

[0392] The terminal generates a list of the missing items and displays it to the user.

[0393] Step 12: Search shopping sites

[0394] The terminal transmits the list of missing items to the server.

[0395] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[0396] Step 13: Submit your product information

[0397] The server extracts related product information from search results on the shopping site.

[0398] The server sends product information (product name, price, link, etc.) to the terminal.

[0399] Step 14: View product information and checkout

[0400] The product information received by the device is displayed on the app.

[0401] The user selects the product they want to purchase and presses the purchase button.

[0402] The device will redirect you to the shopping site's purchase page.

[0403] Step 15: Emotion-based suggestions

[0404] Based on the user's emotional data recognized by the emotion engine, the system suggests products that are best suited to the user's current emotional state, such as relaxing drinks or health foods.

[0405] The terminal displays emotion-based product suggestions to the user.

[0406] Step 16: Analyze long-term consumption trends

[0407] The emotion engine accumulates and records users' emotional data and analyzes their long-term consumption trends.

[0408] The server optimizes weekly or monthly purchase suggestions based on the analysis results.

[0409] In this way, the user can efficiently manage inventory, smoothly purchase the items he or she needs, and furthermore, receive appropriate product suggestions according to the user's emotional state.

[0410] Example 2

[0411] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0412] Inventory management and replenishment of home appliances and storage spaces are important issues for modern households. In particular, there is a need for a system that quickly and accurately replenishes items when they run out, especially in busy daily lives. However, conventional inventory management systems rely heavily on manual input by users, making efficient management difficult. Furthermore, they lack functionality for optimal product recommendations based on users' emotions and preferences. This can lead to inconvenient situations where necessary items cannot be properly replenished. Furthermore, it is difficult to provide services that respond to users' individual needs and emotional changes.

[0413] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0414] In this invention, the server includes means for a user to take an image of the home appliance or storage location, means for transmitting the image to the server, means for the server to analyze the image and identify the type and number of items, means for transmitting the analysis results to a terminal, means for displaying the analysis results on a terminal screen, means for a user to set a desired inventory quantity, means for comparing the desired inventory quantity with the current inventory quantity and listing items that are in short supply, means for searching various e-commerce sites for products that are identical to or similar to the listed items, means for displaying the search results on the terminal, means for analyzing user emotion data and applying the emotion data to inventory management and purchase suggestions, and means for a user to purchase items based on the search results. This enables efficient inventory management and rapid replenishment of items, and also realizes optimal product suggestions based on the user's emotions and preferences.

[0415] "User" refers to a person who operates the system, takes pictures of home appliances and storage locations, and issues various instructions regarding inventory management and purchasing of goods.

[0416] "Home appliances" refers to electrical appliances used to store household items, such as refrigerators and shelves.

[0417] "Storage space" refers to a place or facility for storing items, such as the inside of a refrigerator or a shelf.

[0418] "Images" refers to photos or videos taken by the user of the inside of an appliance or storage space.

[0419] The term "server" refers to a computer system that receives and analyzes image data sent by a user.

[0420] "Goods" refers to goods or products stored in appliances or storage areas, including food items such as milk and eggs.

[0421] "Quantity" refers to the quantity of a particular item.

[0422] "Terminal" refers to a mobile device such as a smartphone or tablet used by a user.

[0423] "Screen" refers to the display shown on the terminal.

[0424] "Inventory" refers to the quantity of items currently in stock.

[0425] "Desired inventory quantity" refers to the quantity of items desired by the user to be stored.

[0426] "Listing" refers to comparing the current inventory with the desired inventory and listing items that are in short supply.

[0427] "E-commerce site" refers to a website for buying and selling products online, including, for example, Amazon, Yahoo, and Rakuten.

[0428] "Emotion data" refers to data related to emotions collected from the user's voice and facial expressions.

[0429] "Purchase method" refers to the system, such as online payment or shopping cart, that a user uses to purchase goods.

[0430] "Machine learning object detection algorithm" refers to a machine learning or artificial intelligence algorithm used to automatically identify objects in images, such as YOLO or SSD.

[0431] This invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchasing suggestions.

[0432] The system includes the following main elements:

[0433] 1. A way for users to take pictures of home appliances and storage locations

[0434] 2. Means of sending the captured image to the server

[0435] 3. Means by which the server analyzes the image and identifies the type and number of items

[0436] 4. Means of sending analysis results to the device

[0437] 5. How to display the analysis results on the device screen

[0438] 6. A way for users to set their desired stock quantity

[0439] 7. A method for comparing desired inventory with current inventory and listing items that are in short supply

[0440] 8. A means to search for the same or similar products of the listed items on various e-commerce sites.

[0441] 9. How to display search results on your device

[0442] 10. Means of analyzing user sentiment data and applying sentiment data to inventory management and purchase recommendations

[0443] 11. How users can purchase items through search results

[0444] Users launch the app on their smartphone or other device and take a photo of the contents of their refrigerator or shelves. The device compresses the image and sends it to the server. The server then uses an AI object detection algorithm (e.g., YOLO or SSD) to identify the type and number of items in the image and sends the analysis results back to the device.

[0445] Users can check the analysis results on the device screen and set the desired inventory quantity as needed. The device compares the current inventory quantity with the desired inventory quantity and lists the items that are in short supply. For the listed items, the server uses the APIs of various e-commerce sites (e.g., Amazon, Yahoo, Rakuten) to search for similar products and send the search results to the device.

[0446] The emotion engine analyzes the user's voice and facial expressions to collect emotional data. This emotional data is then applied to inventory management and purchase suggestions. For example, if a user is feeling stressed, it will suggest products that have a relaxing effect. The user can select the suggested products on the screen and proceed with the purchase.

[0447] As a concrete example, suppose a user is preparing breakfast and notices that there is little milk in the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm is used to identify the type and number of bottles of milk. The analysis results show that there are only two bottles of milk left in the refrigerator. Since the user's desired stock is three bottles, the system lists the one bottle that is missing.

[0448] The server uses the API of the e-commerce site to search for the missing item and display it to the user. Furthermore, the emotion engine recognizes that the user is feeling stressed while preparing breakfast and suggests relaxing herbal teas.

[0449] The user can select the appropriate product from the options presented and quickly complete the purchase procedure.

[0450] Example prompts to input to the generative AI model

[0451] 1. Inventory count prompt:

[0452] "When the user takes a photo of the refrigerator, count the number of bottles of milk using YOLO and return the result in JSON format."

[0453] 2. Prompt to generate a list of missing items:

[0454] "Compare the current stock quantity data with the user's desired stock quantity data and generate a list of items that are in short supply."

[0455] 3. Emotion-Based Product Recommendation Prompt:

[0456] "Analyze the collected user emotional data and suggest products that have a relaxing effect when the user is feeling stressed."

[0457] As a result, a system is realized that not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows users to receive optimal product suggestions based on their emotions.

[0458] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0459] Explain the process step by step

[0460] User operations

[0461] Step 1: Launch the app

[0462] A user launches an app on their smartphone. The input is a touch operation, and the output is the main screen of the app.

[0463] The device loads configuration files and user data in the background and prepares to operate normally.

[0464] Step 2: Take a photo

[0465] The user takes a photo of the inside of the refrigerator or shelf. The device's camera function is used as input, and image data is generated as output.

[0466] The device activates the camera function, and when the user presses the capture button, a photo is taken. This image data is temporarily stored in the device's memory.

[0467] Step 3: Inventory Setup

[0468] The user manually inputs the desired inventory quantity into the app, and the desired inventory data is generated as output.

[0469] The terminal displays an inventory setting screen, and the user inputs the items they need (e.g., 3 bottles of milk, 12 eggs). This information is stored in the terminal's internal database.

[0470] Step 4: Enter emotions

[0471] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. The user's voice and facial expression data are used as input, and emotion data is generated as output.

[0472] The device collects and temporarily stores the user's facial expressions through voice input and a camera, and this data is later sent to a server.

[0473] Image analysis and data processing

[0474] Step 5: Send image

[0475] The terminal sends the captured image to the server. The input is image data, and the output is compressed image data sent to the server.

[0476] The device compresses the image data and sends it to the server's API endpoint.

[0477] Step 6: Image analysis

[0478] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The input is image data, and the output is data on the type and number of objects.

[0479] The server receives the image data and runs an object detection algorithm to extract the type and location of the item, and stores the results in a database.

[0480] Step 7: Counting items

[0481] The server counts the number of each item and sends the result to the terminal. The input is the identified item data, and the output is the counted item data.

[0482] The server categorizes the items, counts the number of each item, and structures the results in JSON format. It then checks the data for integrity before sending it to the device.

[0483] Display inventory information and check shortages

[0484] Step 8: View inventory information

[0485] The terminal displays a list of items and their quantities. The input is the item count data, and the output is the inventory list displayed on the screen.

[0486] The terminal deserializes the received data and displays it in a list format on the screen, allowing the user to check the current inventory status.

[0487] Step 9: List the missing items

[0488] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists the items that are in short supply. The input is the desired inventory data and the current inventory data, and the output is a list of items that are in short supply.

[0489] The terminal compares the two and calculates the shortage. For example, if the desired stock is 3 bottles of milk and the current stock is 2 bottles, the shortage will be 1 bottle.

[0490] Step 10: View the Missing Items List

[0491] The terminal displays the generated shortage list. The input is the shortage list data, and the shortage list is displayed on the screen as the output.

[0492] The shortage list is displayed in a visually easy-to-understand format, allowing users to quickly take next steps.

[0493] Product search and purchase

[0494] Step 11: Search for shopping sites

[0495] The terminal sends the list of items in short supply to the server. The input is the list of items in short supply, and the output is the search results for similar items.

[0496] The server uses the APIs of various e-commerce sites to search for products that correspond to the missing items and sends the results to the terminal.

[0497] Step 12: Submit your product information

[0498] The server sends related product information from search results on a shopping site to the terminal. The input is search result data, and the output is product information.

[0499] Data including product information (product name, price, link, etc.) is sent to the terminal. The terminal analyzes the received data and displays it to the user in an appropriate format.

[0500] Step 13: Checkout

[0501] The user selects a product on the app and completes the purchase process. The input is product selection data, and the output is purchase confirmation data.

[0502] The terminal displays the received product information, and when the user presses the purchase button, the terminal redirects the user to the shopping site's purchase page. Once the purchase procedure is complete, the terminal displays a confirmation message.

[0503] Use of emotion engine

[0504] Step 14: Analyze and apply emotion data

[0505] The emotion engine analyzes the user's voice and facial expression data in real time to recognize emotions. The input is voice and facial expression data, and the output is emotional data.

[0506] For example, if a user is feeling stressed, the system will suggest products that will help them relax (e.g., herbal tea), which will then be applied to inventory management and purchasing suggestions.

[0507] Step 15: Record frequency of use and emotions

[0508] The emotion engine accumulates and records the user's emotional data and predicts the user's long-term consumption trends. The input is the accumulated emotional data, and the output is predicted consumption trends.

[0509] For example, it is possible to identify the time periods during which a user is likely to feel stressed and suggest products suitable for those time periods.

[0510] Through these steps, users can efficiently manage their inventory, quickly replenish needed items, and receive optimal product recommendations based on their emotions.

[0511] (Application example 2)

[0512] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0513] In recent years, there has been a demand for improved cargo management efficiency in autonomous vehicles. However, with conventional inventory management systems, drivers must manually check inventory levels, identify missing items, and then carry out replenishment procedures, which requires time and effort. Furthermore, simply indicating missing items without considering the driver's feelings increases stress and fatigue for the driver, resulting in a decrease in work efficiency. A system that solves these issues, performs efficient inventory management, and reduces the burden on drivers is needed.

[0514] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for the user to take an image of the storage location, a means for transmitting the image to the server, and a means for the server to analyze the image and identify the type and number of items. This allows the driver to easily check the inventory status of the cargo space using the wearable device, as well as identify missing items in real time and automatically receive replenishment suggestions. In addition, the system includes a means for recognizing the driver's emotions and a means for suggesting appropriate products based on the recognized emotion data, thereby reducing the driver's stress and allowing them to work efficiently and comfortably.

[0515] "Means for taking images" refers to a function that allows a user to take images of a storage area or cargo space using a smart device or wearable device.

[0516] The "means for transmitting images to a server" is a function for transmitting captured image data to a server via a network such as the Internet.

[0517] "Means for identifying the type and number of items" refers to a function that analyzes the image received by the server and identifies the type and number of items in the image using an AI object detection algorithm.

[0518] "Means for transmitting analysis results to a terminal" refers to a function for transmitting data analyzed by the server to a user's terminal, particularly a smart device or wearable device.

[0519] The "means for displaying the analysis results on the display device of the terminal" is a function for visually displaying the analysis results on the screen of the user's terminal.

[0520] The "means for setting the stock quantity desired by the user" is a function that allows the user to input the stock quantity desired by the user and record it in the system.

[0521] The "means for listing items in short supply" is a function that compares the set desired inventory quantity with the current inventory quantity and displays the items in short supply in a list format.

[0522] "Means for searching for identical or similar products in various shopping systems" refers to a function for searching online shopping sites for products that are identical or similar to the missing item.

[0523] The "means for displaying search results on a terminal" is a function for displaying product search results obtained from a shopping site on a user's terminal.

[0524] The "means for purchasing goods" is a function for a user to complete the purchase procedure for the product selected by the user through the terminal.

[0525] "Means for managing items in the cargo space, including a wearable device" refers to a function that uses a wearable device worn by the driver to manage inventory in the cargo space.

[0526] The "means for recognizing emotions" is a function that analyzes the user's facial expressions and tone of voice through the wearable device's camera or voice input, and recognizes their emotions.

[0527] The "means for suggesting appropriate products based on recognized emotional data" is a function that suggests products that have stress-reducing or relaxing effects based on the user's emotional data.

[0528] The "means for displaying the proposed results" is a function for displaying the products and information proposed to the user on the terminal.

[0529] This invention relates to a system for improving the efficiency of cargo management in autonomous vehicles. Specifically, the system takes images of the cargo space, identifies the type and number of items using an AI object detection algorithm, and performs inventory management while recognizing the driver's emotions to suggest optimal products. This system is intended to be operated by the user using a wearable device.

[0530] Hardware Configuration

[0531] 1. Wearable devices (e.g., smart glasses)

[0532] It is worn by the driver and takes pictures of the cargo space.

[0533] It has a built-in camera and microphone to collect image and audio data.

[0534] 2. Cloud Server

[0535] Performs image analysis, object detection, and emotion recognition

[0536] Responsible for storing and processing data

[0537] 3. Smart Devices

[0538] Driver-carried smartphones and tablets

[0539] Communicates with the server and displays analysis results and product suggestions

[0540] Software Configuration

[0541] 1. AI object detection algorithms (e.g., YOLO)

[0542] Analyze images captured by the wearable device to identify the type and quantity of cargo

[0543] 2. Sentiment analysis algorithms (e.g., EmotionAnalyzer)

[0544] Analyzes emotions based on the driver's facial expressions and voice data captured through cameras and microphones

[0545] 3. Data transmission and reception module

[0546] It serves as a data transfer mechanism between wearable devices and cloud servers.

[0547] 4. Inventory Management Module

[0548] Manage inventory data on the server

[0549] 5. Proposal Module

[0550] Recommend products to drivers based on emotion data

[0551] Detailed process flow

[0552] 1. Image capture and transmission

[0553] A user takes an image of the cargo space using a wearable device

[0554] The captured image is sent to the server

[0555] 2. Image Analysis

[0556] The server uses AI object detection algorithms to identify objects in the image and determine their type and quantity.

[0557] 3. Inventory Management

[0558] The user inputs the desired inventory quantity into the smart device, and the server compares it with the current inventory to identify any shortages.

[0559] 4. Emotion Recognition and Product Recommendations

[0560] Using the camera and microphone of the wearable device, the driver's facial expressions and voice data are analyzed with an emotion analysis algorithm to recognize their emotions.

[0561] Makes product recommendations based on emotions and displays the results on a smart device

[0562] Specific examples

[0563] While replenishing cargo, the driver uses smart glasses to take images of the cargo space. The images are sent to a server, where an AI object detection algorithm analyzes inventory shortages. At the same time, if the driver's stress level is high, suggestions for relaxation products (e.g., herbal tea) are displayed.

[0564] Prompt Sentence Examples

[0565] "Check stock availability"

[0566] "Please suggest a relaxing product."

[0567] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0568] Step 1:

[0569] A user takes an image of the cargo space using a wearable device (smart glasses).

[0570] Input: Cargo space image

[0571] Output: Captured image data

[0572] Specific operation: Using the camera function of the smart glasses, the user faces the cargo space and presses the capture button to capture an image.

[0573] Step 2:

[0574] The device sends the captured image to a cloud server.

[0575] Input: Photographed image data

[0576] Output: Image data sent to the server

[0577] Specific operation: The transmission module on the smart glasses is activated and sends the image data to the server via the Internet.

[0578] Step 3:

[0579] The server uses an AI object detection algorithm to analyze the image and identify the type and number of items.

[0580] Input: Image data sent to the server

[0581] Output: Data on type and number of items

[0582] Specific operation: The server runs image analysis software (e.g., YOLO) to identify and count the items, and stores the results in a database.

[0583] Step 4:

[0584] The server sends the analysis results to the user's smart device.

[0585] Input: Data on type and quantity of items

[0586] Output: Analysis data sent to smart device

[0587] Specific operation: The server sends the generated item list to the user's smartphone or tablet via the data communication module.

[0588] Step 5:

[0589] The terminal displays the analysis results on a display device.

[0590] Input: Submitted analysis data

[0591] Output: Inventory information displayed on screen

[0592] Specific operation: The display module of the smart device displays the analysis results in list format on the screen so that the user can check them.

[0593] Step 6:

[0594] The user sets the desired stock quantity via a smart device.

[0595] Input: The desired stock quantity entered by the user

[0596] Output: Desired inventory quantity data

[0597] Specific operation: The user enters the desired stock quantity of each item on the application screen of their smart device, and the data is saved within the app.

[0598] Step 7:

[0599] The server compares the current stock with the desired stock and lists the items that are in short supply.

[0600] Input: Current stock quantity data and desired stock quantity data

[0601] Output: List of missing items

[0602] Specific operation: The server compares the stored inventory data with the desired inventory data, calculates the difference, and generates a list of items that are in short supply.

[0603] Step 8:

[0604] The server searches various shopping systems for the same or similar products as the missing item.

[0605] Input: List of missing items

[0606] Output: Shopping system search results

[0607] Specific operation: The server uses the API of each shopping system (e.g., Amazon, Rakuten) to search for products similar to the missing item.

[0608] Step 9:

[0609] The server sends the search results to the terminal.

[0610] Input: Shopping system search results

[0611] Output: Search results sent to your device

[0612] Specific operation: The server organizes the search results and sends them to the user's smart device.

[0613] Step 10:

[0614] The terminal displays the search results to the user, and the user purchases the item.

[0615] Input: Submitted search results

[0616] Output: User completes purchase

[0617] Specific operation: The user selects a product from the search results displayed through the smart device application and presses the purchase button to complete the purchase process.

[0618] Step 11:

[0619] The wearable device on the terminal recognizes the driver's emotions and transmits the emotional data to a server.

[0620] Input: Driver's facial expressions and voice data

[0621] Output: Emotion data sent to the server

[0622] Specific operation: Emotion data collected through the camera and microphone of the wearable device is analyzed and sent to the server.

[0623] Step 12:

[0624] The server suggests appropriate products based on the recognized emotion data and displays them on the device.

[0625] Input: Emotion data

[0626] Output: Appropriate product suggestions

[0627] How it works: The server's emotion analysis algorithm (e.g., EmotionAnalyzer) analyzes the emotion data and generates possible product suggestions. These suggestions are sent to the smart device and displayed to the user.

[0628] The above are the specific processing steps.

[0629] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0630] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0631] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0632] [Second embodiment]

[0633] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0634] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0635] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0636] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0637] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0638] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0639] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0640] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0641] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0642] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0643] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0644] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0645] This invention is a system that allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items, thereby managing inventory and automatically replenishing items.

[0646] System Overview

[0647] The system includes the following main elements:

[0648] 1. Filming Method

[0649] 2. Image transmission method

[0650] 3. Image analysis methods

[0651] 4. Data transmission and reception means

[0652] 5. Inventory Display Method

[0653] 6. Inventory Setting Method

[0654] 7. How to create a list of items that are in short supply

[0655] 8. Product Search Methods

[0656] 9. Purchasing Method

[0657] Program processing

[0658] User operations

[0659] Launching the app

[0660] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[0661] photo shoot

[0662] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[0663] Inventory Settings

[0664] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[0665] Image analysis and data processing

[0666] Image transmission

[0667] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[0668] Image analysis

[0669] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[0670] Counting items

[0671] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them to the terminal.

[0672] Display inventory information and check shortages

[0673] View inventory information

[0674] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[0675] Listing shortages

[0676] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[0677] View the missing items list

[0678] The terminal displays a list of the missing items to the user. A list of missing items is generated so that the user can check the missing items.

[0679] Product search and purchase

[0680] Shopping site search

[0681] The server searches for similar products on each shopping site based on the missing items. The terminal sends the list of missing items to the server, which then searches for the products using the shopping site's API.

[0682] Sending product information

[0683] The server searches for the relevant product and sends the information to the terminal. The product information is received from the search results and displayed on the terminal.

[0684] Purchase procedure

[0685] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[0686] Specific examples

[0687] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[0688] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[0689] This system allows users to efficiently manage inventory and smoothly replenish necessary items.

[0690] The processing flow will be explained below.

[0691] Step 1: Launch the app

[0692] The user launches the app on their smartphone.

[0693] Step 2: Take a photo

[0694] The user takes a photo of the inside of the refrigerator or shelves.

[0695] The device will launch the camera function and display the capture button.

[0696] The user presses the capture button and an image is captured.

[0697] Step 3: Send image

[0698] The terminal compresses the captured image data and prepares it for transmission.

[0699] The device sends image data to the server's API endpoint.

[0700] Step 4: Image analysis

[0701] The server receives the image data.

[0702] The server uses AI object detection algorithms to identify items in the image.

[0703] The server extracts the type and location information of the identified item.

[0704] Step 5: Counting items

[0705] The server counts the number of each item.

[0706] The server structures the counting results in JSON format and sends them to the terminal.

[0707] Step 6: View inventory information

[0708] The device deserializes the data received from the server.

[0709] The device will display a list of items and their quantities on the app.

[0710] Step 7: Inventory Setup

[0711] The user enters the desired stock quantity into the app.

[0712] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[0713] Step 8: List the missing items

[0714] The terminal compares the current stock quantity with the desired stock quantity.

[0715] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[0716] Step 9: View the Missing Items List

[0717] The terminal generates a list of the missing items and displays it to the user.

[0718] Step 10: Search shopping sites

[0719] The terminal transmits the list of missing items to the server.

[0720] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[0721] Step 11: Submit your product information

[0722] The server extracts related product information from search results on the shopping site.

[0723] The server sends product information (product name, price, link, etc.) to the terminal.

[0724] Step 12: View product information and checkout

[0725] The product information received by the device is displayed on the app.

[0726] The user selects the product they want to purchase and presses the purchase button.

[0727] The device will redirect you to the shopping site's purchase page.

[0728] By following the steps above, the user can efficiently manage inventory and smoothly purchase the items they need.

[0729] Example 1

[0730] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] In today's modern lifestyle, inventory management of household devices and storage spaces is extremely cumbersome, and managing consumables is particularly difficult. Users must regularly check the inventory of consumables and manually purchase the necessary items, which is time-consuming. Furthermore, inventory surpluses and shortages can lead to unnecessary expenses and the inconvenience of not having the items available when needed. An efficient system that solves these problems is needed.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0733] In this invention, the server includes a means for a user to take an image of a home appliance or storage location, a means for transmitting the image to a processing device, and a means for the processing device to analyze the image and identify the type and number of items, thereby enabling the user to efficiently manage inventory in the home and automatically replenish necessary items.

[0734] "User" refers to the individual who operates the system and manages the inventory of home devices and storage locations.

[0735] "Household appliances" refers to devices used in the home to store items, such as refrigerators and shelves.

[0736] "Image" refers to still image data taken by a user of a home device or storage location.

[0737] "Processing device" refers to a computer device used to analyze image data and identify the type and number of items.

[0738] "Terminal" refers to a digital device that a user operates at hand, and specifically includes smartphones and tablets.

[0739] "AI object detection algorithm" refers to a machine learning model or deep learning algorithm that identifies objects in an image and determines their type and number.

[0740] "Inventory" refers to the quantity of a particular item in a household.

[0741] "Desired inventory quantity" refers to the target quantity of an item that a user wishes to keep in stock.

[0742] "Shortage items" refer to items whose current inventory is lower than the desired inventory.

[0743] An "e-commerce platform" refers to a website or app that allows people to purchase goods online.

[0744] "Search Results" refers to the product information searched for based on the missing items on the e-commerce platform.

[0745] This invention is a system for managing inventory and automatically replenishing items in the home. The system allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items and manage inventory information.

[0746] First, the user launches a dedicated app on a device such as a smartphone or tablet. The user selects the inventory management function within the app and activates the camera. The user then takes pictures of storage areas in the home, such as refrigerators and shelves. The device compresses the captured image data and sends it to a server via the Internet.

[0747] The server analyzes the received image data. Specifically, it uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image and determine their type and number. The analysis results are structured in JSON format as data including the type and number of objects, and sent to the device. This analysis is often performed by a server equipped with a high-performance GPU.

[0748] Next, the device deserializes the data received from the server and displays it as a list to the user. The user checks this list and, if necessary, enters the desired stock quantity into the app. The app saves the desired stock quantity entered by the user and compares it with the current stock quantity to calculate the shortage of items. The calculation results are also displayed as a list to the user.

[0749] Furthermore, the terminal sends a list of missing items to the server, and the server uses the API of the e-commerce platform to search for the relevant items. The search results are sent to the terminal as specific product information and displayed for the user to review. This allows the user to easily purchase the missing items. This function allows users to efficiently manage inventory and streamline purchasing procedures.

[0750] Examples:

[0751] For example, if a user notices that they are low on milk in the refrigerator while preparing breakfast, they can launch the app and take a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm identifies the type and quantity of each item (e.g., milk, eggs, butter, etc.) in the refrigerator.

[0752] The analysis reveals that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system would automatically list the missing bottle of milk. This list would be displayed on the terminal for the user to review. The server would then use the API of the e-commerce platform to search for available products of the missing milk and send the results to the terminal. The user could then select from the list and easily complete the purchase process.

[0753] The system allows users to efficiently manage their home inventory and automatically replenish items as needed.

[0754] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0755] Step 1:

[0756] The user launches the app on their smartphone. They select the inventory management function from the app's main screen and move to the image capture screen. At this time, the device's camera function is activated and a capture button is displayed.

[0757] Input: User actions

[0758] Output: Activating the camera function and displaying the shooting screen

[0759] Step 2:

[0760] When the user takes a photo of a home device or storage location, the device acquires the image data and temporarily stores it in its internal memory.

[0761] Input: An image taken by the user

[0762] Output: Image data stored in the device's memory

[0763] Step 3:

[0764] The device compresses the captured image data and sends it to the server's API endpoint, where it compresses the image data into JPEG format and sends it to the server via the Internet.

[0765] Input: Image data stored in memory

[0766] Output: Compressed image data sent to the server

[0767] Step 4:

[0768] The server analyzes the received image data. It uses an AI object detection algorithm (e.g., YOLO, SSD) to identify the objects in the image. The server determines the type and number of objects and structures the results in JSON format.

[0769] Input: Compressed image data

[0770] Output: Parsed results in JSON format (e.g., {"Milk": 2, "Egg": 8})

[0771] Step 5:

[0772] The server sends the parsed results in JSON format to the device, which receives them and stores them as internal data.

[0773] Input: Analysis results sent from the server

[0774] Output: Analysis results stored on the device

[0775] Step 6:

[0776] The device deserializes the analysis results and displays them to the user as a list, showing the type and quantity of each item.

[0777] Input: Parsed result in JSON format

[0778] Output: Inventory information displayed to the user (e.g., Milk: 2 bottles, Eggs: 8)

[0779] Step 7:

[0780] The user inputs the desired stock quantity into the app. The device displays the stock setting screen, and when the user inputs the desired stock quantity for each item, it is saved in the app.

[0781] Input: The desired stock quantity entered by the user

[0782] Output: Desired stock quantity saved in the app

[0783] Step 8:

[0784] The terminal compares the current stock with the stock desired by the user and lists the items that are in short supply. As a result of the calculation, a list of items that are in short supply is generated.

[0785] Input: Current stock quantity and desired stock quantity

[0786] Output: List of items in short supply (e.g. Milk: 1 bottle missing)

[0787] Step 9:

[0788] The terminal sends a list of missing items to the server, which then searches for the relevant items on the e-commerce platform and retrieves product information using the API of each platform.

[0789] Input: List of missing items

[0790] Output: Product information obtained from the e-commerce platform

[0791] Step 10:

[0792] The server sends the acquired product information to the terminal, which then displays it to the user, who can then check the displayed product information.

[0793] Input: Product information obtained from the e-commerce platform

[0794] Output: Product information displayed to the user

[0795] Step 11:

[0796] The user completes the purchase process. When the user presses the purchase button, the terminal redirects them to the purchase page of the e-commerce platform to complete the purchase process.

[0797] Input: User purchase operation

[0798] Output: Purchase procedure completed

[0799] (Application example 1)

[0800] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0801] In modern brick-and-mortar stores, inventory management is a time-consuming and labor-intensive task. Accurately tracking inventory levels on shelves and display areas and replenishing them in a timely manner are particularly important and directly affect customer satisfaction. However, traditional manual inventory checks are inefficient and carry a high risk of errors. Therefore, a more efficient and accurate method of inventory management is needed.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0803] In this invention, the server includes means for a user to take images of home appliances, storage locations, shelves, and display corners in a store, means for transmitting the images to the server, means for the server to analyze the images and identify the type and number of items, and means for transmitting the analysis results to a terminal. This enables efficient and accurate inventory management in a physical store and enables a quick listing and notification of products that need to be replenished.

[0804] "Home appliances" is a general term for electrical and electronic devices used in the home.

[0805] "Storage area" refers to a space or container for storing items.

[0806] A "server" is a computer system that stores data and provides services to clients on a network.

[0807] "Image analysis" is the process of computer-based analysis of digital images to identify objects and features within the images.

[0808] "Goods" is a general term that refers to tangible goods or merchandise.

[0809] A "species" is a taxonomic group with different characteristics or properties.

[0810] "Quantity" indicates the quantity of an item.

[0811] A "terminal" is a computer device or smart device that a user operates.

[0812] "Inventory" means the total quantity of items currently stored or on display.

[0813] "Listing" is the act of displaying a list of items based on specific criteria.

[0814] A "shopping site" is a website where you can purchase goods and services over the Internet.

[0815] A "shelf" is a structure for displaying and storing items.

[0816] A "display corner" is an area within a store where items are displayed and sold.

[0817] An "AI object detection algorithm" is a computational method for identifying objects in an image using artificial intelligence technology.

[0818] "Management" means organizing and operating things according to a purpose.

[0819] "Replenishing" is the act of adding items that are lacking.

[0820] This invention relates to a system for streamlining store inventory management. This system uses image analysis technology and AI object detection algorithms to grasp the inventory status of items on shelves and display corners in a store and provides users with a list of items that are in short supply.

[0821] System configuration

[0822] 1. Photography Method:

[0823] The user takes an image of the shelf or display area using the camera on their smartphone. There are no particular restrictions on the smartphone as long as it has a general camera function.

[0824] 2. Image transmission method:

[0825] The image is sent to the server. The smartphone sends the captured image data to the server via an Internet connection. An HTTP request library (e.g., requests) is used.

[0826] 3. Image analysis methods:

[0827] The server analyzes the images using an AI object detection algorithm to identify the type and number of items, preferably with advanced GPU capabilities to run specific object detection models (e.g., YOLO, SSD).

[0828] 4. Data transmission and reception means:

[0829] The server sends the analysis results to the smartphone, which receives them. The analyzed product information is sent and received using JSON format data.

[0830] 5. Inventory Display Method:

[0831] The smartphone displays the analysis results on its screen. The user can visualize the inventory status of the items through the application. A common front-end framework (e.g., React Native, Flutter) is used to provide the user interface (UI).

[0832] 6. Inventory setting method:

[0833] The user sets the desired stock quantity, which is stored in the application and used as a basis for comparison.

[0834] 7. Listing Methods:

[0835] The smartphone compares the desired inventory quantity with the current inventory quantity and creates a list of items that are in short supply based on the inventory data received from the server.

[0836] 8. Search Methods:

[0837] The smartphone sends the entire list of missing items to a server, which then searches for the items via the API of the shopping site.

[0838] 9. Purchasing Method:

[0839] The user purchases an item through the search results, and the smartphone redirects the user to a purchase page on the shopping site.

[0840] Hardware and software used

[0841] Hardware:

[0842] Smartphone (camera function and internet connection)

[0843] Server (with GPU functionality)

[0844] software:

[0845] Camera API

[0846] HTTP request library (e.g., requests)

[0847] AI object detection models (e.g., YOLO, SSD)

[0848] JSON Data Format

[0849] User interface frameworks (e.g., React Native, Flutter)

[0850] Shopping site API

[0851] Specific examples

[0852] For example, consider a situation where a store clerk uses a smartphone app to check the inventory in the candy section. When the app takes a photo of the shelves, the AI ​​object detection algorithm automatically recognizes the type and quantity of each item on the shelves and lists any items that are missing. Based on the analysis results, the server searches for identical or similar products on the shopping site and presents the search results to the user. This allows the clerk to efficiently manage inventory and quickly replenish products as needed.

[0853] Prompt Sentence Examples

[0854] A store inventory management app is used to check the stock in the snack section. When a store associate takes a photo of the shelf with the smartphone app, the app automatically recognizes the type and quantity of each item and lists any items that are low in stock. Specifically, if it finds that potato chips are out of stock, the app notifies the associate. Please describe in detail a situation in which the associate can efficiently check the stock based on this prompt.

[0855] The above is a specific embodiment for carrying out the present invention.

[0856] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0857] Step 1:

[0858] The user takes pictures of home appliances, storage areas, shelves, and display corners in a store.

[0859] Users use the camera function of their smartphone to take photos of home appliances, storage areas, or shelves and display areas in stores. These images are used as visual data for the items to be managed in inventory.

[0860] Input: Image taken with a smartphone camera

[0861] Output: Digital image data

[0862] Step 2:

[0863] The terminal transmits the image to the server.

[0864] The device uses an HTTP request over its internet connection to send the captured image data to the server, where it is compressed and encoded as appropriate and sent to the server's API.

[0865] Input: Digital image data

[0866] Output: Image data sent to the server

[0867] Step 3:

[0868] The server analyzes the image and identifies the type and number of items.

[0869] The server analyzes the received image data using an AI object detection algorithm (e.g., YOLO, SSD), which identifies the type and number of each item in the image.

[0870] Input: Image data sent to the server

[0871] Output: Analysis results regarding the type and number of items

[0872] Step 4:

[0873] The server sends the analysis results to the terminal.

[0874] The server structures the analysis results as JSON data and sends it to the terminal via an HTTP request. This JSON data includes the type and quantity of each item.

[0875] Input: Analysis results regarding type and number of items

[0876] Output: JSON data of the analysis results sent to the terminal

[0877] Step 5:

[0878] The terminal displays the analysis results on the screen.

[0879] The terminal deserializes the JSON data received from the server and displays the inventory status of the items on the screen through a user interface, where the user can check the type and quantity of each item.

[0880] Input: JSON data of analysis results

[0881] Output: Inventory information displayed on a smartphone screen

[0882] Step 6:

[0883] Set the stock quantity desired by the user

[0884] The user enters the desired inventory quantity in the application, for example, 12 eggs or 3 bottles of milk, and this information becomes the baseline data.

[0885] Input: The number of items in stock desired by the user

[0886] Output: The desired stock quantity stored in the application

[0887] Step 7:

[0888] The terminal compares the desired inventory with the current inventory and lists the items that are in short supply.

[0889] The terminal compares the current inventory data with the desired inventory quantity set by the user, calculates the shortage, and lists it. The list of shortage items is used in the next step.

[0890] Input: Desired stock quantity and current stock quantity

[0891] Output: List of missing items

[0892] Step 8:

[0893] The server searches various shopping sites for the same or similar products of the listed items.

[0894] The server uses the API of each shopping site to search for the same or similar products based on the list of missing items, and the search results are sent to the terminal.

[0895] Input: List of missing items

[0896] Output: Search results from a shopping site

[0897] Step 9:

[0898] The terminal displays the search results on the screen.

[0899] The terminal displays the search results received from the server, and the user can use them as a reference to purchase the necessary items.

[0900] Input: Search results sent from the server

[0901] Output: Search results displayed on a smartphone screen

[0902] Step 10:

[0903] The user purchases an item through the search result.

[0904] The user can then proceed to purchase the selected item within the application, and the device will redirect the user to the shopping site's purchase page.

[0905] Input: Select a user based on search results

[0906] Output: Purchase completed

[0907] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0908] The present invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchase suggestions.

[0909] System Overview

[0910] The system includes the following main elements:

[0911] 1. Filming Method

[0912] 2. Image transmission method

[0913] 3. Image analysis methods

[0914] 4. Data transmission and reception means

[0915] 5. Inventory Display Method

[0916] 6. Inventory Setting Method

[0917] 7. How to create a list of items that are in short supply

[0918] 8. Product Search Methods

[0919] 9. Purchasing Method

[0920] 10. Emotion Engine

[0921] Program processing

[0922] User operations

[0923] Launching the app

[0924] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[0925] photo shoot

[0926] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[0927] Inventory Settings

[0928] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[0929] Emotion input

[0930] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. Emotional data is collected by the user's voice input or facial expressions displayed through the camera. This data is used for inventory management and purchasing suggestions.

[0931] Image analysis and data processing

[0932] Image transmission

[0933] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[0934] Image analysis

[0935] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[0936] Counting items

[0937] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them.

[0938] Display inventory information and check shortages

[0939] View inventory information

[0940] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[0941] Listing shortages

[0942] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[0943] View the missing items list

[0944] The terminal generates a list of the missing items and displays it to the user.

[0945] Product search and purchase

[0946] Shopping site search

[0947] The terminal sends a list of missing items to the server, which then uses the APIs of various shopping sites (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[0948] Sending product information

[0949] The server extracts related product information from search results on the shopping site, and sends the product information (product name, price, link, etc.) to the device.

[0950] Purchase procedure

[0951] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[0952] Use of emotion engine

[0953] Emotional Data Analysis and Applications

[0954] The emotion engine analyzes the user's voice and facial expression data in real time to recognize their emotions. This emotion data is then applied to inventory management and purchasing suggestions. For example, if the user is feeling stressed, it will suggest products that will help them relax.

[0955] Recording frequency of use and emotions

[0956] The emotion engine accumulates and records the user's emotional data and predicts their long-term consumption trends, enabling optimal product recommendations based on the user's preferences and tendencies.

[0957] Specific examples

[0958] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[0959] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[0960] Also, if a user is feeling stressed during busy morning hours, the emotion engine will recognize this and suggest products that will help reduce stress (such as relaxing herbal tea). In this way, appropriate product suggestions are made based on the user's emotions.

[0961] This system not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows them to receive optimal product suggestions based on their emotions.

[0962] The processing flow will be explained below.

[0963] Step 1: Launch the app

[0964] The user launches the app on their smartphone.

[0965] Step 2: Take a photo

[0966] The user takes a photo of the inside of the refrigerator or shelves.

[0967] The device will launch the camera function and display the capture button.

[0968] The user presses the capture button and an image is captured.

[0969] Step 3: Send image

[0970] The terminal compresses the captured image data and prepares it for transmission.

[0971] The device sends image data to the server's API endpoint.

[0972] Step 4: Image analysis

[0973] The server receives the image data.

[0974] The server uses AI object detection algorithms to identify items in the image.

[0975] The server extracts the type and location information of the identified item.

[0976] Step 5: Counting items

[0977] The server counts the number of each item.

[0978] The server structures the counting results in JSON format and sends them to the terminal.

[0979] Step 6: View inventory information

[0980] The device deserializes the data received from the server.

[0981] The device will display a list of items and their quantities on the app.

[0982] Step 7: Inventory Setup

[0983] The user enters the desired stock quantity into the app.

[0984] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[0985] Step 8: Enter emotions

[0986] The user displays facial expressions through voice input or a camera.

[0987] The device sends voice data and facial expression images to the emotion engine.

[0988] Step 9: Sentiment Analysis

[0989] The server's emotion engine analyzes voice data and facial expression images in real time to recognize the user's emotions.

[0990] The server stores the recognized emotion data and uses it for subsequent inventory management and purchasing suggestions.

[0991] Step 10: List the missing items

[0992] The terminal compares the current stock quantity with the desired stock quantity.

[0993] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[0994] Step 11: View the Shortage List

[0995] The terminal generates a list of the missing items and displays it to the user.

[0996] Step 12: Search shopping sites

[0997] The terminal transmits the list of missing items to the server.

[0998] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[0999] Step 13: Submit your product information

[1000] The server extracts related product information from search results on the shopping site.

[1001] The server sends product information (product name, price, link, etc.) to the terminal.

[1002] Step 14: View product information and checkout

[1003] The product information received by the device is displayed on the app.

[1004] The user selects the product they want to purchase and presses the purchase button.

[1005] The device will redirect you to the shopping site's purchase page.

[1006] Step 15: Emotion-based suggestions

[1007] Based on the user's emotional data recognized by the emotion engine, the system suggests products that are best suited to the user's current emotional state, such as relaxing drinks or health foods.

[1008] The terminal displays emotion-based product suggestions to the user.

[1009] Step 16: Analyze long-term consumption trends

[1010] The emotion engine accumulates and records users' emotional data and analyzes their long-term consumption trends.

[1011] The server optimizes weekly or monthly purchase suggestions based on the analysis results.

[1012] In this way, the user can efficiently manage inventory, smoothly purchase the items he or she needs, and furthermore, receive appropriate product suggestions according to the user's emotional state.

[1013] Example 2

[1014] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1015] Inventory management and replenishment of home appliances and storage spaces are important issues for modern households. In particular, there is a need for a system that quickly and accurately replenishes items when they run out, especially in busy daily lives. However, conventional inventory management systems rely heavily on manual input by users, making efficient management difficult. Furthermore, they lack functionality for optimal product recommendations based on users' emotions and preferences. This can lead to inconvenient situations where necessary items cannot be properly replenished. Furthermore, it is difficult to provide services that respond to users' individual needs and emotional changes.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1017] In this invention, the server includes means for a user to take an image of the home appliance or storage location, means for transmitting the image to the server, means for the server to analyze the image and identify the type and number of items, means for transmitting the analysis results to a terminal, means for displaying the analysis results on a terminal screen, means for a user to set a desired inventory quantity, means for comparing the desired inventory quantity with the current inventory quantity and listing items that are in short supply, means for searching various e-commerce sites for products that are identical to or similar to the listed items, means for displaying the search results on the terminal, means for analyzing user emotion data and applying the emotion data to inventory management and purchase suggestions, and means for a user to purchase items based on the search results. This enables efficient inventory management and rapid replenishment of items, and also realizes optimal product suggestions based on the user's emotions and preferences.

[1018] "User" refers to a person who operates the system, takes pictures of home appliances and storage locations, and issues various instructions regarding inventory management and purchasing of goods.

[1019] "Home appliances" refers to electrical appliances used to store household items, such as refrigerators and shelves.

[1020] "Storage space" refers to a place or facility for storing items, such as the inside of a refrigerator or a shelf.

[1021] "Images" refers to photos or videos taken by the user of the inside of an appliance or storage space.

[1022] The term "server" refers to a computer system that receives and analyzes image data sent by a user.

[1023] "Goods" refers to goods or products stored in appliances or storage areas, including food items such as milk and eggs.

[1024] "Quantity" refers to the quantity of a particular item.

[1025] "Terminal" refers to a mobile device such as a smartphone or tablet used by a user.

[1026] "Screen" refers to the display shown on the terminal.

[1027] "Inventory" refers to the quantity of items currently in stock.

[1028] "Desired inventory quantity" refers to the quantity of items desired by the user to be stored.

[1029] "Listing" refers to comparing the current inventory with the desired inventory and listing items that are in short supply.

[1030] "E-commerce site" refers to a website for buying and selling products online, including, for example, Amazon, Yahoo, and Rakuten.

[1031] "Emotion data" refers to data related to emotions collected from the user's voice and facial expressions.

[1032] "Purchase method" refers to the system, such as online payment or shopping cart, that a user uses to purchase goods.

[1033] "Machine learning object detection algorithm" refers to a machine learning or artificial intelligence algorithm used to automatically identify objects in images, such as YOLO or SSD.

[1034] This invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchasing suggestions.

[1035] The system includes the following main elements:

[1036] 1. A way for users to take pictures of home appliances and storage locations

[1037] 2. Means of sending the captured image to the server

[1038] 3. Means by which the server analyzes the image and identifies the type and number of items

[1039] 4. Means of sending analysis results to the device

[1040] 5. How to display the analysis results on the device screen

[1041] 6. A way for users to set their desired stock quantity

[1042] 7. A method for comparing desired inventory with current inventory and listing items that are in short supply

[1043] 8. A means to search for the same or similar products of the listed items on various e-commerce sites.

[1044] 9. How to display search results on your device

[1045] 10. Means of analyzing user sentiment data and applying sentiment data to inventory management and purchase recommendations

[1046] 11. How users can purchase items through search results

[1047] Users launch the app on their smartphone or other device and take a photo of the contents of their refrigerator or shelves. The device compresses the image and sends it to the server. The server then uses an AI object detection algorithm (e.g., YOLO or SSD) to identify the type and number of items in the image and sends the analysis results back to the device.

[1048] Users can check the analysis results on the device screen and set the desired inventory quantity as needed. The device compares the current inventory quantity with the desired inventory quantity and lists the items that are in short supply. For the listed items, the server uses the APIs of various e-commerce sites (e.g., Amazon, Yahoo, Rakuten) to search for similar products and send the search results to the device.

[1049] The emotion engine analyzes the user's voice and facial expressions to collect emotional data. This emotional data is then applied to inventory management and purchase suggestions. For example, if a user is feeling stressed, it will suggest products that have a relaxing effect. The user can select the suggested products on the screen and proceed with the purchase.

[1050] As a concrete example, suppose a user is preparing breakfast and notices that there is little milk in the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm is used to identify the type and number of bottles of milk. The analysis results show that there are only two bottles of milk left in the refrigerator. Since the user's desired stock is three bottles, the system lists the one bottle that is missing.

[1051] The server uses the API of the e-commerce site to search for the missing item and display it to the user. Furthermore, the emotion engine recognizes that the user is feeling stressed while preparing breakfast and suggests relaxing herbal teas.

[1052] The user can select the appropriate product from the options presented and quickly complete the purchase procedure.

[1053] Example prompts to input to the generative AI model

[1054] 1. Inventory count prompt:

[1055] "When the user takes a photo of the refrigerator, count the number of bottles of milk using YOLO and return the result in JSON format."

[1056] 2. Prompt to generate a list of missing items:

[1057] "Compare the current stock quantity data with the user's desired stock quantity data and generate a list of items that are in short supply."

[1058] 3. Emotion-Based Product Recommendation Prompt:

[1059] "Analyze the collected user emotional data and suggest products that have a relaxing effect when the user is feeling stressed."

[1060] As a result, a system is realized that not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows users to receive optimal product suggestions based on their emotions.

[1061] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1062] Explain the process step by step

[1063] User operations

[1064] Step 1: Launch the app

[1065] A user launches an app on their smartphone. The input is a touch operation, and the output is the main screen of the app.

[1066] The device loads configuration files and user data in the background and prepares to operate normally.

[1067] Step 2: Take a photo

[1068] The user takes a photo of the inside of the refrigerator or shelf. The device's camera function is used as input, and image data is generated as output.

[1069] The device activates the camera function, and when the user presses the capture button, a photo is taken. This image data is temporarily stored in the device's memory.

[1070] Step 3: Inventory Setup

[1071] The user manually inputs the desired inventory quantity into the app, and the desired inventory data is generated as output.

[1072] The terminal displays an inventory setting screen, and the user inputs the items they need (e.g., 3 bottles of milk, 12 eggs). This information is stored in the terminal's internal database.

[1073] Step 4: Enter emotions

[1074] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. The user's voice and facial expression data are used as input, and emotion data is generated as output.

[1075] The device collects and temporarily stores the user's facial expressions through voice input and a camera, and this data is later sent to a server.

[1076] Image analysis and data processing

[1077] Step 5: Send image

[1078] The terminal sends the captured image to the server. The input is image data, and the output is compressed image data sent to the server.

[1079] The device compresses the image data and sends it to the server's API endpoint.

[1080] Step 6: Image analysis

[1081] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The input is image data, and the output is data on the type and number of objects.

[1082] The server receives the image data and runs an object detection algorithm to extract the type and location of the item, and stores the results in a database.

[1083] Step 7: Counting items

[1084] The server counts the number of each item and sends the result to the terminal. The input is the identified item data, and the output is the counted item data.

[1085] The server categorizes the items, counts the number of each item, and structures the results in JSON format. It then checks the data for integrity before sending it to the device.

[1086] Display inventory information and check shortages

[1087] Step 8: View inventory information

[1088] The terminal displays a list of items and their quantities. The input is the item count data, and the output is the inventory list displayed on the screen.

[1089] The terminal deserializes the received data and displays it in a list format on the screen, allowing the user to check the current inventory status.

[1090] Step 9: List the missing items

[1091] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists the items that are in short supply. The input is the desired inventory data and the current inventory data, and the output is a list of items that are in short supply.

[1092] The terminal compares the two and calculates the shortage. For example, if the desired stock is 3 bottles of milk and the current stock is 2 bottles, the shortage will be 1 bottle.

[1093] Step 10: View the Missing Items List

[1094] The terminal displays the generated shortage list. The input is the shortage list data, and the shortage list is displayed on the screen as the output.

[1095] The shortage list is displayed in a visually easy-to-understand format, allowing users to quickly take next steps.

[1096] Product search and purchase

[1097] Step 11: Search for shopping sites

[1098] The terminal sends the list of items in short supply to the server. The input is the list of items in short supply, and the output is the search results for similar items.

[1099] The server uses the APIs of various e-commerce sites to search for products that correspond to the missing items and sends the results to the terminal.

[1100] Step 12: Submit your product information

[1101] The server sends related product information from search results on a shopping site to the terminal. The input is search result data, and the output is product information.

[1102] Data including product information (product name, price, link, etc.) is sent to the terminal. The terminal analyzes the received data and displays it to the user in an appropriate format.

[1103] Step 13: Checkout

[1104] The user selects a product on the app and completes the purchase process. The input is product selection data, and the output is purchase confirmation data.

[1105] The terminal displays the received product information, and when the user presses the purchase button, the terminal redirects the user to the shopping site's purchase page. Once the purchase procedure is complete, the terminal displays a confirmation message.

[1106] Use of emotion engine

[1107] Step 14: Analyze and apply emotion data

[1108] The emotion engine analyzes the user's voice and facial expression data in real time to recognize emotions. The input is voice and facial expression data, and the output is emotional data.

[1109] For example, if a user is feeling stressed, the system will suggest products that will help them relax (e.g., herbal tea), which will then be applied to inventory management and purchasing suggestions.

[1110] Step 15: Record frequency of use and emotions

[1111] The emotion engine accumulates and records the user's emotional data and predicts the user's long-term consumption trends. The input is the accumulated emotional data, and the output is predicted consumption trends.

[1112] For example, it is possible to identify the time periods during which a user is likely to feel stressed and suggest products suitable for those time periods.

[1113] Through these steps, users can efficiently manage their inventory, quickly replenish needed items, and receive optimal product recommendations based on their emotions.

[1114] (Application example 2)

[1115] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1116] In recent years, there has been a demand for improved cargo management efficiency in autonomous vehicles. However, with conventional inventory management systems, drivers must manually check inventory levels, identify missing items, and then carry out replenishment procedures, which requires time and effort. Furthermore, simply indicating missing items without considering the driver's feelings increases stress and fatigue for the driver, resulting in a decrease in work efficiency. A system that solves these issues, performs efficient inventory management, and reduces the burden on drivers is needed.

[1117] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for the user to take an image of the storage location, a means for transmitting the image to the server, and a means for the server to analyze the image and identify the type and number of items. This allows the driver to easily check the inventory status of the cargo space using the wearable device, as well as identify missing items in real time and automatically receive replenishment suggestions. In addition, the system includes a means for recognizing the driver's emotions and a means for suggesting appropriate products based on the recognized emotion data, thereby reducing the driver's stress and allowing them to work efficiently and comfortably.

[1118] "Means for taking images" refers to a function that allows a user to take images of a storage area or cargo space using a smart device or wearable device.

[1119] The "means for transmitting images to a server" is a function for transmitting captured image data to a server via a network such as the Internet.

[1120] "Means for identifying the type and number of items" refers to a function that analyzes the image received by the server and identifies the type and number of items in the image using an AI object detection algorithm.

[1121] "Means for transmitting analysis results to a terminal" refers to a function for transmitting data analyzed by the server to a user's terminal, particularly a smart device or wearable device.

[1122] The "means for displaying the analysis results on the display device of the terminal" is a function for visually displaying the analysis results on the screen of the user's terminal.

[1123] The "means for setting the stock quantity desired by the user" is a function that allows the user to input the stock quantity desired by the user and record it in the system.

[1124] The "means for listing items in short supply" is a function that compares the set desired inventory quantity with the current inventory quantity and displays the items in short supply in a list format.

[1125] "Means for searching for identical or similar products in various shopping systems" refers to a function for searching online shopping sites for products that are identical or similar to the missing item.

[1126] The "means for displaying search results on a terminal" is a function for displaying product search results obtained from a shopping site on a user's terminal.

[1127] The "means for purchasing goods" is a function for a user to complete the purchase procedure for the product selected by the user through the terminal.

[1128] "Means for managing items in the cargo space, including a wearable device" refers to a function that uses a wearable device worn by the driver to manage inventory in the cargo space.

[1129] The "means for recognizing emotions" is a function that analyzes the user's facial expressions and tone of voice through the wearable device's camera or voice input, and recognizes their emotions.

[1130] The "means for suggesting appropriate products based on recognized emotional data" is a function that suggests products that have stress-reducing or relaxing effects based on the user's emotional data.

[1131] The "means for displaying the proposed results" is a function for displaying the products and information proposed to the user on the terminal.

[1132] This invention relates to a system for improving the efficiency of cargo management in autonomous vehicles. Specifically, the system takes images of the cargo space, identifies the type and number of items using an AI object detection algorithm, and performs inventory management while recognizing the driver's emotions to suggest optimal products. This system is intended to be operated by the user using a wearable device.

[1133] Hardware Configuration

[1134] 1. Wearable devices (e.g., smart glasses)

[1135] It is worn by the driver and takes pictures of the cargo space.

[1136] It has a built-in camera and microphone to collect image and audio data.

[1137] 2. Cloud Server

[1138] Performs image analysis, object detection, and emotion recognition

[1139] Responsible for storing and processing data

[1140] 3. Smart Devices

[1141] Driver-carried smartphones and tablets

[1142] Communicates with the server and displays analysis results and product suggestions

[1143] Software Configuration

[1144] 1. AI object detection algorithms (e.g., YOLO)

[1145] Analyze images captured by the wearable device to identify the type and quantity of cargo

[1146] 2. Sentiment analysis algorithms (e.g., EmotionAnalyzer)

[1147] Analyzes emotions based on the driver's facial expressions and voice data captured through cameras and microphones

[1148] 3. Data transmission and reception module

[1149] It serves as a data transfer mechanism between wearable devices and cloud servers.

[1150] 4. Inventory Management Module

[1151] Manage inventory data on the server

[1152] 5. Proposal Module

[1153] Recommend products to drivers based on emotion data

[1154] Detailed process flow

[1155] 1. Image capture and transmission

[1156] A user takes an image of the cargo space using a wearable device

[1157] The captured image is sent to the server

[1158] 2. Image Analysis

[1159] The server uses AI object detection algorithms to identify objects in the image and determine their type and quantity.

[1160] 3. Inventory Management

[1161] The user inputs the desired inventory quantity into the smart device, and the server compares it with the current inventory to identify any shortages.

[1162] 4. Emotion Recognition and Product Recommendations

[1163] Using the camera and microphone of the wearable device, the driver's facial expressions and voice data are analyzed with an emotion analysis algorithm to recognize their emotions.

[1164] Makes product recommendations based on emotions and displays the results on a smart device

[1165] Specific examples

[1166] While replenishing cargo, the driver uses smart glasses to take images of the cargo space. The images are sent to a server, where an AI object detection algorithm analyzes inventory shortages. At the same time, if the driver's stress level is high, suggestions for relaxation products (e.g., herbal tea) are displayed.

[1167] Prompt Sentence Examples

[1168] "Check stock availability"

[1169] "Please suggest a relaxing product."

[1170] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1171] Step 1:

[1172] A user takes an image of the cargo space using a wearable device (smart glasses).

[1173] Input: Cargo space image

[1174] Output: Captured image data

[1175] Specific operation: Using the camera function of the smart glasses, the user faces the cargo space and presses the capture button to capture an image.

[1176] Step 2:

[1177] The device sends the captured image to a cloud server.

[1178] Input: Photographed image data

[1179] Output: Image data sent to the server

[1180] Specific operation: The transmission module on the smart glasses is activated and sends the image data to the server via the Internet.

[1181] Step 3:

[1182] The server uses an AI object detection algorithm to analyze the image and identify the type and number of items.

[1183] Input: Image data sent to the server

[1184] Output: Data on type and number of items

[1185] Specific operation: The server runs image analysis software (e.g., YOLO) to identify and count the items, and stores the results in a database.

[1186] Step 4:

[1187] The server sends the analysis results to the user's smart device.

[1188] Input: Data on type and quantity of items

[1189] Output: Analysis data sent to smart device

[1190] Specific operation: The server sends the generated item list to the user's smartphone or tablet via the data communication module.

[1191] Step 5:

[1192] The terminal displays the analysis results on a display device.

[1193] Input: Submitted analysis data

[1194] Output: Inventory information displayed on screen

[1195] Specific operation: The display module of the smart device displays the analysis results in list format on the screen so that the user can check them.

[1196] Step 6:

[1197] The user sets the desired stock quantity via a smart device.

[1198] Input: The desired stock quantity entered by the user

[1199] Output: Desired inventory quantity data

[1200] Specific operation: The user enters the desired stock quantity of each item on the application screen of their smart device, and the data is saved within the app.

[1201] Step 7:

[1202] The server compares the current stock with the desired stock and lists the items that are in short supply.

[1203] Input: Current stock quantity data and desired stock quantity data

[1204] Output: List of missing items

[1205] Specific operation: The server compares the stored inventory data with the desired inventory data, calculates the difference, and generates a list of items that are in short supply.

[1206] Step 8:

[1207] The server searches various shopping systems for the same or similar products as the missing item.

[1208] Input: List of missing items

[1209] Output: Shopping system search results

[1210] Specific operation: The server uses the API of each shopping system (e.g., Amazon, Rakuten) to search for products similar to the missing item.

[1211] Step 9:

[1212] The server sends the search results to the terminal.

[1213] Input: Shopping system search results

[1214] Output: Search results sent to your device

[1215] Specific operation: The server organizes the search results and sends them to the user's smart device.

[1216] Step 10:

[1217] The terminal displays the search results to the user, and the user purchases the item.

[1218] Input: Submitted search results

[1219] Output: User completes purchase

[1220] Specific operation: The user selects a product from the search results displayed through the smart device application and presses the purchase button to complete the purchase process.

[1221] Step 11:

[1222] The wearable device on the terminal recognizes the driver's emotions and transmits the emotional data to a server.

[1223] Input: Driver's facial expressions and voice data

[1224] Output: Emotion data sent to the server

[1225] Specific operation: Emotion data collected through the camera and microphone of the wearable device is analyzed and sent to the server.

[1226] Step 12:

[1227] The server suggests appropriate products based on the recognized emotion data and displays them on the device.

[1228] Input: Emotion data

[1229] Output: Appropriate product suggestions

[1230] How it works: The server's emotion analysis algorithm (e.g., EmotionAnalyzer) analyzes the emotion data and generates possible product suggestions. These suggestions are sent to the smart device and displayed to the user.

[1231] The above are the specific processing steps.

[1232] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1233] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1234] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1235] [Third embodiment]

[1236] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1237] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1238] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1239] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1240] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1241] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1242] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1243] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1244] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1245] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1246] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1247] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1248] This invention is a system that allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items, thereby managing inventory and automatically replenishing items.

[1249] System Overview

[1250] The system includes the following main elements:

[1251] 1. Filming Method

[1252] 2. Image transmission method

[1253] 3. Image analysis methods

[1254] 4. Data transmission and reception means

[1255] 5. Inventory Display Method

[1256] 6. Inventory Setting Method

[1257] 7. How to create a list of items that are in short supply

[1258] 8. Product Search Methods

[1259] 9. Purchasing Method

[1260] Program processing

[1261] User operations

[1262] Launching the app

[1263] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[1264] photo shoot

[1265] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[1266] Inventory Settings

[1267] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[1268] Image analysis and data processing

[1269] Image transmission

[1270] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[1271] Image analysis

[1272] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[1273] Counting items

[1274] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them to the terminal.

[1275] Display inventory information and check shortages

[1276] View inventory information

[1277] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[1278] Listing shortages

[1279] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[1280] View the missing items list

[1281] The terminal displays a list of the missing items to the user. A list of missing items is generated so that the user can check the missing items.

[1282] Product search and purchase

[1283] Shopping site search

[1284] The server searches for similar products on each shopping site based on the missing items. The terminal sends the list of missing items to the server, which then searches for the products using the shopping site's API.

[1285] Sending product information

[1286] The server searches for the relevant product and sends the information to the terminal. The product information is received from the search results and displayed on the terminal.

[1287] Purchase procedure

[1288] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[1289] Specific examples

[1290] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[1291] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[1292] This system allows users to efficiently manage inventory and smoothly replenish necessary items.

[1293] The processing flow will be explained below.

[1294] Step 1: Launch the app

[1295] The user launches the app on their smartphone.

[1296] Step 2: Take a photo

[1297] The user takes a photo of the inside of the refrigerator or shelves.

[1298] The device will launch the camera function and display the capture button.

[1299] The user presses the capture button and an image is captured.

[1300] Step 3: Send image

[1301] The terminal compresses the captured image data and prepares it for transmission.

[1302] The device sends image data to the server's API endpoint.

[1303] Step 4: Image analysis

[1304] The server receives the image data.

[1305] The server uses AI object detection algorithms to identify items in the image.

[1306] The server extracts the type and location information of the identified item.

[1307] Step 5: Counting items

[1308] The server counts the number of each item.

[1309] The server structures the counting results in JSON format and sends them to the terminal.

[1310] Step 6: View inventory information

[1311] The device deserializes the data received from the server.

[1312] The device will display a list of items and their quantities on the app.

[1313] Step 7: Inventory Setup

[1314] The user enters the desired stock quantity into the app.

[1315] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[1316] Step 8: List the missing items

[1317] The terminal compares the current stock quantity with the desired stock quantity.

[1318] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[1319] Step 9: View the Missing Items List

[1320] The terminal generates a list of the missing items and displays it to the user.

[1321] Step 10: Search shopping sites

[1322] The terminal transmits the list of missing items to the server.

[1323] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[1324] Step 11: Submit your product information

[1325] The server extracts related product information from search results on the shopping site.

[1326] The server sends product information (product name, price, link, etc.) to the terminal.

[1327] Step 12: View product information and checkout

[1328] The product information received by the device is displayed on the app.

[1329] The user selects the product they want to purchase and presses the purchase button.

[1330] The device will redirect you to the shopping site's purchase page.

[1331] By following the steps above, the user can efficiently manage inventory and smoothly purchase the items they need.

[1332] Example 1

[1333] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1334] In today's modern lifestyle, inventory management of household devices and storage spaces is extremely cumbersome, and managing consumables is particularly difficult. Users must regularly check the inventory of consumables and manually purchase the necessary items, which is time-consuming. Furthermore, inventory surpluses and shortages can lead to unnecessary expenses and the inconvenience of not having the items available when needed. An efficient system that solves these problems is needed.

[1335] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1336] In this invention, the server includes a means for a user to take an image of a home appliance or storage location, a means for transmitting the image to a processing device, and a means for the processing device to analyze the image and identify the type and number of items, thereby enabling the user to efficiently manage inventory in the home and automatically replenish necessary items.

[1337] "User" refers to the individual who operates the system and manages the inventory of home devices and storage locations.

[1338] "Household appliances" refers to devices used in the home to store items, such as refrigerators and shelves.

[1339] "Image" refers to still image data taken by a user of a home device or storage location.

[1340] "Processing device" refers to a computer device used to analyze image data and identify the type and number of items.

[1341] "Terminal" refers to a digital device that a user operates at hand, and specifically includes smartphones and tablets.

[1342] "AI object detection algorithm" refers to a machine learning model or deep learning algorithm that identifies objects in an image and determines their type and number.

[1343] "Inventory" refers to the quantity of a particular item in a household.

[1344] "Desired inventory quantity" refers to the target quantity of an item that a user wishes to keep in stock.

[1345] "Shortage items" refer to items whose current inventory is lower than the desired inventory.

[1346] An "e-commerce platform" refers to a website or app that allows people to purchase goods online.

[1347] "Search Results" refers to the product information searched for based on the missing items on the e-commerce platform.

[1348] This invention is a system for managing inventory and automatically replenishing items in the home. The system allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items and manage inventory information.

[1349] First, the user launches a dedicated app on a device such as a smartphone or tablet. The user selects the inventory management function within the app and activates the camera. The user then takes pictures of storage areas in the home, such as refrigerators and shelves. The device compresses the captured image data and sends it to a server via the Internet.

[1350] The server analyzes the received image data. Specifically, it uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image and determine their type and number. The analysis results are structured in JSON format as data including the type and number of objects, and sent to the device. This analysis is often performed by a server equipped with a high-performance GPU.

[1351] Next, the device deserializes the data received from the server and displays it as a list to the user. The user checks this list and, if necessary, enters the desired stock quantity into the app. The app saves the desired stock quantity entered by the user and compares it with the current stock quantity to calculate the shortage of items. The calculation results are also displayed as a list to the user.

[1352] Furthermore, the terminal sends a list of missing items to the server, and the server uses the API of the e-commerce platform to search for the relevant items. The search results are sent to the terminal as specific product information and displayed for the user to review. This allows the user to easily purchase the missing items. This function allows users to efficiently manage inventory and streamline purchasing procedures.

[1353] Examples:

[1354] For example, if a user notices that they are low on milk in the refrigerator while preparing breakfast, they can launch the app and take a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm identifies the type and quantity of each item (e.g., milk, eggs, butter, etc.) in the refrigerator.

[1355] The analysis reveals that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system would automatically list the missing bottle of milk. This list would be displayed on the terminal for the user to review. The server would then use the API of the e-commerce platform to search for available products of the missing milk and send the results to the terminal. The user could then select from the list and easily complete the purchase process.

[1356] The system allows users to efficiently manage their home inventory and automatically replenish items as needed.

[1357] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1358] Step 1:

[1359] The user launches the app on their smartphone. They select the inventory management function from the app's main screen and move to the image capture screen. At this time, the device's camera function is activated and a capture button is displayed.

[1360] Input: User actions

[1361] Output: Activating the camera function and displaying the shooting screen

[1362] Step 2:

[1363] When the user takes a photo of a home device or storage location, the device acquires the image data and temporarily stores it in its internal memory.

[1364] Input: An image taken by the user

[1365] Output: Image data stored in the device's memory

[1366] Step 3:

[1367] The device compresses the captured image data and sends it to the server's API endpoint, where it compresses the image data into JPEG format and sends it to the server via the Internet.

[1368] Input: Image data stored in memory

[1369] Output: Compressed image data sent to the server

[1370] Step 4:

[1371] The server analyzes the received image data. It uses an AI object detection algorithm (e.g., YOLO, SSD) to identify the objects in the image. The server determines the type and number of objects and structures the results in JSON format.

[1372] Input: Compressed image data

[1373] Output: Parsed results in JSON format (e.g., {"Milk": 2, "Egg": 8})

[1374] Step 5:

[1375] The server sends the parsed results in JSON format to the device, which receives them and stores them as internal data.

[1376] Input: Analysis results sent from the server

[1377] Output: Analysis results stored on the device

[1378] Step 6:

[1379] The device deserializes the analysis results and displays them to the user as a list, showing the type and quantity of each item.

[1380] Input: Parsed result in JSON format

[1381] Output: Inventory information displayed to the user (e.g., Milk: 2 bottles, Eggs: 8)

[1382] Step 7:

[1383] The user inputs the desired stock quantity into the app. The device displays the stock setting screen, and when the user inputs the desired stock quantity for each item, it is saved in the app.

[1384] Input: The desired stock quantity entered by the user

[1385] Output: Desired stock quantity saved in the app

[1386] Step 8:

[1387] The terminal compares the current stock with the stock desired by the user and lists the items that are in short supply. As a result of the calculation, a list of items that are in short supply is generated.

[1388] Input: Current stock quantity and desired stock quantity

[1389] Output: List of items in short supply (e.g. Milk: 1 bottle missing)

[1390] Step 9:

[1391] The terminal sends a list of missing items to the server, which then searches for the relevant items on the e-commerce platform and retrieves product information using the API of each platform.

[1392] Input: List of missing items

[1393] Output: Product information obtained from the e-commerce platform

[1394] Step 10:

[1395] The server sends the acquired product information to the terminal, which then displays it to the user, who can then check the displayed product information.

[1396] Input: Product information obtained from the e-commerce platform

[1397] Output: Product information displayed to the user

[1398] Step 11:

[1399] The user completes the purchase process. When the user presses the purchase button, the terminal redirects them to the purchase page of the e-commerce platform to complete the purchase process.

[1400] Input: User purchase operation

[1401] Output: Purchase procedure completed

[1402] (Application example 1)

[1403] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1404] In modern brick-and-mortar stores, inventory management is a time-consuming and labor-intensive task. Accurately tracking inventory levels on shelves and display areas and replenishing them in a timely manner are particularly important and directly affect customer satisfaction. However, traditional manual inventory checks are inefficient and carry a high risk of errors. Therefore, a more efficient and accurate method of inventory management is needed.

[1405] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1406] In this invention, the server includes means for a user to take images of home appliances, storage locations, shelves, and display corners in a store, means for transmitting the images to the server, means for the server to analyze the images and identify the type and number of items, and means for transmitting the analysis results to a terminal. This enables efficient and accurate inventory management in a physical store and enables a quick listing and notification of products that need to be replenished.

[1407] "Home appliances" is a general term for electrical and electronic devices used in the home.

[1408] "Storage area" refers to a space or container for storing items.

[1409] A "server" is a computer system that stores data and provides services to clients on a network.

[1410] "Image analysis" is the process of computer-based analysis of digital images to identify objects and features within the images.

[1411] "Goods" is a general term that refers to tangible goods or merchandise.

[1412] A "species" is a taxonomic group with different characteristics or properties.

[1413] "Quantity" indicates the quantity of an item.

[1414] A "terminal" is a computer device or smart device that a user operates.

[1415] "Inventory" means the total quantity of items currently stored or on display.

[1416] "Listing" is the act of displaying a list of items based on specific criteria.

[1417] A "shopping site" is a website where you can purchase goods and services over the Internet.

[1418] A "shelf" is a structure for displaying and storing items.

[1419] A "display corner" is an area within a store where items are displayed and sold.

[1420] An "AI object detection algorithm" is a computational method for identifying objects in an image using artificial intelligence technology.

[1421] "Management" means organizing and operating things according to a purpose.

[1422] "Replenishing" is the act of adding items that are lacking.

[1423] This invention relates to a system for streamlining store inventory management. This system uses image analysis technology and AI object detection algorithms to grasp the inventory status of items on shelves and display corners in a store and provides users with a list of items that are in short supply.

[1424] System configuration

[1425] 1. Photography Method:

[1426] The user takes an image of the shelf or display area using the camera on their smartphone. There are no particular restrictions on the smartphone as long as it has a general camera function.

[1427] 2. Image transmission method:

[1428] The image is sent to the server. The smartphone sends the captured image data to the server via an Internet connection. An HTTP request library (e.g., requests) is used.

[1429] 3. Image analysis methods:

[1430] The server analyzes the images using an AI object detection algorithm to identify the type and number of items, preferably with advanced GPU capabilities to run specific object detection models (e.g., YOLO, SSD).

[1431] 4. Data transmission and reception means:

[1432] The server sends the analysis results to the smartphone, which receives them. The analyzed product information is sent and received using JSON format data.

[1433] 5. Inventory Display Method:

[1434] The smartphone displays the analysis results on its screen. The user can visualize the inventory status of the items through the application. A common front-end framework (e.g., React Native, Flutter) is used to provide the user interface (UI).

[1435] 6. Inventory setting method:

[1436] The user sets the desired stock quantity, which is stored in the application and used as a basis for comparison.

[1437] 7. Listing Methods:

[1438] The smartphone compares the desired inventory quantity with the current inventory quantity and creates a list of items that are in short supply based on the inventory data received from the server.

[1439] 8. Search Methods:

[1440] The smartphone sends the entire list of missing items to a server, which then searches for the items via the API of the shopping site.

[1441] 9. Purchasing Method:

[1442] The user purchases an item through the search results, and the smartphone redirects the user to a purchase page on the shopping site.

[1443] Hardware and software used

[1444] Hardware:

[1445] Smartphone (camera function and internet connection)

[1446] Server (with GPU functionality)

[1447] software:

[1448] Camera API

[1449] HTTP request library (e.g., requests)

[1450] AI object detection models (e.g., YOLO, SSD)

[1451] JSON Data Format

[1452] User interface frameworks (e.g., React Native, Flutter)

[1453] Shopping site API

[1454] Specific examples

[1455] For example, consider a situation where a store clerk uses a smartphone app to check the inventory in the candy section. When the app takes a photo of the shelves, the AI ​​object detection algorithm automatically recognizes the type and quantity of each item on the shelves and lists any items that are missing. Based on the analysis results, the server searches for identical or similar products on the shopping site and presents the search results to the user. This allows the clerk to efficiently manage inventory and quickly replenish products as needed.

[1456] Prompt Sentence Examples

[1457] A store inventory management app is used to check the stock in the snack section. When a store associate takes a photo of the shelf with the smartphone app, the app automatically recognizes the type and quantity of each item and lists any items that are low in stock. Specifically, if it finds that potato chips are out of stock, the app notifies the associate. Please describe in detail a situation in which the associate can efficiently check the stock based on this prompt.

[1458] The above is a specific embodiment for carrying out the present invention.

[1459] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1460] Step 1:

[1461] The user takes pictures of home appliances, storage areas, shelves, and display corners in a store.

[1462] Users use the camera function of their smartphone to take photos of home appliances, storage areas, or shelves and display areas in stores. These images are used as visual data for the items to be managed in inventory.

[1463] Input: Image taken with a smartphone camera

[1464] Output: Digital image data

[1465] Step 2:

[1466] The terminal transmits the image to the server.

[1467] The device uses an HTTP request over its internet connection to send the captured image data to the server, where it is compressed and encoded as appropriate and sent to the server's API.

[1468] Input: Digital image data

[1469] Output: Image data sent to the server

[1470] Step 3:

[1471] The server analyzes the image and identifies the type and number of items.

[1472] The server analyzes the received image data using an AI object detection algorithm (e.g., YOLO, SSD), which identifies the type and number of each item in the image.

[1473] Input: Image data sent to the server

[1474] Output: Analysis results regarding the type and number of items

[1475] Step 4:

[1476] The server sends the analysis results to the terminal.

[1477] The server structures the analysis results as JSON data and sends it to the terminal via an HTTP request. This JSON data includes the type and quantity of each item.

[1478] Input: Analysis results regarding type and number of items

[1479] Output: JSON data of the analysis results sent to the terminal

[1480] Step 5:

[1481] The terminal displays the analysis results on the screen.

[1482] The terminal deserializes the JSON data received from the server and displays the inventory status of the items on the screen through a user interface, where the user can check the type and quantity of each item.

[1483] Input: JSON data of analysis results

[1484] Output: Inventory information displayed on a smartphone screen

[1485] Step 6:

[1486] Set the stock quantity desired by the user

[1487] The user enters the desired inventory quantity in the application, for example, 12 eggs or 3 bottles of milk, and this information becomes the baseline data.

[1488] Input: The number of items in stock desired by the user

[1489] Output: The desired stock quantity stored in the application

[1490] Step 7:

[1491] The terminal compares the desired inventory with the current inventory and lists the items that are in short supply.

[1492] The terminal compares the current inventory data with the desired inventory quantity set by the user, calculates the shortage, and lists it. The list of shortage items is used in the next step.

[1493] Input: Desired stock quantity and current stock quantity

[1494] Output: List of missing items

[1495] Step 8:

[1496] The server searches various shopping sites for the same or similar products of the listed items.

[1497] The server uses the API of each shopping site to search for the same or similar products based on the list of missing items, and the search results are sent to the terminal.

[1498] Input: List of missing items

[1499] Output: Search results from a shopping site

[1500] Step 9:

[1501] The terminal displays the search results on the screen.

[1502] The terminal displays the search results received from the server, and the user can use them as a reference to purchase the necessary items.

[1503] Input: Search results sent from the server

[1504] Output: Search results displayed on a smartphone screen

[1505] Step 10:

[1506] The user purchases an item through the search result.

[1507] The user can then proceed to purchase the selected item within the application, and the device will redirect the user to the shopping site's purchase page.

[1508] Input: Select a user based on search results

[1509] Output: Purchase completed

[1510] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1511] The present invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchase suggestions.

[1512] System Overview

[1513] The system includes the following main elements:

[1514] 1. Filming Method

[1515] 2. Image transmission method

[1516] 3. Image analysis methods

[1517] 4. Data transmission and reception means

[1518] 5. Inventory Display Method

[1519] 6. Inventory Setting Method

[1520] 7. How to create a list of items that are in short supply

[1521] 8. Product Search Methods

[1522] 9. Purchasing Method

[1523] 10. Emotion Engine

[1524] Program processing

[1525] User operations

[1526] Launching the app

[1527] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[1528] photo shoot

[1529] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[1530] Inventory Settings

[1531] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[1532] Emotion input

[1533] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. Emotional data is collected by the user's voice input or facial expressions displayed through the camera. This data is used for inventory management and purchasing suggestions.

[1534] Image analysis and data processing

[1535] Image transmission

[1536] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[1537] Image analysis

[1538] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[1539] Counting items

[1540] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them.

[1541] Display inventory information and check shortages

[1542] View inventory information

[1543] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[1544] Listing shortages

[1545] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[1546] View the missing items list

[1547] The terminal generates a list of the missing items and displays it to the user.

[1548] Product search and purchase

[1549] Shopping site search

[1550] The terminal sends a list of missing items to the server, which then uses the APIs of various shopping sites (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[1551] Sending product information

[1552] The server extracts related product information from search results on the shopping site, and sends the product information (product name, price, link, etc.) to the device.

[1553] Purchase procedure

[1554] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[1555] Use of emotion engine

[1556] Emotional Data Analysis and Applications

[1557] The emotion engine analyzes the user's voice and facial expression data in real time to recognize their emotions. This emotion data is then applied to inventory management and purchasing suggestions. For example, if the user is feeling stressed, it will suggest products that will help them relax.

[1558] Recording frequency of use and emotions

[1559] The emotion engine accumulates and records the user's emotional data and predicts their long-term consumption trends, enabling optimal product recommendations based on the user's preferences and tendencies.

[1560] Specific examples

[1561] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[1562] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[1563] Also, if a user is feeling stressed during busy morning hours, the emotion engine will recognize this and suggest products that will help reduce stress (such as relaxing herbal tea). In this way, appropriate product suggestions are made based on the user's emotions.

[1564] This system not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows them to receive optimal product suggestions based on their emotions.

[1565] The processing flow will be explained below.

[1566] Step 1: Launch the app

[1567] The user launches the app on their smartphone.

[1568] Step 2: Take a photo

[1569] The user takes a photo of the inside of the refrigerator or shelves.

[1570] The device will launch the camera function and display the capture button.

[1571] The user presses the capture button and an image is captured.

[1572] Step 3: Send image

[1573] The terminal compresses the captured image data and prepares it for transmission.

[1574] The device sends image data to the server's API endpoint.

[1575] Step 4: Image analysis

[1576] The server receives the image data.

[1577] The server uses AI object detection algorithms to identify items in the image.

[1578] The server extracts the type and location information of the identified item.

[1579] Step 5: Counting items

[1580] The server counts the number of each item.

[1581] The server structures the counting results in JSON format and sends them to the terminal.

[1582] Step 6: View inventory information

[1583] The device deserializes the data received from the server.

[1584] The device will display a list of items and their quantities on the app.

[1585] Step 7: Inventory Setup

[1586] The user enters the desired stock quantity into the app.

[1587] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[1588] Step 8: Enter emotions

[1589] The user displays facial expressions through voice input or a camera.

[1590] The device sends voice data and facial expression images to the emotion engine.

[1591] Step 9: Sentiment Analysis

[1592] The server's emotion engine analyzes voice data and facial expression images in real time to recognize the user's emotions.

[1593] The server stores the recognized emotion data and uses it for subsequent inventory management and purchasing suggestions.

[1594] Step 10: List the missing items

[1595] The terminal compares the current stock quantity with the desired stock quantity.

[1596] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[1597] Step 11: View the Shortage List

[1598] The terminal generates a list of the missing items and displays it to the user.

[1599] Step 12: Search shopping sites

[1600] The terminal transmits the list of missing items to the server.

[1601] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[1602] Step 13: Submit your product information

[1603] The server extracts related product information from search results on the shopping site.

[1604] The server sends product information (product name, price, link, etc.) to the terminal.

[1605] Step 14: View product information and checkout

[1606] The product information received by the device is displayed on the app.

[1607] The user selects the product they want to purchase and presses the purchase button.

[1608] The device will redirect you to the shopping site's purchase page.

[1609] Step 15: Emotion-based suggestions

[1610] Based on the user's emotional data recognized by the emotion engine, the system suggests products that are best suited to the user's current emotional state, such as relaxing drinks or health foods.

[1611] The terminal displays emotion-based product suggestions to the user.

[1612] Step 16: Analyze long-term consumption trends

[1613] The emotion engine accumulates and records users' emotional data and analyzes their long-term consumption trends.

[1614] The server optimizes weekly or monthly purchase suggestions based on the analysis results.

[1615] In this way, the user can efficiently manage inventory, smoothly purchase the items he or she needs, and furthermore, receive appropriate product suggestions according to the user's emotional state.

[1616] Example 2

[1617] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1618] Inventory management and replenishment of home appliances and storage spaces are important issues for modern households. In particular, there is a need for a system that quickly and accurately replenishes items when they run out, especially in busy daily lives. However, conventional inventory management systems rely heavily on manual input by users, making efficient management difficult. Furthermore, they lack functionality for optimal product recommendations based on users' emotions and preferences. This can lead to inconvenient situations where necessary items cannot be properly replenished. Furthermore, it is difficult to provide services that respond to users' individual needs and emotional changes.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1620] In this invention, the server includes means for a user to take an image of the home appliance or storage location, means for transmitting the image to the server, means for the server to analyze the image and identify the type and number of items, means for transmitting the analysis results to a terminal, means for displaying the analysis results on a terminal screen, means for a user to set a desired inventory quantity, means for comparing the desired inventory quantity with the current inventory quantity and listing items that are in short supply, means for searching various e-commerce sites for products that are identical to or similar to the listed items, means for displaying the search results on the terminal, means for analyzing user emotion data and applying the emotion data to inventory management and purchase suggestions, and means for a user to purchase items based on the search results. This enables efficient inventory management and rapid replenishment of items, and also realizes optimal product suggestions based on the user's emotions and preferences.

[1621] "User" refers to a person who operates the system, takes pictures of home appliances and storage locations, and issues various instructions regarding inventory management and purchasing of goods.

[1622] "Home appliances" refers to electrical appliances used to store household items, such as refrigerators and shelves.

[1623] "Storage space" refers to a place or facility for storing items, such as the inside of a refrigerator or a shelf.

[1624] "Images" refers to photos or videos taken by the user of the inside of an appliance or storage space.

[1625] The term "server" refers to a computer system that receives and analyzes image data sent by a user.

[1626] "Goods" refers to goods or products stored in appliances or storage areas, including food items such as milk and eggs.

[1627] "Quantity" refers to the quantity of a particular item.

[1628] "Terminal" refers to a mobile device such as a smartphone or tablet used by a user.

[1629] "Screen" refers to the display shown on the terminal.

[1630] "Inventory" refers to the quantity of items currently in stock.

[1631] "Desired inventory quantity" refers to the quantity of items desired by the user to be stored.

[1632] "Listing" refers to comparing the current inventory with the desired inventory and listing items that are in short supply.

[1633] "E-commerce site" refers to a website for buying and selling products online, including, for example, Amazon, Yahoo, and Rakuten.

[1634] "Emotion data" refers to data related to emotions collected from the user's voice and facial expressions.

[1635] "Purchase method" refers to the system, such as online payment or shopping cart, that a user uses to purchase goods.

[1636] "Machine learning object detection algorithm" refers to a machine learning or artificial intelligence algorithm used to automatically identify objects in images, such as YOLO or SSD.

[1637] This invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchasing suggestions.

[1638] The system includes the following main elements:

[1639] 1. A way for users to take pictures of home appliances and storage locations

[1640] 2. Means of sending the captured image to the server

[1641] 3. Means by which the server analyzes the image and identifies the type and number of items

[1642] 4. Means of sending analysis results to the device

[1643] 5. How to display the analysis results on the device screen

[1644] 6. A way for users to set their desired stock quantity

[1645] 7. A method for comparing desired inventory with current inventory and listing items that are in short supply

[1646] 8. A means to search for the same or similar products of the listed items on various e-commerce sites.

[1647] 9. How to display search results on your device

[1648] 10. Means of analyzing user sentiment data and applying sentiment data to inventory management and purchase recommendations

[1649] 11. How users can purchase items through search results

[1650] Users launch the app on their smartphone or other device and take a photo of the contents of their refrigerator or shelves. The device compresses the image and sends it to the server. The server then uses an AI object detection algorithm (e.g., YOLO or SSD) to identify the type and number of items in the image and sends the analysis results back to the device.

[1651] Users can check the analysis results on the device screen and set the desired inventory quantity as needed. The device compares the current inventory quantity with the desired inventory quantity and lists the items that are in short supply. For the listed items, the server uses the APIs of various e-commerce sites (e.g., Amazon, Yahoo, Rakuten) to search for similar products and send the search results to the device.

[1652] The emotion engine analyzes the user's voice and facial expressions to collect emotional data. This emotional data is then applied to inventory management and purchase suggestions. For example, if a user is feeling stressed, it will suggest products that have a relaxing effect. The user can select the suggested products on the screen and proceed with the purchase.

[1653] As a concrete example, suppose a user is preparing breakfast and notices that there is little milk in the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm is used to identify the type and number of bottles of milk. The analysis results show that there are only two bottles of milk left in the refrigerator. Since the user's desired stock is three bottles, the system lists the one bottle that is missing.

[1654] The server uses the API of the e-commerce site to search for the missing item and display it to the user. Furthermore, the emotion engine recognizes that the user is feeling stressed while preparing breakfast and suggests relaxing herbal teas.

[1655] The user can select the appropriate product from the options presented and quickly complete the purchase procedure.

[1656] Example prompts to input to the generative AI model

[1657] 1. Inventory count prompt:

[1658] "When the user takes a photo of the refrigerator, count the number of bottles of milk using YOLO and return the result in JSON format."

[1659] 2. Prompt to generate a list of missing items:

[1660] "Compare the current stock quantity data with the user's desired stock quantity data and generate a list of items that are in short supply."

[1661] 3. Emotion-Based Product Recommendation Prompt:

[1662] "Analyze the collected user emotional data and suggest products that have a relaxing effect when the user is feeling stressed."

[1663] As a result, a system is realized that not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows users to receive optimal product suggestions based on their emotions.

[1664] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1665] Explain the process step by step

[1666] User operations

[1667] Step 1: Launch the app

[1668] A user launches an app on their smartphone. The input is a touch operation, and the output is the main screen of the app.

[1669] The device loads configuration files and user data in the background and prepares to operate normally.

[1670] Step 2: Take a photo

[1671] The user takes a photo of the inside of the refrigerator or shelf. The device's camera function is used as input, and image data is generated as output.

[1672] The device activates the camera function, and when the user presses the capture button, a photo is taken. This image data is temporarily stored in the device's memory.

[1673] Step 3: Inventory Setup

[1674] The user manually inputs the desired inventory quantity into the app, and the desired inventory data is generated as output.

[1675] The terminal displays an inventory setting screen, and the user inputs the items they need (e.g., 3 bottles of milk, 12 eggs). This information is stored in the terminal's internal database.

[1676] Step 4: Enter emotions

[1677] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. The user's voice and facial expression data are used as input, and emotion data is generated as output.

[1678] The device collects and temporarily stores the user's facial expressions through voice input and a camera, and this data is later sent to a server.

[1679] Image analysis and data processing

[1680] Step 5: Send image

[1681] The terminal sends the captured image to the server. The input is image data, and the output is compressed image data sent to the server.

[1682] The device compresses the image data and sends it to the server's API endpoint.

[1683] Step 6: Image analysis

[1684] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The input is image data, and the output is data on the type and number of objects.

[1685] The server receives the image data and runs an object detection algorithm to extract the type and location of the item, and stores the results in a database.

[1686] Step 7: Counting items

[1687] The server counts the number of each item and sends the result to the terminal. The input is the identified item data, and the output is the counted item data.

[1688] The server categorizes the items, counts the number of each item, and structures the results in JSON format. It then checks the data for integrity before sending it to the device.

[1689] Display inventory information and check shortages

[1690] Step 8: View inventory information

[1691] The terminal displays a list of items and their quantities. The input is the item count data, and the output is the inventory list displayed on the screen.

[1692] The terminal deserializes the received data and displays it in a list format on the screen, allowing the user to check the current inventory status.

[1693] Step 9: List the missing items

[1694] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists the items that are in short supply. The input is the desired inventory data and the current inventory data, and the output is a list of items that are in short supply.

[1695] The terminal compares the two and calculates the shortage. For example, if the desired stock is 3 bottles of milk and the current stock is 2 bottles, the shortage will be 1 bottle.

[1696] Step 10: View the Missing Items List

[1697] The terminal displays the generated shortage list. The input is the shortage list data, and the shortage list is displayed on the screen as the output.

[1698] The shortage list is displayed in a visually easy-to-understand format, allowing users to quickly take next steps.

[1699] Product search and purchase

[1700] Step 11: Search for shopping sites

[1701] The terminal sends the list of items in short supply to the server. The input is the list of items in short supply, and the output is the search results for similar items.

[1702] The server uses the APIs of various e-commerce sites to search for products that correspond to the missing items and sends the results to the terminal.

[1703] Step 12: Submit your product information

[1704] The server sends related product information from search results on a shopping site to the terminal. The input is search result data, and the output is product information.

[1705] Data including product information (product name, price, link, etc.) is sent to the terminal. The terminal analyzes the received data and displays it to the user in an appropriate format.

[1706] Step 13: Checkout

[1707] The user selects a product on the app and completes the purchase process. The input is product selection data, and the output is purchase confirmation data.

[1708] The terminal displays the received product information, and when the user presses the purchase button, the terminal redirects the user to the shopping site's purchase page. Once the purchase procedure is complete, the terminal displays a confirmation message.

[1709] Use of emotion engine

[1710] Step 14: Analyze and apply emotion data

[1711] The emotion engine analyzes the user's voice and facial expression data in real time to recognize emotions. The input is voice and facial expression data, and the output is emotional data.

[1712] For example, if a user is feeling stressed, the system will suggest products that will help them relax (e.g., herbal tea), which will then be applied to inventory management and purchasing suggestions.

[1713] Step 15: Record frequency of use and emotions

[1714] The emotion engine accumulates and records the user's emotional data and predicts the user's long-term consumption trends. The input is the accumulated emotional data, and the output is predicted consumption trends.

[1715] For example, it is possible to identify the time periods during which a user is likely to feel stressed and suggest products suitable for those time periods.

[1716] Through these steps, users can efficiently manage their inventory, quickly replenish needed items, and receive optimal product recommendations based on their emotions.

[1717] (Application example 2)

[1718] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1719] In recent years, there has been a demand for improved cargo management efficiency in autonomous vehicles. However, with conventional inventory management systems, drivers must manually check inventory levels, identify missing items, and then carry out replenishment procedures, which requires time and effort. Furthermore, simply indicating missing items without considering the driver's feelings increases stress and fatigue for the driver, resulting in a decrease in work efficiency. A system that solves these issues, performs efficient inventory management, and reduces the burden on drivers is needed.

[1720] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for the user to take an image of the storage location, a means for transmitting the image to the server, and a means for the server to analyze the image and identify the type and number of items. This allows the driver to easily check the inventory status of the cargo space using the wearable device, as well as identify missing items in real time and automatically receive replenishment suggestions. In addition, the system includes a means for recognizing the driver's emotions and a means for suggesting appropriate products based on the recognized emotion data, thereby reducing the driver's stress and allowing them to work efficiently and comfortably.

[1721] "Means for taking images" refers to a function that allows a user to take images of a storage area or cargo space using a smart device or wearable device.

[1722] The "means for transmitting images to a server" is a function for transmitting captured image data to a server via a network such as the Internet.

[1723] "Means for identifying the type and number of items" refers to a function that analyzes the image received by the server and identifies the type and number of items in the image using an AI object detection algorithm.

[1724] "Means for transmitting analysis results to a terminal" refers to a function for transmitting data analyzed by the server to a user's terminal, particularly a smart device or wearable device.

[1725] The "means for displaying the analysis results on the display device of the terminal" is a function for visually displaying the analysis results on the screen of the user's terminal.

[1726] The "means for setting the stock quantity desired by the user" is a function that allows the user to input the stock quantity desired by the user and record it in the system.

[1727] The "means for listing items in short supply" is a function that compares the set desired inventory quantity with the current inventory quantity and displays the items in short supply in a list format.

[1728] "Means for searching for identical or similar products in various shopping systems" refers to a function for searching online shopping sites for products that are identical or similar to the missing item.

[1729] The "means for displaying search results on a terminal" is a function for displaying product search results obtained from a shopping site on a user's terminal.

[1730] The "means for purchasing goods" is a function for a user to complete the purchase procedure for the product selected by the user through the terminal.

[1731] "Means for managing items in the cargo space, including a wearable device" refers to a function that uses a wearable device worn by the driver to manage inventory in the cargo space.

[1732] The "means for recognizing emotions" is a function that analyzes the user's facial expressions and tone of voice through the wearable device's camera or voice input, and recognizes their emotions.

[1733] The "means for suggesting appropriate products based on recognized emotional data" is a function that suggests products that have stress-reducing or relaxing effects based on the user's emotional data.

[1734] The "means for displaying the proposed results" is a function for displaying the products and information proposed to the user on the terminal.

[1735] This invention relates to a system for improving the efficiency of cargo management in autonomous vehicles. Specifically, the system takes images of the cargo space, identifies the type and number of items using an AI object detection algorithm, and performs inventory management while recognizing the driver's emotions to suggest optimal products. This system is intended to be operated by the user using a wearable device.

[1736] Hardware Configuration

[1737] 1. Wearable devices (e.g., smart glasses)

[1738] It is worn by the driver and takes pictures of the cargo space.

[1739] It has a built-in camera and microphone to collect image and audio data.

[1740] 2. Cloud Server

[1741] Performs image analysis, object detection, and emotion recognition

[1742] Responsible for storing and processing data

[1743] 3. Smart Devices

[1744] Driver-carried smartphones and tablets

[1745] Communicates with the server and displays analysis results and product suggestions

[1746] Software Configuration

[1747] 1. AI object detection algorithms (e.g., YOLO)

[1748] Analyze images captured by the wearable device to identify the type and quantity of cargo

[1749] 2. Sentiment analysis algorithms (e.g., EmotionAnalyzer)

[1750] Analyzes emotions based on the driver's facial expressions and voice data captured through cameras and microphones

[1751] 3. Data transmission and reception module

[1752] It serves as a data transfer mechanism between wearable devices and cloud servers.

[1753] 4. Inventory Management Module

[1754] Manage inventory data on the server

[1755] 5. Proposal Module

[1756] Recommend products to drivers based on emotion data

[1757] Detailed process flow

[1758] 1. Image capture and transmission

[1759] A user takes an image of the cargo space using a wearable device

[1760] The captured image is sent to the server

[1761] 2. Image Analysis

[1762] The server uses AI object detection algorithms to identify objects in the image and determine their type and quantity.

[1763] 3. Inventory Management

[1764] The user inputs the desired inventory quantity into the smart device, and the server compares it with the current inventory to identify any shortages.

[1765] 4. Emotion Recognition and Product Recommendations

[1766] Using the camera and microphone of the wearable device, the driver's facial expressions and voice data are analyzed with an emotion analysis algorithm to recognize their emotions.

[1767] Makes product recommendations based on emotions and displays the results on a smart device

[1768] Specific examples

[1769] While replenishing cargo, the driver uses smart glasses to take images of the cargo space. The images are sent to a server, where an AI object detection algorithm analyzes inventory shortages. At the same time, if the driver's stress level is high, suggestions for relaxation products (e.g., herbal tea) are displayed.

[1770] Prompt Sentence Examples

[1771] "Check stock availability"

[1772] "Please suggest a relaxing product."

[1773] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1774] Step 1:

[1775] A user takes an image of the cargo space using a wearable device (smart glasses).

[1776] Input: Cargo space image

[1777] Output: Captured image data

[1778] Specific operation: Using the camera function of the smart glasses, the user faces the cargo space and presses the capture button to capture an image.

[1779] Step 2:

[1780] The device sends the captured image to a cloud server.

[1781] Input: Photographed image data

[1782] Output: Image data sent to the server

[1783] Specific operation: The transmission module on the smart glasses is activated and sends the image data to the server via the Internet.

[1784] Step 3:

[1785] The server uses an AI object detection algorithm to analyze the image and identify the type and number of items.

[1786] Input: Image data sent to the server

[1787] Output: Data on type and number of items

[1788] Specific operation: The server runs image analysis software (e.g., YOLO) to identify and count the items, and stores the results in a database.

[1789] Step 4:

[1790] The server sends the analysis results to the user's smart device.

[1791] Input: Data on type and quantity of items

[1792] Output: Analysis data sent to smart device

[1793] Specific operation: The server sends the generated item list to the user's smartphone or tablet via the data communication module.

[1794] Step 5:

[1795] The terminal displays the analysis results on a display device.

[1796] Input: Submitted analysis data

[1797] Output: Inventory information displayed on screen

[1798] Specific operation: The display module of the smart device displays the analysis results in list format on the screen so that the user can check them.

[1799] Step 6:

[1800] The user sets the desired stock quantity via a smart device.

[1801] Input: The desired stock quantity entered by the user

[1802] Output: Desired inventory quantity data

[1803] Specific operation: The user enters the desired stock quantity of each item on the application screen of their smart device, and the data is saved within the app.

[1804] Step 7:

[1805] The server compares the current stock with the desired stock and lists the items that are in short supply.

[1806] Input: Current stock quantity data and desired stock quantity data

[1807] Output: List of missing items

[1808] Specific operation: The server compares the stored inventory data with the desired inventory data, calculates the difference, and generates a list of items that are in short supply.

[1809] Step 8:

[1810] The server searches various shopping systems for the same or similar products as the missing item.

[1811] Input: List of missing items

[1812] Output: Shopping system search results

[1813] Specific operation: The server uses the API of each shopping system (e.g., Amazon, Rakuten) to search for products similar to the missing item.

[1814] Step 9:

[1815] The server sends the search results to the terminal.

[1816] Input: Shopping system search results

[1817] Output: Search results sent to your device

[1818] Specific operation: The server organizes the search results and sends them to the user's smart device.

[1819] Step 10:

[1820] The terminal displays the search results to the user, and the user purchases the item.

[1821] Input: Submitted search results

[1822] Output: User completes purchase

[1823] Specific operation: The user selects a product from the search results displayed through the smart device application and presses the purchase button to complete the purchase process.

[1824] Step 11:

[1825] The wearable device on the terminal recognizes the driver's emotions and transmits the emotional data to a server.

[1826] Input: Driver's facial expressions and voice data

[1827] Output: Emotion data sent to the server

[1828] Specific operation: Emotion data collected through the camera and microphone of the wearable device is analyzed and sent to the server.

[1829] Step 12:

[1830] The server suggests appropriate products based on the recognized emotion data and displays them on the device.

[1831] Input: Emotion data

[1832] Output: Appropriate product suggestions

[1833] How it works: The server's emotion analysis algorithm (e.g., EmotionAnalyzer) analyzes the emotion data and generates possible product suggestions. These suggestions are sent to the smart device and displayed to the user.

[1834] The above are the specific processing steps.

[1835] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1836] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1837] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1838] [Fourth embodiment]

[1839] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1840] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1841] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1842] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1843] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1844] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1845] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1846] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1847] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1848] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1849] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1850] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1851] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1852] This invention is a system that allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items, thereby managing inventory and automatically replenishing items.

[1853] System Overview

[1854] The system includes the following main elements:

[1855] 1. Filming Method

[1856] 2. Image transmission method

[1857] 3. Image analysis methods

[1858] 4. Data transmission and reception means

[1859] 5. Inventory Display Method

[1860] 6. Inventory Setting Method

[1861] 7. How to create a list of items that are in short supply

[1862] 8. Product Search Methods

[1863] 9. Purchasing Method

[1864] Program processing

[1865] User operations

[1866] Launching the app

[1867] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[1868] photo shoot

[1869] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[1870] Inventory Settings

[1871] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[1872] Image analysis and data processing

[1873] Image transmission

[1874] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[1875] Image analysis

[1876] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[1877] Counting items

[1878] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them to the terminal.

[1879] Display inventory information and check shortages

[1880] View inventory information

[1881] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[1882] Listing shortages

[1883] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[1884] View the missing items list

[1885] The terminal displays a list of the missing items to the user. A list of missing items is generated so that the user can check the missing items.

[1886] Product search and purchase

[1887] Shopping site search

[1888] The server searches for similar products on each shopping site based on the missing items. The terminal sends the list of missing items to the server, which then searches for the products using the shopping site's API.

[1889] Sending product information

[1890] The server searches for the relevant product and sends the information to the terminal. The product information is received from the search results and displayed on the terminal.

[1891] Purchase procedure

[1892] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[1893] Specific examples

[1894] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[1895] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[1896] This system allows users to efficiently manage inventory and smoothly replenish necessary items.

[1897] The processing flow will be explained below.

[1898] Step 1: Launch the app

[1899] The user launches the app on their smartphone.

[1900] Step 2: Take a photo

[1901] The user takes a photo of the inside of the refrigerator or shelves.

[1902] The device will launch the camera function and display the capture button.

[1903] The user presses the capture button and an image is captured.

[1904] Step 3: Send image

[1905] The terminal compresses the captured image data and prepares it for transmission.

[1906] The device sends image data to the server's API endpoint.

[1907] Step 4: Image analysis

[1908] The server receives the image data.

[1909] The server uses AI object detection algorithms to identify items in the image.

[1910] The server extracts the type and location information of the identified item.

[1911] Step 5: Counting items

[1912] The server counts the number of each item.

[1913] The server structures the counting results in JSON format and sends them to the terminal.

[1914] Step 6: View inventory information

[1915] The device deserializes the data received from the server.

[1916] The device will display a list of items and their quantities on the app.

[1917] Step 7: Inventory Setup

[1918] The user enters the desired stock quantity into the app.

[1919] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[1920] Step 8: List the missing items

[1921] The terminal compares the current stock quantity with the desired stock quantity.

[1922] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[1923] Step 9: View the Missing Items List

[1924] The terminal generates a list of the missing items and displays it to the user.

[1925] Step 10: Search shopping sites

[1926] The terminal transmits the list of missing items to the server.

[1927] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[1928] Step 11: Submit your product information

[1929] The server extracts related product information from search results on the shopping site.

[1930] The server sends product information (product name, price, link, etc.) to the terminal.

[1931] Step 12: View product information and checkout

[1932] The product information received by the device is displayed on the app.

[1933] The user selects the product they want to purchase and presses the purchase button.

[1934] The device will redirect you to the shopping site's purchase page.

[1935] By following the steps above, the user can efficiently manage inventory and smoothly purchase the items they need.

[1936] Example 1

[1937] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1938] In today's modern lifestyle, inventory management of household devices and storage spaces is extremely cumbersome, and managing consumables is particularly difficult. Users must regularly check the inventory of consumables and manually purchase the necessary items, which is time-consuming. Furthermore, inventory surpluses and shortages can lead to unnecessary expenses and the inconvenience of not having the items available when needed. An efficient system that solves these problems is needed.

[1939] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1940] In this invention, the server includes a means for a user to take an image of a home appliance or storage location, a means for transmitting the image to a processing device, and a means for the processing device to analyze the image and identify the type and number of items, thereby enabling the user to efficiently manage inventory in the home and automatically replenish necessary items.

[1941] "User" refers to the individual who operates the system and manages the inventory of home devices and storage locations.

[1942] "Household appliances" refers to devices used in the home to store items, such as refrigerators and shelves.

[1943] "Image" refers to still image data taken by a user of a home device or storage location.

[1944] "Processing device" refers to a computer device used to analyze image data and identify the type and number of items.

[1945] "Terminal" refers to a digital device that a user operates at hand, and specifically includes smartphones and tablets.

[1946] "AI object detection algorithm" refers to a machine learning model or deep learning algorithm that identifies objects in an image and determines their type and number.

[1947] "Inventory" refers to the quantity of a particular item in a household.

[1948] "Desired inventory quantity" refers to the target quantity of an item that a user wishes to keep in stock.

[1949] "Shortage items" refer to items whose current inventory is lower than the desired inventory.

[1950] An "e-commerce platform" refers to a website or app that allows people to purchase goods online.

[1951] "Search Results" refers to the product information searched for based on the missing items on the e-commerce platform.

[1952] This invention is a system for managing inventory and automatically replenishing items in the home. The system allows users to take images of home appliances and storage locations, and uses an AI object detection algorithm to identify the type and number of items and manage inventory information.

[1953] First, the user launches a dedicated app on a device such as a smartphone or tablet. The user selects the inventory management function within the app and activates the camera. The user then takes pictures of storage areas in the home, such as refrigerators and shelves. The device compresses the captured image data and sends it to a server via the Internet.

[1954] The server analyzes the received image data. Specifically, it uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image and determine their type and number. The analysis results are structured in JSON format as data including the type and number of objects, and sent to the device. This analysis is often performed by a server equipped with a high-performance GPU.

[1955] Next, the device deserializes the data received from the server and displays it as a list to the user. The user checks this list and, if necessary, enters the desired stock quantity into the app. The app saves the desired stock quantity entered by the user and compares it with the current stock quantity to calculate the shortage of items. The calculation results are also displayed as a list to the user.

[1956] Furthermore, the terminal sends a list of missing items to the server, and the server uses the API of the e-commerce platform to search for the relevant items. The search results are sent to the terminal as specific product information and displayed for the user to review. This allows the user to easily purchase the missing items. This function allows users to efficiently manage inventory and streamline purchasing procedures.

[1957] Examples:

[1958] For example, if a user notices that they are low on milk in the refrigerator while preparing breakfast, they can launch the app and take a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm identifies the type and quantity of each item (e.g., milk, eggs, butter, etc.) in the refrigerator.

[1959] The analysis reveals that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system would automatically list the missing bottle of milk. This list would be displayed on the terminal for the user to review. The server would then use the API of the e-commerce platform to search for available products of the missing milk and send the results to the terminal. The user could then select from the list and easily complete the purchase process.

[1960] The system allows users to efficiently manage their home inventory and automatically replenish items as needed.

[1961] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1962] Step 1:

[1963] The user launches the app on their smartphone. They select the inventory management function from the app's main screen and move to the image capture screen. At this time, the device's camera function is activated and a capture button is displayed.

[1964] Input: User actions

[1965] Output: Activating the camera function and displaying the shooting screen

[1966] Step 2:

[1967] When the user takes a photo of a home device or storage location, the device acquires the image data and temporarily stores it in its internal memory.

[1968] Input: An image taken by the user

[1969] Output: Image data stored in the device's memory

[1970] Step 3:

[1971] The device compresses the captured image data and sends it to the server's API endpoint, where it compresses the image data into JPEG format and sends it to the server via the Internet.

[1972] Input: Image data stored in memory

[1973] Output: Compressed image data sent to the server

[1974] Step 4:

[1975] The server analyzes the received image data. It uses an AI object detection algorithm (e.g., YOLO, SSD) to identify the objects in the image. The server determines the type and number of objects and structures the results in JSON format.

[1976] Input: Compressed image data

[1977] Output: Parsed results in JSON format (e.g., {"Milk": 2, "Egg": 8})

[1978] Step 5:

[1979] The server sends the parsed results in JSON format to the device, which receives them and stores them as internal data.

[1980] Input: Analysis results sent from the server

[1981] Output: Analysis results stored on the device

[1982] Step 6:

[1983] The device deserializes the analysis results and displays them to the user as a list, showing the type and quantity of each item.

[1984] Input: Parsed result in JSON format

[1985] Output: Inventory information displayed to the user (e.g., Milk: 2 bottles, Eggs: 8)

[1986] Step 7:

[1987] The user inputs the desired stock quantity into the app. The device displays the stock setting screen, and when the user inputs the desired stock quantity for each item, it is saved in the app.

[1988] Input: The desired stock quantity entered by the user

[1989] Output: Desired stock quantity saved in the app

[1990] Step 8:

[1991] The terminal compares the current stock with the stock desired by the user and lists the items that are in short supply. As a result of the calculation, a list of items that are in short supply is generated.

[1992] Input: Current stock quantity and desired stock quantity

[1993] Output: List of items in short supply (e.g. Milk: 1 bottle missing)

[1994] Step 9:

[1995] The terminal sends a list of missing items to the server, which then searches for the relevant items on the e-commerce platform and retrieves product information using the API of each platform.

[1996] Input: List of missing items

[1997] Output: Product information obtained from the e-commerce platform

[1998] Step 10:

[1999] The server sends the acquired product information to the terminal, which then displays it to the user, who can then check the displayed product information.

[2000] Input: Product information obtained from the e-commerce platform

[2001] Output: Product information displayed to the user

[2002] Step 11:

[2003] The user completes the purchase process. When the user presses the purchase button, the terminal redirects them to the purchase page of the e-commerce platform to complete the purchase process.

[2004] Input: User purchase operation

[2005] Output: Purchase procedure completed

[2006] (Application example 1)

[2007] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2008] In modern brick-and-mortar stores, inventory management is a time-consuming and labor-intensive task. Accurately tracking inventory levels on shelves and display areas and replenishing them in a timely manner are particularly important and directly affect customer satisfaction. However, traditional manual inventory checks are inefficient and carry a high risk of errors. Therefore, a more efficient and accurate method of inventory management is needed.

[2009] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2010] In this invention, the server includes means for a user to take images of home appliances, storage locations, shelves, and display corners in a store, means for transmitting the images to the server, means for the server to analyze the images and identify the type and number of items, and means for transmitting the analysis results to a terminal. This enables efficient and accurate inventory management in a physical store and enables a quick listing and notification of products that need to be replenished.

[2011] "Home appliances" is a general term for electrical and electronic devices used in the home.

[2012] "Storage area" refers to a space or container for storing items.

[2013] A "server" is a computer system that stores data and provides services to clients on a network.

[2014] "Image analysis" is the process of computer-based analysis of digital images to identify objects and features within the images.

[2015] "Goods" is a general term that refers to tangible goods or merchandise.

[2016] A "species" is a taxonomic group with different characteristics or properties.

[2017] "Quantity" indicates the quantity of an item.

[2018] A "terminal" is a computer device or smart device that a user operates.

[2019] "Inventory" means the total quantity of items currently stored or on display.

[2020] "Listing" is the act of displaying a list of items based on specific criteria.

[2021] A "shopping site" is a website where you can purchase goods and services over the Internet.

[2022] A "shelf" is a structure for displaying and storing items.

[2023] A "display corner" is an area within a store where items are displayed and sold.

[2024] An "AI object detection algorithm" is a computational method for identifying objects in an image using artificial intelligence technology.

[2025] "Management" means organizing and operating things according to a purpose.

[2026] "Replenishing" is the act of adding items that are lacking.

[2027] This invention relates to a system for streamlining store inventory management. This system uses image analysis technology and AI object detection algorithms to grasp the inventory status of items on shelves and display corners in a store and provides users with a list of items that are in short supply.

[2028] System configuration

[2029] 1. Photography Method:

[2030] The user takes an image of the shelf or display area using the camera on their smartphone. There are no particular restrictions on the smartphone as long as it has a general camera function.

[2031] 2. Image transmission method:

[2032] The image is sent to the server. The smartphone sends the captured image data to the server via an Internet connection. An HTTP request library (e.g., requests) is used.

[2033] 3. Image analysis methods:

[2034] The server analyzes the images using an AI object detection algorithm to identify the type and number of items, preferably with advanced GPU capabilities to run specific object detection models (e.g., YOLO, SSD).

[2035] 4. Data transmission and reception means:

[2036] The server sends the analysis results to the smartphone, which receives them. The analyzed product information is sent and received using JSON format data.

[2037] 5. Inventory Display Method:

[2038] The smartphone displays the analysis results on its screen. The user can visualize the inventory status of the items through the application. A common front-end framework (e.g., React Native, Flutter) is used to provide the user interface (UI).

[2039] 6. Inventory setting method:

[2040] The user sets the desired stock quantity, which is stored in the application and used as a basis for comparison.

[2041] 7. Listing Methods:

[2042] The smartphone compares the desired inventory quantity with the current inventory quantity and creates a list of items that are in short supply based on the inventory data received from the server.

[2043] 8. Search Methods:

[2044] The smartphone sends the entire list of missing items to a server, which then searches for the items via the API of the shopping site.

[2045] 9. Purchasing Method:

[2046] The user purchases an item through the search results, and the smartphone redirects the user to a purchase page on the shopping site.

[2047] Hardware and software used

[2048] Hardware:

[2049] Smartphone (camera function and internet connection)

[2050] Server (with GPU functionality)

[2051] software:

[2052] Camera API

[2053] HTTP request library (e.g., requests)

[2054] AI object detection models (e.g., YOLO, SSD)

[2055] JSON Data Format

[2056] User interface frameworks (e.g., React Native, Flutter)

[2057] Shopping site API

[2058] Specific examples

[2059] For example, consider a situation where a store clerk uses a smartphone app to check the inventory in the candy section. When the app takes a photo of the shelves, the AI ​​object detection algorithm automatically recognizes the type and quantity of each item on the shelves and lists any items that are missing. Based on the analysis results, the server searches for identical or similar products on the shopping site and presents the search results to the user. This allows the clerk to efficiently manage inventory and quickly replenish products as needed.

[2060] Prompt Sentence Examples

[2061] A store inventory management app is used to check the stock in the snack section. When a store associate takes a photo of the shelf with the smartphone app, the app automatically recognizes the type and quantity of each item and lists any items that are low in stock. Specifically, if it finds that potato chips are out of stock, the app notifies the associate. Please describe in detail a situation in which the associate can efficiently check the stock based on this prompt.

[2062] The above is a specific embodiment for carrying out the present invention.

[2063] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2064] Step 1:

[2065] The user takes pictures of home appliances, storage areas, shelves, and display corners in a store.

[2066] Users use the camera function of their smartphone to take photos of home appliances, storage areas, or shelves and display areas in stores. These images are used as visual data for the items to be managed in inventory.

[2067] Input: Image taken with a smartphone camera

[2068] Output: Digital image data

[2069] Step 2:

[2070] The terminal transmits the image to the server.

[2071] The device uses an HTTP request over its internet connection to send the captured image data to the server, where it is compressed and encoded as appropriate and sent to the server's API.

[2072] Input: Digital image data

[2073] Output: Image data sent to the server

[2074] Step 3:

[2075] The server analyzes the image and identifies the type and number of items.

[2076] The server analyzes the received image data using an AI object detection algorithm (e.g., YOLO, SSD), which identifies the type and number of each item in the image.

[2077] Input: Image data sent to the server

[2078] Output: Analysis results regarding the type and number of items

[2079] Step 4:

[2080] The server sends the analysis results to the terminal.

[2081] The server structures the analysis results as JSON data and sends it to the terminal via an HTTP request. This JSON data includes the type and quantity of each item.

[2082] Input: Analysis results regarding type and number of items

[2083] Output: JSON data of the analysis results sent to the terminal

[2084] Step 5:

[2085] The terminal displays the analysis results on the screen.

[2086] The terminal deserializes the JSON data received from the server and displays the inventory status of the items on the screen through a user interface, where the user can check the type and quantity of each item.

[2087] Input: JSON data of analysis results

[2088] Output: Inventory information displayed on a smartphone screen

[2089] Step 6:

[2090] Set the stock quantity desired by the user

[2091] The user enters the desired inventory quantity in the application, for example, 12 eggs or 3 bottles of milk, and this information becomes the baseline data.

[2092] Input: The number of items in stock desired by the user

[2093] Output: The desired stock quantity stored in the application

[2094] Step 7:

[2095] The terminal compares the desired inventory with the current inventory and lists the items that are in short supply.

[2096] The terminal compares the current inventory data with the desired inventory quantity set by the user, calculates the shortage, and lists it. The list of shortage items is used in the next step.

[2097] Input: Desired stock quantity and current stock quantity

[2098] Output: List of missing items

[2099] Step 8:

[2100] The server searches various shopping sites for the same or similar products of the listed items.

[2101] The server uses the API of each shopping site to search for the same or similar products based on the list of missing items, and the search results are sent to the terminal.

[2102] Input: List of missing items

[2103] Output: Search results from a shopping site

[2104] Step 9:

[2105] The terminal displays the search results on the screen.

[2106] The terminal displays the search results received from the server, and the user can use them as a reference to purchase the necessary items.

[2107] Input: Search results sent from the server

[2108] Output: Search results displayed on a smartphone screen

[2109] Step 10:

[2110] The user purchases an item through the search result.

[2111] The user can then proceed to purchase the selected item within the application, and the device will redirect the user to the shopping site's purchase page.

[2112] Input: Select a user based on search results

[2113] Output: Purchase completed

[2114] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2115] The present invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchase suggestions.

[2116] System Overview

[2117] The system includes the following main elements:

[2118] 1. Filming Method

[2119] 2. Image transmission method

[2120] 3. Image analysis methods

[2121] 4. Data transmission and reception means

[2122] 5. Inventory Display Method

[2123] 6. Inventory Setting Method

[2124] 7. How to create a list of items that are in short supply

[2125] 8. Product Search Methods

[2126] 9. Purchasing Method

[2127] 10. Emotion Engine

[2128] Program processing

[2129] User operations

[2130] Launching the app

[2131] The user launches the app on their smartphone, selects the inventory management function from the app's main screen, and moves to the image capture screen.

[2132] photo shoot

[2133] The user takes a photo of the inside of the refrigerator or shelf. The device activates the camera function and displays a capture button, which the user presses to capture the image.

[2134] Inventory Settings

[2135] The user inputs the desired inventory quantity into the app. The device displays the inventory setting screen, where the user inputs the required items (e.g., 3 bottles of milk, 12 eggs), and saves them in the app.

[2136] Emotion input

[2137] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. Emotional data is collected by the user's voice input or facial expressions displayed through the camera. This data is used for inventory management and purchasing suggestions.

[2138] Image analysis and data processing

[2139] Image transmission

[2140] The device sends the captured image to the server, which then compresses the image data and sends it to the server's API endpoint.

[2141] Image analysis

[2142] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The server receives the image data and runs the object detection algorithm to extract the object type and location information.

[2143] Counting items

[2144] The server counts the number of each item and sends the results to the terminal. The server classifies the identified items, counts the number of each, structures the results in JSON format, and sends them.

[2145] Display inventory information and check shortages

[2146] View inventory information

[2147] The device displays a list of items and their quantities on the app. The device deserializes the received data and displays it on the screen in list format.

[2148] Listing shortages

[2149] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists out any items that are in short supply.The terminal compares the current inventory data with the desired inventory quantity, calculates the number of items that are in short supply, and generates a list.

[2150] View the missing items list

[2151] The terminal generates a list of the missing items and displays it to the user.

[2152] Product search and purchase

[2153] Shopping site search

[2154] The terminal sends a list of missing items to the server, which then uses the APIs of various shopping sites (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[2155] Sending product information

[2156] The server extracts related product information from search results on the shopping site, and sends the product information (product name, price, link, etc.) to the device.

[2157] Purchase procedure

[2158] The user selects a product on the app and completes the purchase process. The received product information is displayed, and when the user presses the purchase button, they are redirected to the shopping site's purchase page.

[2159] Use of emotion engine

[2160] Emotional Data Analysis and Applications

[2161] The emotion engine analyzes the user's voice and facial expression data in real time to recognize their emotions. This emotion data is then applied to inventory management and purchasing suggestions. For example, if the user is feeling stressed, it will suggest products that will help them relax.

[2162] Recording frequency of use and emotions

[2163] The emotion engine accumulates and records the user's emotional data and predicts their long-term consumption trends, enabling optimal product recommendations based on the user's preferences and tendencies.

[2164] Specific examples

[2165] For example, imagine a user is preparing breakfast and notices they're low on milk in the refrigerator. They launch the app and take a photo of the inside of the refrigerator. The image is sent to a server, where an AI object detection algorithm is used to identify the type and quantity of each item.

[2166] As a result, it turns out that there are only two bottles of milk in the refrigerator. If the user had set the desired inventory quantity to three bottles, the system will list the one bottle of milk that is missing. The system then searches for this missing item on various shopping sites and displays a list of products that match the user's needs. The user can then select from the list and easily complete the purchase process.

[2167] Also, if a user is feeling stressed during busy morning hours, the emotion engine will recognize this and suggest products that will help reduce stress (such as relaxing herbal tea). In this way, appropriate product suggestions are made based on the user's emotions.

[2168] This system not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows them to receive optimal product suggestions based on their emotions.

[2169] The processing flow will be explained below.

[2170] Step 1: Launch the app

[2171] The user launches the app on their smartphone.

[2172] Step 2: Take a photo

[2173] The user takes a photo of the inside of the refrigerator or shelves.

[2174] The device will launch the camera function and display the capture button.

[2175] The user presses the capture button and an image is captured.

[2176] Step 3: Send image

[2177] The terminal compresses the captured image data and prepares it for transmission.

[2178] The device sends image data to the server's API endpoint.

[2179] Step 4: Image analysis

[2180] The server receives the image data.

[2181] The server uses AI object detection algorithms to identify items in the image.

[2182] The server extracts the type and location information of the identified item.

[2183] Step 5: Counting items

[2184] The server counts the number of each item.

[2185] The server structures the counting results in JSON format and sends them to the terminal.

[2186] Step 6: View inventory information

[2187] The device deserializes the data received from the server.

[2188] The device will display a list of items and their quantities on the app.

[2189] Step 7: Inventory Setup

[2190] The user enters the desired stock quantity into the app.

[2191] The terminal displays an inventory setting screen, and the user inputs and saves the desired inventory quantity.

[2192] Step 8: Enter emotions

[2193] The user displays facial expressions through voice input or a camera.

[2194] The device sends voice data and facial expression images to the emotion engine.

[2195] Step 9: Sentiment Analysis

[2196] The server's emotion engine analyzes voice data and facial expression images in real time to recognize the user's emotions.

[2197] The server stores the recognized emotion data and uses it for subsequent inventory management and purchasing suggestions.

[2198] Step 10: List the missing items

[2199] The terminal compares the current stock quantity with the desired stock quantity.

[2200] The terminal calculates and lists the items that are in short supply and the number of items that are in short supply.

[2201] Step 11: View the Shortage List

[2202] The terminal generates a list of the missing items and displays it to the user.

[2203] Step 12: Search shopping sites

[2204] The terminal transmits the list of missing items to the server.

[2205] The server uses the API of each shopping site (Amazon, Yahoo, Rakuten, etc.) to search for similar products that correspond to the missing items.

[2206] Step 13: Submit your product information

[2207] The server extracts related product information from search results on the shopping site.

[2208] The server sends product information (product name, price, link, etc.) to the terminal.

[2209] Step 14: View product information and checkout

[2210] The product information received by the device is displayed on the app.

[2211] The user selects the product they want to purchase and presses the purchase button.

[2212] The device will redirect you to the shopping site's purchase page.

[2213] Step 15: Emotion-based suggestions

[2214] Based on the user's emotional data recognized by the emotion engine, the system suggests products that are best suited to the user's current emotional state, such as relaxing drinks or health foods.

[2215] The terminal displays emotion-based product suggestions to the user.

[2216] Step 16: Analyze long-term consumption trends

[2217] The emotion engine accumulates and records users' emotional data and analyzes their long-term consumption trends.

[2218] The server optimizes weekly or monthly purchase suggestions based on the analysis results.

[2219] In this way, the user can efficiently manage inventory, smoothly purchase the items he or she needs, and furthermore, receive appropriate product suggestions according to the user's emotional state.

[2220] Example 2

[2221] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2222] Inventory management and replenishment of home appliances and storage spaces are important issues for modern households. In particular, there is a need for a system that quickly and accurately replenishes items when they run out, especially in busy daily lives. However, conventional inventory management systems rely heavily on manual input by users, making efficient management difficult. Furthermore, they lack functionality for optimal product recommendations based on users' emotions and preferences. This can lead to inconvenient situations where necessary items cannot be properly replenished. Furthermore, it is difficult to provide services that respond to users' individual needs and emotional changes.

[2223] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2224] In this invention, the server includes means for a user to take an image of the home appliance or storage location, means for transmitting the image to the server, means for the server to analyze the image and identify the type and number of items, means for transmitting the analysis results to a terminal, means for displaying the analysis results on a terminal screen, means for a user to set a desired inventory quantity, means for comparing the desired inventory quantity with the current inventory quantity and listing items that are in short supply, means for searching various e-commerce sites for products that are identical to or similar to the listed items, means for displaying the search results on the terminal, means for analyzing user emotion data and applying the emotion data to inventory management and purchase suggestions, and means for a user to purchase items based on the search results. This enables efficient inventory management and rapid replenishment of items, and also realizes optimal product suggestions based on the user's emotions and preferences.

[2225] "User" refers to a person who operates the system, takes pictures of home appliances and storage locations, and issues various instructions regarding inventory management and purchasing of goods.

[2226] "Home appliances" refers to electrical appliances used to store household items, such as refrigerators and shelves.

[2227] "Storage space" refers to a place or facility for storing items, such as the inside of a refrigerator or a shelf.

[2228] "Images" refers to photos or videos taken by the user of the inside of an appliance or storage space.

[2229] The term "server" refers to a computer system that receives and analyzes image data sent by a user.

[2230] "Goods" refers to goods or products stored in appliances or storage areas, including food items such as milk and eggs.

[2231] "Quantity" refers to the quantity of a particular item.

[2232] "Terminal" refers to a mobile device such as a smartphone or tablet used by a user.

[2233] "Screen" refers to the display shown on the terminal.

[2234] "Inventory" refers to the quantity of items currently in stock.

[2235] "Desired inventory quantity" refers to the quantity of items desired by the user to be stored.

[2236] "Listing" refers to comparing the current inventory with the desired inventory and listing items that are in short supply.

[2237] "E-commerce site" refers to a website for buying and selling products online, including, for example, Amazon, Yahoo, and Rakuten.

[2238] "Emotion data" refers to data related to emotions collected from the user's voice and facial expressions.

[2239] "Purchase method" refers to the system, such as online payment or shopping cart, that a user uses to purchase goods.

[2240] "Machine learning object detection algorithm" refers to a machine learning or artificial intelligence algorithm used to automatically identify objects in images, such as YOLO or SSD.

[2241] This invention combines a system in which a user takes images of home appliances and storage locations, identifies the type and number of items using an AI object detection algorithm, and performs inventory management and automatic replenishment of items with an emotion engine that recognizes the user's emotions and optimizes inventory management and purchasing suggestions.

[2242] The system includes the following main elements:

[2243] 1. A way for users to take pictures of home appliances and storage locations

[2244] 2. Means of sending the captured image to the server

[2245] 3. Means by which the server analyzes the image and identifies the type and number of items

[2246] 4. Means of sending analysis results to the device

[2247] 5. How to display the analysis results on the device screen

[2248] 6. A way for users to set their desired stock quantity

[2249] 7. A method for comparing desired inventory with current inventory and listing items that are in short supply

[2250] 8. A means to search for the same or similar products of the listed items on various e-commerce sites.

[2251] 9. How to display search results on your device

[2252] 10. Means of analyzing user sentiment data and applying sentiment data to inventory management and purchase recommendations

[2253] 11. How users can purchase items through search results

[2254] Users launch the app on their smartphone or other device and take a photo of the contents of their refrigerator or shelves. The device compresses the image and sends it to the server. The server then uses an AI object detection algorithm (e.g., YOLO or SSD) to identify the type and number of items in the image and sends the analysis results back to the device.

[2255] Users can check the analysis results on the device screen and set the desired inventory quantity as needed. The device compares the current inventory quantity with the desired inventory quantity and lists the items that are in short supply. For the listed items, the server uses the APIs of various e-commerce sites (e.g., Amazon, Yahoo, Rakuten) to search for similar products and send the search results to the device.

[2256] The emotion engine analyzes the user's voice and facial expressions to collect emotional data. This emotional data is then applied to inventory management and purchase suggestions. For example, if a user is feeling stressed, it will suggest products that have a relaxing effect. The user can select the suggested products on the screen and proceed with the purchase.

[2257] As a concrete example, suppose a user is preparing breakfast and notices that there is little milk in the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The image is sent to the server, where an AI object detection algorithm is used to identify the type and number of bottles of milk. The analysis results show that there are only two bottles of milk left in the refrigerator. Since the user's desired stock is three bottles, the system lists the one bottle that is missing.

[2258] The server uses the API of the e-commerce site to search for the missing item and display it to the user. Furthermore, the emotion engine recognizes that the user is feeling stressed while preparing breakfast and suggests relaxing herbal teas.

[2259] The user can select the appropriate product from the options presented and quickly complete the purchase procedure.

[2260] Example prompts to input to the generative AI model

[2261] 1. Inventory count prompt:

[2262] "When the user takes a photo of the refrigerator, count the number of bottles of milk using YOLO and return the result in JSON format."

[2263] 2. Prompt to generate a list of missing items:

[2264] "Compare the current stock quantity data with the user's desired stock quantity data and generate a list of items that are in short supply."

[2265] 3. Emotion-Based Product Recommendation Prompt:

[2266] "Analyze the collected user emotional data and suggest products that have a relaxing effect when the user is feeling stressed."

[2267] As a result, a system is realized that not only allows users to efficiently manage inventory and smoothly replenish necessary items, but also allows users to receive optimal product suggestions based on their emotions.

[2268] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2269] Explain the process step by step

[2270] User operations

[2271] Step 1: Launch the app

[2272] A user launches an app on their smartphone. The input is a touch operation, and the output is the main screen of the app.

[2273] The device loads configuration files and user data in the background and prepares to operate normally.

[2274] Step 2: Take a photo

[2275] The user takes a photo of the inside of the refrigerator or shelf. The device's camera function is used as input, and image data is generated as output.

[2276] The device activates the camera function, and when the user presses the capture button, a photo is taken. This image data is temporarily stored in the device's memory.

[2277] Step 3: Inventory Setup

[2278] The user manually inputs the desired inventory quantity into the app, and the desired inventory data is generated as output.

[2279] The terminal displays an inventory setting screen, and the user inputs the items they need (e.g., 3 bottles of milk, 12 eggs). This information is stored in the terminal's internal database.

[2280] Step 4: Enter emotions

[2281] The emotion engine analyzes the user's voice and facial expressions to recognize emotions. The user's voice and facial expression data are used as input, and emotion data is generated as output.

[2282] The device collects and temporarily stores the user's facial expressions through voice input and a camera, and this data is later sent to a server.

[2283] Image analysis and data processing

[2284] Step 5: Send image

[2285] The terminal sends the captured image to the server. The input is image data, and the output is compressed image data sent to the server.

[2286] The device compresses the image data and sends it to the server's API endpoint.

[2287] Step 6: Image analysis

[2288] The server uses an AI object detection algorithm (e.g., YOLO, SSD) to identify objects in the image. The input is image data, and the output is data on the type and number of objects.

[2289] The server receives the image data and runs an object detection algorithm to extract the type and location of the item, and stores the results in a database.

[2290] Step 7: Counting items

[2291] The server counts the number of each item and sends the result to the terminal. The input is the identified item data, and the output is the counted item data.

[2292] The server categorizes the items, counts the number of each item, and structures the results in JSON format. It then checks the data for integrity before sending it to the device.

[2293] Display inventory information and check shortages

[2294] Step 8: View inventory information

[2295] The terminal displays a list of items and their quantities. The input is the item count data, and the output is the inventory list displayed on the screen.

[2296] The terminal deserializes the received data and displays it in a list format on the screen, allowing the user to check the current inventory status.

[2297] Step 9: List the missing items

[2298] The terminal compares the user's desired inventory quantity with the current inventory quantity and lists the items that are in short supply. The input is the desired inventory data and the current inventory data, and the output is a list of items that are in short supply.

[2299] The terminal compares the two and calculates the shortage. For example, if the desired stock is 3 bottles of milk and the current stock is 2 bottles, the shortage will be 1 bottle.

[2300] Step 10: View the Missing Items List

[2301] The terminal displays the generated shortage list. The input is the shortage list data, and the shortage list is displayed on the screen as the output.

[2302] The shortage list is displayed in a visually easy-to-understand format, allowing users to quickly take next steps.

[2303] Product search and purchase

[2304] Step 11: Search for shopping sites

[2305] The terminal sends the list of items in short supply to the server. The input is the list of items in short supply, and the output is the search results for similar items.

[2306] The server uses the APIs of various e-commerce sites to search for products that correspond to the missing items and sends the results to the terminal.

[2307] Step 12: Submit your product information

[2308] The server sends related product information from search results on a shopping site to the terminal. The input is search result data, and the output is product information.

[2309] Data including product information (product name, price, link, etc.) is sent to the terminal. The terminal analyzes the received data and displays it to the user in an appropriate format.

[2310] Step 13: Checkout

[2311] The user selects a product on the app and completes the purchase process. The input is product selection data, and the output is purchase confirmation data.

[2312] The terminal displays the received product information, and when the user presses the purchase button, the terminal redirects the user to the shopping site's purchase page. Once the purchase procedure is complete, the terminal displays a confirmation message.

[2313] Use of emotion engine

[2314] Step 14: Analyze and apply emotion data

[2315] The emotion engine analyzes the user's voice and facial expression data in real time to recognize emotions. The input is voice and facial expression data, and the output is emotional data.

[2316] For example, if a user is feeling stressed, the system will suggest products that will help them relax (e.g., herbal tea), which will then be applied to inventory management and purchasing suggestions.

[2317] Step 15: Record frequency of use and emotions

[2318] The emotion engine accumulates and records the user's emotional data and predicts the user's long-term consumption trends. The input is the accumulated emotional data, and the output is predicted consumption trends.

[2319] For example, it is possible to identify the time periods during which a user is likely to feel stressed and suggest products suitable for those time periods.

[2320] Through these steps, users can efficiently manage their inventory, quickly replenish needed items, and receive optimal product recommendations based on their emotions.

[2321] (Application example 2)

[2322] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2323] In recent years, there has been a demand for improved cargo management efficiency in autonomous vehicles. However, with conventional inventory management systems, drivers must manually check inventory levels, identify missing items, and then carry out replenishment procedures, which requires time and effort. Furthermore, simply indicating missing items without considering the driver's feelings increases stress and fatigue for the driver, resulting in a decrease in work efficiency. A system that solves these issues, performs efficient inventory management, and reduces the burden on drivers is needed.

[2324] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for the user to take an image of the storage location, a means for transmitting the image to the server, and a means for the server to analyze the image and identify the type and number of items. This allows the driver to easily check the inventory status of the cargo space using the wearable device, as well as identify missing items in real time and automatically receive replenishment suggestions. In addition, the system includes a means for recognizing the driver's emotions and a means for suggesting appropriate products based on the recognized emotion data, thereby reducing the driver's stress and allowing them to work efficiently and comfortably.

[2325] "Means for taking images" refers to a function that allows a user to take images of a storage area or cargo space using a smart device or wearable device.

[2326] The "means for transmitting images to a server" is a function for transmitting captured image data to a server via a network such as the Internet.

[2327] "Means for identifying the type and number of items" refers to a function that analyzes the image received by the server and identifies the type and number of items in the image using an AI object detection algorithm.

[2328] "Means for transmitting analysis results to a terminal" refers to a function for transmitting data analyzed by the server to a user's terminal, particularly a smart device or wearable device.

[2329] The "means for displaying the analysis results on the display device of the terminal" is a function for visually displaying the analysis results on the screen of the user's terminal.

[2330] The "means for setting the stock quantity desired by the user" is a function that allows the user to input the stock quantity desired by the user and record it in the system.

[2331] The "means for listing items in short supply" is a function that compares the set desired inventory quantity with the current inventory quantity and displays the items in short supply in a list format.

[2332] "Means for searching for identical or similar products in various shopping systems" refers to a function for searching online shopping sites for products that are identical or similar to the missing item.

[2333] The "means for displaying search results on a terminal" is a function for displaying product search results obtained from a shopping site on a user's terminal.

[2334] The "means for purchasing goods" is a function for a user to complete the purchase procedure for the product selected by the user through the terminal.

[2335] "Means for managing items in the cargo space, including a wearable device" refers to a function that uses a wearable device worn by the driver to manage inventory in the cargo space.

[2336] The "means for recognizing emotions" is a function that analyzes the user's facial expressions and tone of voice through the wearable device's camera or voice input, and recognizes their emotions.

[2337] The "means for suggesting appropriate products based on recognized emotional data" is a function that suggests products that have stress-reducing or relaxing effects based on the user's emotional data.

[2338] The "means for displaying the proposed results" is a function for displaying the products and information proposed to the user on the terminal.

[2339] This invention relates to a system for improving the efficiency of cargo management in autonomous vehicles. Specifically, the system takes images of the cargo space, identifies the type and number of items using an AI object detection algorithm, and performs inventory management while recognizing the driver's emotions to suggest optimal products. This system is intended to be operated by the user using a wearable device.

[2340] Hardware Configuration

[2341] 1. Wearable devices (e.g., smart glasses)

[2342] It is worn by the driver and takes pictures of the cargo space.

[2343] It has a built-in camera and microphone to collect image and audio data.

[2344] 2. Cloud Server

[2345] Performs image analysis, object detection, and emotion recognition

[2346] Responsible for storing and processing data

[2347] 3. Smart Devices

[2348] Driver-carried smartphones and tablets

[2349] Communicates with the server and displays analysis results and product suggestions

[2350] Software Configuration

[2351] 1. AI object detection algorithms (e.g., YOLO)

[2352] Analyze images captured by the wearable device to identify the type and quantity of cargo

[2353] 2. Sentiment analysis algorithms (e.g., EmotionAnalyzer)

[2354] Analyzes emotions based on the driver's facial expressions and voice data captured through cameras and microphones

[2355] 3. Data transmission and reception module

[2356] It serves as a data transfer mechanism between wearable devices and cloud servers.

[2357] 4. Inventory Management Module

[2358] Manage inventory data on the server

[2359] 5. Proposal Module

[2360] Recommend products to drivers based on emotion data

[2361] Detailed process flow

[2362] 1. Image capture and transmission

[2363] A user takes an image of the cargo space using a wearable device

[2364] The captured image is sent to the server

[2365] 2. Image Analysis

[2366] The server uses AI object detection algorithms to identify objects in the image and determine their type and quantity.

[2367] 3. Inventory Management

[2368] The user inputs the desired inventory quantity into the smart device, and the server compares it with the current inventory to identify any shortages.

[2369] 4. Emotion Recognition and Product Recommendations

[2370] Using the camera and microphone of the wearable device, the driver's facial expressions and voice data are analyzed with an emotion analysis algorithm to recognize their emotions.

[2371] Makes product recommendations based on emotions and displays the results on a smart device

[2372] Specific examples

[2373] While replenishing cargo, the driver uses smart glasses to take images of the cargo space. The images are sent to a server, where an AI object detection algorithm analyzes inventory shortages. At the same time, if the driver's stress level is high, suggestions for relaxation products (e.g., herbal tea) are displayed.

[2374] Prompt Sentence Examples

[2375] "Check stock availability"

[2376] "Please suggest a relaxing product."

[2377] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2378] Step 1:

[2379] A user takes an image of the cargo space using a wearable device (smart glasses).

[2380] Input: Cargo space image

[2381] Output: Captured image data

[2382] Specific operation: Using the camera function of the smart glasses, the user faces the cargo space and presses the capture button to capture an image.

[2383] Step 2:

[2384] The device sends the captured image to a cloud server.

[2385] Input: Photographed image data

[2386] Output: Image data sent to the server

[2387] Specific operation: The transmission module on the smart glasses is activated and sends the image data to the server via the Internet.

[2388] Step 3:

[2389] The server uses an AI object detection algorithm to analyze the image and identify the type and number of items.

[2390] Input: Image data sent to the server

[2391] Output: Data on type and number of items

[2392] Specific operation: The server runs image analysis software (e.g., YOLO) to identify and count the items, and stores the results in a database.

[2393] Step 4:

[2394] The server sends the analysis results to the user's smart device.

[2395] Input: Data on type and quantity of items

[2396] Output: Analysis data sent to smart device

[2397] Specific operation: The server sends the generated item list to the user's smartphone or tablet via the data communication module.

[2398] Step 5:

[2399] The terminal displays the analysis results on a display device.

[2400] Input: Submitted analysis data

[2401] Output: Inventory information displayed on screen

[2402] Specific operation: The display module of the smart device displays the analysis results in list format on the screen so that the user can check them.

[2403] Step 6:

[2404] The user sets the desired stock quantity via a smart device.

[2405] Input: The desired stock quantity entered by the user

[2406] Output: Desired inventory quantity data

[2407] Specific operation: The user enters the desired stock quantity of each item on the application screen of their smart device, and the data is saved within the app.

[2408] Step 7:

[2409] The server compares the current stock with the desired stock and lists the items that are in short supply.

[2410] Input: Current stock quantity data and desired stock quantity data

[2411] Output: List of missing items

[2412] Specific operation: The server compares the stored inventory data with the desired inventory data, calculates the difference, and generates a list of items that are in short supply.

[2413] Step 8:

[2414] The serv...

Claims

1. A means for a user to take an image of the home appliance or storage location; means for transmitting the image to a server; A server analyzes the image and identifies the type and number of items; means for transmitting the analysis result to a terminal; means for displaying the analysis results on a terminal screen; A means for a user to set the desired inventory quantity; means for comparing the desired inventory quantity with the current inventory quantity and listing out any items that are in short supply; A means for searching for the same or similar products of the listed items on various shopping sites; means for displaying the search results on a terminal; A means for a user to purchase an item through the search results; A system including:

2. The system of claim 1, wherein the server analyzes the image using an AI object detection algorithm to identify the type and number of items.

3. 2. The system according to claim 1, wherein the means for listing the shortage items automatically compares the current inventory with the desired inventory, calculates the shortage number, and generates the list.

Images