System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing an image; means for transmitting the image to a server; means for analyzing the image using a generative AI model to extract merchandise information; means for organizing and storing the merchandise information as digital material; means for repeating the merchandise information using audio output means; means for correcting false positives using audio input means; means for analyzing inventory and generating ordering and discount suggestions; and means for notifying a user of the suggestions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional inventory management methods in the retail industry rely on manual barcode reading, which is time-consuming and labor-intensive, and can lead to discrepancies in management accuracy. Furthermore, accurate inventory counts are difficult to grasp due to product depth and overlap, leading to frequent misjudgments. Furthermore, when making management decisions based on inventory data, it is difficult to determine appropriate ordering and inventory disposal. The present invention aims to solve these issues by utilizing image recognition and voice recognition technologies, thereby achieving efficient and accurate inventory management. [Means for solving the problem]

[0005] The system of the present invention has the following configuration. It provides an image capturing means and a means for transmitting image data to a server, and includes a means for analyzing the image data using a generative AI model and extracting product information. It also has a means for organizing and saving product information as digital data, and includes a means for audibly repeating the product information using a voice output means. In this case, it provides a means for correcting erroneous judgments using a voice input means, and adds a means for analyzing inventory data and generating order proposals and discount proposals. This provides a system that includes a means for notifying the user of the proposal results. It also includes a means for recognizing stockpiled inventory, and a means for recognizing product part numbers and quantities when the generative AI model extracts product information, thereby achieving efficient and accurate inventory management.

[0006] "Image capture means" refers to a device or function used to capture an image of the shelf.

[0007] The "means for transmitting image data to a server" is a function for transferring captured images to a server for processing.

[0008] A "generative AI model" is an artificial intelligence model used for image recognition and data analysis.

[0009] The "means for analyzing image data and extracting product information" is a function for identifying and extracting specific product information from image data.

[0010] "Means for organizing and storing product information as digital data" refers to the function of preparing extracted product information in a digital format and storing it appropriately.

[0011] The "audio output means" is a function that outputs stored or processed information as audio.

[0012] The "voice input means" is a function for receiving instructions from the user by voice.

[0013] The "means for correcting erroneous determination" is a function for correcting erroneous determination.

[0014] The "means for analyzing inventory data and generating order proposals and discount proposals" is a function for analyzing inventory status and proposing optimal orders and discounts.

[0015] The "means for notifying the user of the proposal result" is a function for communicating the generated proposal to the user.

[0016] "Means to recognize stock that is stacked underneath" is a function to recognize products that are not visible in the image or stock that is hidden in the background.

[0017] "Means for recognizing the product number and quantity" is a function for recognizing the identification number of a specific product and its quantity. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and management advice provision.

[0040] 1. Image capture and data transmission

[0041] Subject: User and Device

[0042] A user uses a terminal (e.g., a smartphone or dedicated image capture device) to take a photo of the storage shelf or the area where products are displayed. The captured image is sent to the server by the terminal. The terminal then uploads the image data to the server via the Internet.

[0043] 2. Image Recognition and Product Information Extraction

[0044] Subject: Server and Generative AI Model

[0045] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[0046] 3. Organize and store product information

[0047] Subject: Server

[0048] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[0049] 4. Voice confirmation and correction

[0050] Subject: Terminal and User

[0051] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instructions. They can give instructions such as, "There are 4 units of product A."

[0052] 5. Correction and storage of data

[0053] Subject: Server

[0054] The server analyzes the user's voice commands and modifies the data, updating the database based on the voice commands to maintain accurate inventory information.

[0055] 6. Providing management advice

[0056] Subject: Server

[0057] The server analyzes the inventory data and generates ordering suggestions and discount suggestions for inventory clearance. The server then notifies the user of the generated suggestions. For example, the notification may be in the form of "Product A is low in stock, so we recommend ordering more. Also, product B is not selling well, so we suggest a 20% discount."

[0058] 7. Providing display proposals

[0059] Subject: Server

[0060] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided.

[0061] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] Subject: User

[0065] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[0066] Step 2:

[0067] Subject: Terminal

[0068] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[0069] Step 3:

[0070] Subject: Server

[0071] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[0072] Step 4:

[0073] Subject: Generative AI model

[0074] A generative AI model analyzes image data and extracts product information. The model identifies each product in the image, recognizes the product number and quantity associated with each product, and recognizes overlapping products and hidden inventory.

[0075] Step 5:

[0076] Subject: Server

[0077] The server receives product information from the generative AI model, organizes the received product information as digital data, and stores information such as the number of items for each product number in a database.

[0078] Step 6:

[0079] Subject: Terminal

[0080] The terminal reads back the inventory information obtained from the server by voice. Using the voice output function, it notifies the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0081] Step 7:

[0082] Subject: User

[0083] The user confirms the result by voice and issues instructions to correct any incorrect judgments. For example, the user can send a correction instruction by voice inputting "There are four units of product A."

[0084] Step 8:

[0085] Subject: Terminal

[0086] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[0087] Step 9:

[0088] Subject: Server

[0089] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is saved back into the database.

[0090] Step 10:

[0091] Subject: Server

[0092] The server analyzes inventory data and generates ordering and discount proposals. Data analysis algorithms are used to make appropriate proposals based on the inventory situation.

[0093] Step 11:

[0094] Subject: Server

[0095] The server notifies the user of the recommendation results, such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount."

[0096] Step 12:

[0097] Subject: Server

[0098] The server analyzes data on best-selling products and generates optimization proposals for display and shelf layout. The proposals include specific advice such as "Product A should be placed in a prominent position."

[0099] Step 13:

[0100] Subject: Server

[0101] The server notifies the user of the generated display proposal, who receives the proposal via app notification or email and makes changes to the product placement, etc.

[0102] This is the specific process flow of the inventory management system that uses image and voice recognition technology. This system enables efficient and accurate inventory management and management advice.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional inventory management systems require a lot of manual checking, which makes them inefficient and prone to misjudgments and errors in updating information. They also lack the data needed to make management decisions based on inventory status, or advice on optimal product display methods, making comprehensive inventory management and sales promotion difficult.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, a means for generating shelf optimization proposals, and a means for notifying the user of the shelf optimization proposals. This not only enables efficient and accurate inventory management, but also enables the provision of data useful for management decisions and specific advice on optimal product display methods.

[0108] "Image capturing means" refers to a device or mechanism for capturing images of product shelves or display shelves.

[0109] The "means for transmitting image data to a server" is a communication means for uploading captured image data to a server via the Internet.

[0110] "Means for analyzing image data using a generative AI model and extracting product information" refers to analytical means that uses a generative AI model to recognize the product number and quantity of products in the image data.

[0111] "Means for organizing and storing product information as digital data" refers to means for systematizing product information extracted by a generative AI model and storing it in digital data format.

[0112] The "means for repeating product information by voice using voice output means" refers to voice output means for outputting product information as voice and notifying the user.

[0113] The "means for correcting erroneous determination using a voice input means" is a means for receiving voice input from the user and correcting erroneously determined information.

[0114] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to analytical means for analyzing inventory data and generating product ordering and discount proposals.

[0115] The "means for notifying the user of the proposal results" is a means for notifying the user of the generated order proposal or discount proposal.

[0116] The "means for generating shelf optimization proposals" is an analytical means for generating proposals for optimizing the display and placement of products.

[0117] The "means for notifying the user of the shelf optimization proposal" is a means for notifying the user of the generated shelf optimization proposal.

[0118] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and provision of management advice and shelf layout optimization suggestions.

[0119] First, a user uses a smartphone or a dedicated image capture device to take a photo of a storage shelf or an area where products are displayed. For example, a user takes a photo of a beverage shelf in a supermarket with a smartphone. This image capture means is a smartphone or a high-performance camera device.

[0120] Next, the device sends the captured image data to a server. The device uploads the image data to the server using Wi-Fi or mobile data communication. For example, a smartphone sends the captured image to a server via Wi-Fi.

[0121] The server passes the received image data to an AI model for image recognition. The generative AI model identifies the product and determines the product number and quantity. This analysis includes the ability to recognize inventory stacked at the back. For example, the server analyzes the image it receives and identifies the type and quantity of beverage bottles. In this process, it also recognizes products hidden at the back.

[0122] The server then organizes the product information extracted by the generative AI model and stores it as digital data. Product numbers, quantities, and other necessary information are stored in a database. For example, the server stores the product numbers and quantities of beverages in a database.

[0123] The device notifies the user of the saved product information by voice. For example, the device may output a voice message saying, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can correct the incorrect judgment using the voice input function. For example, the user may input a voice message saying, "There are 4 units of product A." This voice output and input method is performed using a smart speaker or voice recognition software.

[0124] The server analyzes the user's voice instructions and updates the contents of the database. For example, if the server receives the instruction "There are four units of product A," it updates the quantity of product A in the database to four units.

[0125] Furthermore, the server analyzes the inventory data and generates ordering and discount suggestions. For example, a suggestion might be generated such as, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we suggest a 20% discount." The results of this suggestion are then notified to the user.

[0126] Finally, the server analyzes the data on best-selling products and generates shelf layout optimization suggestions. For example, specific advice such as "Product A should be placed in a prominent position" is provided. This makes it possible to optimize product display and placement.

[0127] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[0128] Example prompt sentence:

[0129] "Please recognize the product in this image and extract the product number and quantity."

[0130] "Please also recognize the inventory in the background of the image and provide the total number of items."

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1:

[0133] Input: Images of product shelves or display cases taken by users using smartphones or dedicated image capture devices

[0134] How it works: A user uses their device to take a photo of a shelf from a suitable angle. They take multiple images to ensure a clear view and to see inventory at the back.

[0135] Output: Image data saved on the device

[0136] Step 2:

[0137] Input: Image data stored on the device

[0138] How it works: The device uses Wi-Fi or mobile data to upload the captured image data to the server.

[0139] Output: Image data transferred to the server

[0140] Step 3:

[0141] Input: Image data transferred to the server

[0142] Operation: The server passes the received image data to the AI ​​model, which then performs image recognition. It identifies the product and determines the product number and quantity. It also recognizes inventory stacked in the back.

[0143] Output: Product information extracted by the generative AI model (product number, quantity, etc.)

[0144] Step 4:

[0145] Input: Product information extracted by the generative AI model

[0146] How it works: The server organizes product information and stores it in a database in digital format. The data stored includes the product number, quantity, and other necessary information for each product.

[0147] Output: Organized product information stored in a database

[0148] Step 5:

[0149] Input: Product information stored in the database

[0150] Operation: The device notifies the user of product information by voice. The device outputs a message such as, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0151] Output: Audio information notified to the user

[0152] Step 6:

[0153] Input: Audio information announced to the user

[0154] Operation: The user uses the voice input function to correct any incorrect judgments as necessary. For example, the user might say, "There are four units of product A."

[0155] Output: Corrected voice instructions entered into the device

[0156] Step 7:

[0157] Input: Modified voice instructions entered into the device

[0158] Operation: The server analyzes the voice instructions and modifies the contents of the database. The modified information is updated in the database.

[0159] Output: Database reflecting revised inventory information

[0160] Step 8:

[0161] Input: Database reflecting corrected inventory information

[0162] How it works: The server analyzes inventory data and generates ordering and discount suggestions. It generates suggestions like, "Product A is low in stock, so we recommend you order more. Product B is not selling well, so we suggest a 20% discount."

[0163] Output: Generated order proposals and discount proposals

[0164] Step 9:

[0165] Input: Generated order proposals and discount proposals

[0166] Operation: The server notifies the user of order suggestions and discount suggestions. For example, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we offer a 20% discount."

[0167] Output: Order proposals and discount proposals notified to the user

[0168] Step 10:

[0169] Input: Order proposals and discount proposals notified to the user

[0170] How it works: The server analyzes data on best-selling products and generates shelf layout optimization suggestions. It provides specific advice such as "Product A should be placed in a prominent position."

[0171] Output: Generated shelf optimization proposals

[0172] Step 11:

[0173] Input: Generated shelf optimization proposals

[0174] Operation: The server notifies the user of shelf optimization suggestions, such as "Product A should be placed in a prominent position."

[0175] Output: Shelf optimization suggestions communicated to the user

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] Modern retail businesses require efficient and accurate inventory management. However, traditional inventory management methods require a lot of manual checking and updating, which takes time and effort and is prone to human error. Additionally, product placement and display optimization often rely on experience and intuition, making management based on objective data difficult. To solve these problems, a system is needed that provides efficient and accurate inventory management and management advice based on objective data.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an interface means for acquiring image data and voice data using a smart device and confirming and correcting the recognition results. This significantly reduces the effort and errors involved in conventional inventory management, enables efficient and accurate inventory management, and further enables optimization of ordering and display based on objective data.

[0181] "Image capturing means" refers to a device or function for capturing images of the arrangement and storage conditions of items and creating image data thereof.

[0182] The "means for transmitting image data to a server" refers to a function or protocol for transmitting captured image data to a server via a network.

[0183] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and automatically performs tasks such as image recognition and data analysis.

[0184] "Means for extracting product information" refers to analytical technology for extracting detailed product information (product number, quantity, etc.) from image data.

[0185] "Means for organizing and storing product information as digital data" refers to a system or method for organizing and storing extracted product information in a database or the like.

[0186] "Means for repeating product information aloud using voice output means" refers to a voice synthesis technology that outputs extracted and organized product information as voice and informs the user.

[0187] The "means for correcting erroneous determinations using a voice input means" refers to a voice recognition technology that accepts and analyzes a user's voice instructions and corrects erroneous data.

[0188] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to an algorithm or system that analyzes inventory data and proposes optimal order quantities and discount strategies.

[0189] The "means for notifying the user of the proposal results" refers to a messaging system or notification function for informing the user of the created order proposal or discount proposal.

[0190] "Interface means for acquiring image data and audio data using a smart device and confirming and correcting the recognition results" refers to a user interface for acquiring image and audio data using a device such as a smartphone or tablet and confirming and correcting the results.

[0191] MODE FOR CARRYING OUT THE INVENTION

[0192] The present invention is an efficient inventory management system using automated image recognition and a voice interface, and is configured as follows.

[0193] 1. Image capture method

[0194] Users use smart devices (smartphones or tablets) to take pictures of product shelves and storage areas, which generates image data that allows users to visually grasp the placement and quantity of items.

[0195] 2. Means of sending image data to the server

[0196] The terminal sends the captured image data to a server via the Internet, so the smart device must have network connectivity.

[0197] 3. A means of analyzing image data and extracting product information using generative AI models

[0198] The server inputs the received image data into a generative AI model for analysis. The generative AI model recognizes the product number and quantity contained in the image, and also analyzes the depth information of the image to recognize inventory that is hidden beneath the surface. This allows accurate product information to be extracted.

[0199] 4. A means of organizing and storing product information as digital data

[0200] The product information extracted by the server is organized as digital data by item, quantity, etc. The organized data is stored in a database and used for subsequent processing.

[0201] 5. A means for repeating product information aloud using a voice output means

[0202] The terminal has the ability to output the inventory information stored in the database aloud, for example, notifying the user in the form of "There are 10 units of product A and 5 units of product B."

[0203] 6. A method for correcting misjudgments using voice input

[0204] Users can use the voice input function to give instructions to the system, such as "I want 9 units of product A," which is then analyzed using voice recognition technology and the information in the database is updated.

[0205] 7. A means of analyzing inventory data and generating ordering and discounting proposals

[0206] The server analyzes the inventory data stored in the database to generate order requirements and discount suggestions, using algorithms to make suggestions based on sales trends and stock levels, and notifies the user.

[0207] 8. Means of notifying users of the proposed results

[0208] The generated ordering and discount proposals are notified to the user via the terminal. For example, the notification may say, "Product B is low in stock, so we recommend you order more. Also, product C is not selling well, so we suggest a 20% discount."

[0209] 9. Interface means for acquiring image data and voice data using a smart device, and for confirming and correcting the recognition results.

[0210] The system uses a smart device to acquire image and voice data and provides a user interface for confirming and correcting the recognition results. For example, a user can take a photo of a shelf with their smartphone and confirm the results by voice.

[0211] This significantly improves the efficiency of inventory management and reduces human error. Accurate product recognition using generative AI models, combined with the ease of a voice interface, improves the accuracy and efficiency of overall management.

[0212] Examples of specific examples and prompts

[0213] As a concrete example, a store clerk takes a photo of a shelf with their smartphone, and the app automatically reads the inventory information and notifies them via voice. The clerk can also give a voice instruction such as "There are 9 units of product A, not 10," and the app will correct the data and notify them again. Furthermore, the app will offer management advice such as "We recommend ordering product B. Also, please place product C in a prominent location."

[0214] Example prompt for a generative AI model:

[0215] "Inventor data: Product A: 10 pieces, Product B: 5 pieces, Product C: 2 pieces. Provide stock refill and display suggestions."

[0216] This prompt sentence generates appropriate management advice and display suggestions.

[0217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0218] Step 1:

[0219] Users use their smart devices to take pictures of product shelves and storage areas, and then use the camera function to obtain image data that allows them to visually grasp the placement and quantity of items.

[0220] Input: Actual footage of shelves and storage areas

[0221] Output: Image data saved on the smart device

[0222] Step 2:

[0223] The device sends the acquired image data to a cloud server via the Internet, using the network connection function to send the image data to a specific server.

[0224] Input: Image data on a smart device

[0225] Output: Image data stored on a cloud server

[0226] Step 3:

[0227] The server inputs the received image data into the generative AI model for analysis. The generative AI model detects and recognizes the product numbers and quantities of the products contained in the image, and also analyzes the depth information of the image if necessary.

[0228] Input: Image data stored on a cloud server

[0229] Output: Specific product information such as product number and quantity

[0230] Step 4:

[0231] The server organizes the product information extracted by the generative AI model as digital data and stores it in a database. Through this process, information such as product numbers and quantities for each product is systematically organized.

[0232] Input: Product information extracted by the generative AI model

[0233] Output: Organized product information stored in a database

[0234] Step 5:

[0235] The terminal notifies the user of the inventory information stored in the database using a voice output function, for example, by repeating aloud, "There are 10 units of product A and 5 units of product B."

[0236] Input: Stock information stored in the database

[0237] Output: Voice output of stock information

[0238] Step 6:

[0239] The user can use the voice input function to give correction instructions to the system as needed, for example, "There are nine units of product A."

[0240] Input: User correction instructions via voice input

[0241] Output: Audio data received by the device

[0242] Step 7:

[0243] The server analyzes the received voice data and corrects any incorrect judgments. It uses voice recognition technology to analyze the instructions and corrects the relevant data in the database.

[0244] Input: Audio data received by the device

[0245] Output: Database with revised product data

[0246] Step 8:

[0247] The server analyzes the inventory data in the database and generates ordering and discount proposals, using algorithms to create optimal proposals based on sales trends, inventory levels, etc.

[0248] Input: Product information and inventory data stored in a database

[0249] Output: Generated order proposals and discount proposals

[0250] Step 9:

[0251] The generated recommendations are then sent to the user via their device. For example, they might say, "Product B is low in stock, so we recommend placing an additional order. Also, product C is not selling well, so we suggest a 20% discount."

[0252] Input: Generated order proposals and discount proposals

[0253] Output: The user is notified of the proposed results.

[0254] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0255] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[0256] 1. Image capture and data transmission

[0257] Subject: User and Device

[0258] A user starts the camera app on their device and takes an image of the display shelf. For example, they can take the image using a smartphone or a dedicated image capture device. The captured image is sent to the server by the device. The device then uploads the image data to the server via the Internet.

[0259] 2. Image Recognition and Product Information Extraction

[0260] Subject: Server and Generative AI Model

[0261] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[0262] 3. Organize and store product information

[0263] Subject: Server

[0264] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[0265] 4. Voice confirmation and correction

[0266] Subject: Terminal and User

[0267] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instruction. They can give instructions such as, "There are 4 units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[0268] 5. Correction and storage of data

[0269] Subject: Server

[0270] The server analyzes the user's voice commands and corrects the data. The database is updated based on the voice commands to maintain accurate inventory information. Voice-to-text conversion is performed within the device, and the converted text data is sent to the server.

[0271] 6. Providing management advice

[0272] Subject: Server

[0273] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine can adjust the suggestions based on the user's emotional state.

[0274] 7. Notification of Proposal

[0275] Subject: Server and terminal

[0276] The server notifies the user of the generated suggestions. For example, it may send a message such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount." Based on the emotion engine, the notification method and content can be adjusted according to the user's emotional state.

[0277] 8. Providing display proposals

[0278] Subject: Server

[0279] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. These include specific advice such as "Product A should be placed in a prominent position." The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[0280] The above is a specific embodiment of the present invention. This system uses image recognition and voice recognition technologies to enable efficient and accurate inventory management, as well as management advice that takes into account the user's emotions. This optimizes the inventory management process and enables more flexible and effective management decisions.

[0281] The processing flow will be explained below.

[0282] Step 1:

[0283] Subject: User

[0284] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[0285] Step 2:

[0286] Subject: Terminal

[0287] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[0288] Step 3:

[0289] Subject: Server

[0290] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[0291] Step 4:

[0292] Subject: Generative AI model

[0293] A generative AI model analyzes image data to extract the part number and quantity of each product. The model identifies products based on their visual characteristics and recognizes overlapping products and hidden inventory.

[0294] Step 5:

[0295] Subject: Server

[0296] The server receives product information from the generative AI model and organizes it as digital data, which is then stored in a database.

[0297] Step 6:

[0298] Subject: Terminal

[0299] The terminal repeats the inventory information received from the server by voice, using the voice output function to notify the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0300] Step 7:

[0301] Subject: User

[0302] The user confirms the result by voice and issues a correction instruction if there is an incorrect judgment. For example, the user can send a correction instruction by voice inputting "There are four units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[0303] Step 8:

[0304] Subject: Terminal

[0305] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[0306] Step 9:

[0307] Subject: Server

[0308] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is stored in the database.

[0309] Step 10:

[0310] Subject: Server

[0311] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine adjusts the suggestions based on the user's emotional state.

[0312] Step 11:

[0313] Subject: Server

[0314] The server notifies the user of the recommendation results. For example, it may send a message such as, "Product A is low in stock, so we recommend you place an additional order," or, "Product B is not selling well, so we suggest a 20% discount." The server adjusts the notification method and content according to the user's emotional state based on the emotion engine.

[0315] Step 12:

[0316] Subject: Server and terminal

[0317] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided. The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[0318] Step 13:

[0319] Subject: User

[0320] The user rearranges or orders products based on the suggestions received from the server. For example, the user may "order additional product A" or "apply a discount on product B."

[0321] This concludes the process flow of an inventory management system that combines image and voice recognition technologies with user emotion recognition. This system enables efficient and accurate inventory management and provides management advice that takes into account the user's emotional state.

[0322] Example 2

[0323] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] Conventional inventory management systems make it difficult to grasp accurate inventory information, often requiring manual confirmation. Furthermore, they do not provide management advice that takes into account the user's emotional state, limiting the effectiveness of management decisions. Furthermore, there is a lack of means to accurately identify hidden inventory and product part numbers and quantities. There is a need for a system that can solve these issues and provide efficient and accurate inventory management.

[0325] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0326] In this invention, the server includes an image capturing means, a means for transmitting image data to an information processing device, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an emotion engine for analyzing the user's emotional state and adjusting the proposal content. This significantly improves the accuracy and efficiency of inventory management and makes it possible to provide management advice tailored to the user's emotional state.

[0327] "Image capturing means" refers to a device that allows a user to capture an image of a product display shelf, and includes a smartphone or a dedicated image capture device.

[0328] The "means for transmitting image data to an information processing device" is a process for transmitting captured image data to a server, and includes a function for uploading data via the Internet.

[0329] A "generative AI model" is an artificial intelligence model that analyzes image data, recognizes objects in the image, and extracts corresponding information.

[0330] "Means for extracting product information" includes a function to identify product part numbers and quantities in an image using a generative AI model.

[0331] "Means for organizing and storing product information as digital data" refers to the process of organizing the extracted product information into a specific format and storing it in a database.

[0332] "Audio output means" refers to a function that enables the terminal to convey stored product information to the user by voice, and includes a voice assistant and a speaker.

[0333] "Voice input means" refers to a function for receiving a user's voice instructions and analyzing the voice data, and includes a microphone and voice recognition software.

[0334] An "emotion engine" is a system that includes algorithms for analyzing voice data and understanding the user's emotional state.

[0335] The "means for analyzing inventory data and generating order proposals and discount proposals" is a process for automatically generating appropriate order and discount proposals based on inventory information.

[0336] "Means for notifying users of proposal results" refers to a function for informing users of the generated management advice and proposal content, and includes push notifications and message sending.

[0337] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[0338] In implementing this system, the following hardware and software is recommended:

[0339] 1. Hardware:

[0340] Users use smartphones or dedicated image capture devices to take images.

[0341] The terminals include computers and smartphones that can connect to the Internet.

[0342] 2. Software:

[0343] Generative AI models: For example, object detection algorithms such as "YOLO" and "Detectron2" are used.

[0344] Database: We use database management systems such as "MySQL" or "PostgreSQL" to store inventory information.

[0345] Speech recognition: Services such as "Google Cloud Speech-to-Text API" and "Amazon Alexa" are used to analyze voice data.

[0346] Emotion Engine: Implements emotion recognition algorithms to analyze the user's emotional state.

[0347] System processing overview

[0348] 1. Image capture and data transmission

[0349] Users use their smartphones to take pictures of the shelves, which are then compressed and uploaded to a server via the Internet.

[0350] 2. Image Recognition and Product Information Extraction

[0351] The server sends the received image data to the generative AI model, which analyzes the products in the image and identifies the product number and quantity. The detection results are returned to the server in JSON format.

[0352] 3. Organize and store product information

[0353] The server analyzes the data from the generative AI model and organizes the product number and quantity information for each product. The organized data is stored in a database and serves as the basis for the next process.

[0354] 4. Voice confirmation and correction

[0355] The device outputs inventory information stored in the database and repeats it back to the user. The user can use voice input to correct any incorrect judgments. The emotion engine analyzes the voice data and understands the user's emotional state.

[0356] 5. Correction and storage of data

[0357] The server analyzes the user's voice commands and updates the database with the corrected data. The voice input is converted into text on the device and sent to the server.

[0358] 6. Providing management advice

[0359] The server analyzes inventory data and generates ordering and discount offers, with an emotional engine adapting the offers based on the user's emotional state.

[0360] 7. Notification of Proposal

[0361] The server notifies the user of the generated suggestions using push notifications or messaging.

[0362] 8. Providing display proposals

[0363] The server analyzes best-selling products and generates optimization proposals for display and shelf layout. The emotion engine makes proposals that take into account the user's emotional state.

[0364] Specific examples

[0365] 1. A user takes a picture of a store's display shelves with their smartphone and sends it to the server.

[0366] 2. The server passes the image data to a generative AI model (e.g., YOLO) and analyzes the product number and quantity.

[0367] 3. The analysis results are saved in the database in the format of "5 units of product A, 3 units of product B, 8 units of product C."

[0368] 4. The device uses Google Assistant to tell the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0369] 5. The user inputs "Product A is 4 units" by voice and gives a correction instruction. The emotion engine analyzes the user's emotional state.

[0370] 6. The corrected data is updated in the database.

[0371] 7. The server uses TensorFlow to analyze the inventory data and notifies the customer that "Product A is low in stock, so we recommend placing an additional order."

[0372] 8. Generate display optimization suggestions for best-selling products, such as "Product A should be placed in a prominent position."

[0373] In this way, the present invention realizes a system in which the user, terminal, and server cooperate to provide efficient and accurate inventory management and management advice that takes into account the user's feelings.

[0374] Prompt Sentence Examples

[0375] "Upload image data of the display shelves and use the generative AI model to analyze the product part numbers and quantities. Then, read back the analysis results and correct any misidentifications with voice instructions. Finally, provide management advice based on the inventory status and generate display suggestions."

[0376] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0377] Step 1:

[0378] Image capture and data transmission

[0379] input:

[0380] A user picks up a smartphone, launches the camera app, and takes a picture of the display shelf.

[0381] Specific behavior:

[0382] The user taps the camera app on the smartphone's operation screen to launch it.

[0383] While looking at the camera preview screen, take a photo of the display shelf at the appropriate angle and distance.

[0384] The captured image file is saved in the device's local storage.

[0385] Data processing and output:

[0386] The image data captured by the device is compressed and saved in a file format such as JPEG.

[0387] The compressed image file is uploaded to a server via the Internet.

[0388] Step 2:

[0389] Image recognition and product information extraction

[0390] input:

[0391] The server receives the image data sent from the terminal.

[0392] Specific behavior:

[0393] The server receives the image data through an HTTP request.

[0394] The received image data is saved in a specific directory.

[0395] Data processing and output:

[0396] The server inputs the stored image data into a generative AI model (e.g., YOLO).

[0397] The generative AI model analyzes the products in the image, identifies the product number and quantity in JSON format, and returns it to the server.

[0398] Step 3:

[0399] Organizing and storing product information

[0400] input:

[0401] The server receives the JSON-formatted product information data returned from the generated AI model.

[0402] Specific behavior:

[0403] The server reads the JSON data and extracts the product number and quantity information.

[0404] The extracted information is then organized and presented in a digital format.

[0405] Data processing and output:

[0406] Save data for each product (e.g., "Product A: 5 units, Product B: 3 units") in the database.

[0407] Save the information in a database management system (e.g. MySQL) using the INSERT statement.

[0408] Step 4:

[0409] Voice confirmation and correction

[0410] input:

[0411] The server transmits the organized inventory information to the terminal, and the terminal starts outputting audio.

[0412] Specific behavior:

[0413] The device converts the inventory information from text to speech, and the voice assistant (e.g., Google Assistant) repeats, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0414] The user can use the voice input function to give instructions to correct any incorrect judgments (e.g., "There are four units of product A").

[0415] Data processing and output:

[0416] The device converts the voice input into text and sends the text data to the server.

[0417] The emotion engine analyzes the voice data and understands the user's emotional state.

[0418] Step 5:

[0419] Data correction and storage

[0420] input:

[0421] The server receives the user's voice input data and begins analyzing it.

[0422] Specific behavior:

[0423] The server analyzes the received text data and identifies the incorrectly determined inventory information.

[0424] Generate the corresponding SQL statements to correct the misjudged inventory information.

[0425] Data processing and output:

[0426] Use the UPDATE statement to update the corrected inventory information in the database (e.g., "UPDATE inventory information table SET quantity=4 WHERE product ID='product A'").

[0427] The corrected accurate inventory information is saved in the database.

[0428] Step 6:

[0429] Providing management advice

[0430] input:

[0431] The server generates ordering suggestions and discount suggestions based on an up-to-date inventory database.

[0432] Specific behavior:

[0433] The server analyzes the inventory data and generates ordering and discount suggestions using data analysis algorithms (e.g., TensorFlow).

[0434] The emotion engine adjusts suggestions based on the user's emotional state.

[0435] Data processing and output:

[0436] Based on the detected trends, suggestions such as "Product A is low in stock, so we recommend placing an additional order" are generated.

[0437] The server sends the generated proposal to the terminal.

[0438] Step 7:

[0439] Proposal Notification

[0440] input:

[0441] The management advice and proposals generated by the server are sent to the terminal.

[0442] Specific behavior:

[0443] The server creates a notification message and sends a push notification to the user's smartphone using Firebase Cloud Messaging or similar.

[0444] The device receives the notification and displays it to the user.

[0445] Data processing and output:

[0446] The proposal is displayed to the user and management advice is provided (e.g., "Product A is low in stock, so we recommend placing an additional order").

[0447] Step 8:

[0448] Providing display proposals

[0449] input:

[0450] The server generates display suggestions based on data on best-selling items.

[0451] Specific behavior:

[0452] The server analyzes top-selling products and generates display optimization suggestions.

[0453] The emotion engine takes into account the user's emotional state and adjusts the suggestions accordingly.

[0454] Data processing and output:

[0455] The proposal is generated in the form of "Product A should be placed in a prominent position."

[0456] The proposal is sent to the terminal and displayed to the user.

[0457] In this way, the system of the present invention combines image recognition technology and voice recognition technology to provide efficient and accurate inventory management and management advice that takes into account the user's emotions.

[0458] (Application example 2)

[0459] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0460] Conventional inventory management systems have the problem that it is difficult to accurately grasp and manage product information, and they lack management advice that takes into account users' emotions and purchasing intentions, making it difficult to fully optimize inventory and promote sales.

[0461] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for capturing and analyzing the user's emotional state using a smart device, and a means for adjusting the content of suggestions and notification methods based on the user's emotional state. This enables accurate understanding of inventory information and flexible management advice based on the user's emotions.

[0462] "Image capturing means" refers to a device or system for capturing images of shelves and inventory.

[0463] "Means for transmitting image data to a server" refers to a function or device for transmitting captured images to a cloud or remote server via a network.

[0464] A "generative AI model" is a machine learning model that uses artificial intelligence technology to analyze image data and extract specific patterns and features.

[0465] "Means for organizing and storing product information as digital data" refers to a device or system for structuring extracted product information and storing it in a database.

[0466] The "audio output means" is a device or system for outputting the stored product information as audio.

[0467] The "voice input means" is a device or system for recognizing voice instructions from the user and converting the contents of the voice instructions into text data.

[0468] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to a device or system for analyzing the current inventory status and generating proposals for the need for additional orders and discounts.

[0469] The "means for notifying the user of the proposal results" is a device or system for notifying the user of the generated order proposal or discount proposal.

[0470] "Means for capturing and analyzing a user's emotional state using a smart device" refers to a device or system that uses smart glasses or other wearable devices to obtain and analyze a user's emotional state from voice and facial expressions.

[0471] "Means for adjusting the content of proposals and notification methods based on the user's emotional state" refers to a device or system for appropriately adjusting the content of order proposals and discount proposals and notification methods based on the analyzed emotional state of the user.

[0472] The present invention is a smart device-based system that combines efficient inventory management in brick-and-mortar stores with user emotion analysis. The system includes image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, provision of management advice, and user emotion recognition using an emotion engine. A specific embodiment of the system is described in detail below.

[0473] Hardware and software used

[0474] Hardware: Smart glasses (with camera, microphone, and display)

[0475] Software: generative AI models, emotion engines, speech recognition software (e.g., Google Cloud Speech-to-Text)

[0476] Image capture and data transmission

[0477] A user wears smart glasses and walks around the store scanning the inventory on the shelves. The smart glasses' camera captures images and sends them over the internet to a cloud server. The image data obtained through this process is then sent to the server via a high-speed data transmission channel.

[0478] Image recognition and product information extraction

[0479] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This makes it possible to determine the product's part number and quantity, as well as recognize inventory stacked deep within the image. This image recognition technology enables more accurate inventory information to be obtained.

[0480] Organizing and storing product information

[0481] The server then processes the product information extracted by the generative AI model and organizes it as digital data for each product. The information includes product numbers, quantities, and other necessary data. This information is then stored in a database and becomes the basis for the next process.

[0482] Voice confirmation and correction

[0483] The smart glasses' voice output function is used to repeat the organized inventory information to the user. For example, the voice output may say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments with voice instructions. During this process, the emotion engine analyzes the voice data and understands the user's emotional state.

[0484] Providing management advice

[0485] The server analyzes inventory data and generates ordering and discount suggestions. It uses analytical algorithms to make appropriate suggestions based on stock availability. The emotion engine can also adjust the suggestions based on the user's emotional state. For example, it can make specific ordering suggestions such as "Product A is low in stock, so we recommend you order more," or discount suggestions such as "Product B is not selling well, so we suggest a 20% discount."

[0486] Suggestions and notifications based on emotional state

[0487] The server notifies the user of the generated order proposals and discount proposals. Based on the emotion engine, the server adjusts the notification method and content according to the user's emotional state. This allows the server to provide information in a form that is psychologically easy for the user to accept.

[0488] Examples of specific examples and prompts

[0489] Examples:

[0490] A user wears smart glasses and takes pictures of the shelves while walking around the store.

[0491] Product information is sent to the server, and the generative AI model analyzes the inventory information.

[0492] When the user requests a correction by voice input, the user may give a voice instruction such as "There are four units of product A."

[0493] Example prompt sentence:

[0494] "Use the camera in your smart glasses to take pictures of your shelves and send the inventory information to the cloud."

[0495] "Capture your customers' facial expressions, analyze them with an emotion engine, and offer sales strategy advice."

[0496] "Propose additional orders or changes to shelf locations based on recognized inventory information."

[0497] This makes it possible to implement a system according to a specific embodiment of the present invention.

[0498] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0499] Step 1:

[0500] A user wears smart glasses and activates the smart glasses' camera to scan the inventory on the shelves while walking around the store. The smart glasses then take an image of the entire display shelf and capture the image data. The input here is the image captured by the camera, and the output is the image data stored in the smart glasses.

[0501] Step 2:

[0502] Image data captured by the device (smart glasses) is sent to a cloud server via the Internet. HTTP or HTTPS is used as the transmission protocol. The input is image data, and the output is image data stored on the cloud server.

[0503] Step 3:

[0504] The server passes the received image data to the generative AI model, which then performs image analysis. The generative AI model extracts the product's part number and quantity from the image data, and recognizes the inventory stacked in the back as well. The input is the transmitted image data, and the output is the extracted product information (part number, quantity).

[0505] Step 4:

[0506] The server organizes the product information extracted by the generative AI model, organizes it as digital data for each product, and stores it in a database. The input is the extracted product information, and the output is organized digital data stored in the database.

[0507] Step 5:

[0508] The server converts the organized inventory information into voice data and repeats it back to the user using the device's (smart glasses') voice output function. This uses a voice synthesis service such as Google Cloud Text-to-Speech. The input is organized digital data, and the output is voice data.

[0509] Step 6:

[0510] When a user wants to correct a false positive using the voice input function, they input the correction instructions through the microphone of the smart glasses. The input is the user's voice instruction, which is converted into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The output is text data.

[0511] Step 7:

[0512] The server analyzes the text data acquired through voice input and compares it with existing database information to correct misjudgments. The input is text data of the user's voice instructions, and the output is corrected inventory information.

[0513] Step 8:

[0514] The server analyzes the inventory data and generates ordering and discount proposals using an AI model, taking into account the user's emotional data using an emotion engine. The input is the revised inventory information and the user's emotional information, and the output is the generated ordering and discount proposals.

[0515] Step 9:

[0516] The server notifies the user of the generated order proposals and discount proposals. When notifying, the method and content of the notification are adjusted based on the user's emotional state using an emotion engine. The input is the proposal data, and the output is a notification to the user.

[0517] The above is a detailed description of the processing steps of the system for specifically implementing the present invention.

[0518] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0519] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0520] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0521] [Second embodiment]

[0522] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0523] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0524] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0525] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0526] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0527] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0528] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0529] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0530] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0531] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0532] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0533] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0534] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and management advice provision.

[0535] 1. Image capture and data transmission

[0536] Subject: User and Device

[0537] A user uses a terminal (e.g., a smartphone or dedicated image capture device) to take a photo of the storage shelf or the area where products are displayed. The captured image is sent to the server by the terminal. The terminal then uploads the image data to the server via the Internet.

[0538] 2. Image Recognition and Product Information Extraction

[0539] Subject: Server and Generative AI Model

[0540] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[0541] 3. Organize and store product information

[0542] Subject: Server

[0543] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[0544] 4. Voice confirmation and correction

[0545] Subject: Terminal and User

[0546] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instructions. They can give instructions such as, "There are 4 units of product A."

[0547] 5. Correction and storage of data

[0548] Subject: Server

[0549] The server analyzes the user's voice commands and modifies the data, updating the database based on the voice commands to maintain accurate inventory information.

[0550] 6. Providing management advice

[0551] Subject: Server

[0552] The server analyzes the inventory data and generates ordering suggestions and discount suggestions for inventory clearance. The server then notifies the user of the generated suggestions. For example, the notification may be in the form of "Product A is low in stock, so we recommend ordering more. Also, product B is not selling well, so we suggest a 20% discount."

[0553] 7. Providing display proposals

[0554] Subject: Server

[0555] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided.

[0556] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[0557] The processing flow will be explained below.

[0558] Step 1:

[0559] Subject: User

[0560] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[0561] Step 2:

[0562] Subject: Terminal

[0563] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[0564] Step 3:

[0565] Subject: Server

[0566] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[0567] Step 4:

[0568] Subject: Generative AI model

[0569] A generative AI model analyzes image data and extracts product information. The model identifies each product in the image, recognizes the product number and quantity associated with each product, and recognizes overlapping products and hidden inventory.

[0570] Step 5:

[0571] Subject: Server

[0572] The server receives product information from the generative AI model, organizes the received product information as digital data, and stores information such as the number of items for each product number in a database.

[0573] Step 6:

[0574] Subject: Terminal

[0575] The terminal reads back the inventory information obtained from the server by voice. Using the voice output function, it notifies the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0576] Step 7:

[0577] Subject: User

[0578] The user confirms the result by voice and issues instructions to correct any incorrect judgments. For example, the user can send a correction instruction by voice inputting "There are four units of product A."

[0579] Step 8:

[0580] Subject: Terminal

[0581] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[0582] Step 9:

[0583] Subject: Server

[0584] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is saved back into the database.

[0585] Step 10:

[0586] Subject: Server

[0587] The server analyzes inventory data and generates ordering and discount proposals. Data analysis algorithms are used to make appropriate proposals based on the inventory situation.

[0588] Step 11:

[0589] Subject: Server

[0590] The server notifies the user of the recommendation results, such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount."

[0591] Step 12:

[0592] Subject: Server

[0593] The server analyzes data on best-selling products and generates optimization proposals for display and shelf layout. The proposals include specific advice such as "Product A should be placed in a prominent position."

[0594] Step 13:

[0595] Subject: Server

[0596] The server notifies the user of the generated display proposal, who receives the proposal via app notification or email and makes changes to the product placement, etc.

[0597] This is the specific process flow of the inventory management system that uses image and voice recognition technology. This system enables efficient and accurate inventory management and management advice.

[0598] Example 1

[0599] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] Conventional inventory management systems require a lot of manual checking, which makes them inefficient and prone to misjudgments and errors in updating information. They also lack the data needed to make management decisions based on inventory status, or advice on optimal product display methods, making comprehensive inventory management and sales promotion difficult.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0602] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, a means for generating shelf optimization proposals, and a means for notifying the user of the shelf optimization proposals. This not only enables efficient and accurate inventory management, but also enables the provision of data useful for management decisions and specific advice on optimal product display methods.

[0603] "Image capturing means" refers to a device or mechanism for capturing images of product shelves or display shelves.

[0604] The "means for transmitting image data to a server" is a communication means for uploading captured image data to a server via the Internet.

[0605] "Means for analyzing image data using a generative AI model and extracting product information" refers to analytical means that uses a generative AI model to recognize the product number and quantity of products in the image data.

[0606] "Means for organizing and storing product information as digital data" refers to means for systematizing product information extracted by a generative AI model and storing it in digital data format.

[0607] The "means for repeating product information by voice using voice output means" refers to voice output means for outputting product information as voice and notifying the user.

[0608] The "means for correcting erroneous determination using a voice input means" is a means for receiving voice input from the user and correcting erroneously determined information.

[0609] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to analytical means for analyzing inventory data and generating product ordering and discount proposals.

[0610] The "means for notifying the user of the proposal results" is a means for notifying the user of the generated order proposal or discount proposal.

[0611] The "means for generating shelf optimization proposals" is an analytical means for generating proposals for optimizing the display and placement of products.

[0612] The "means for notifying the user of the shelf optimization proposal" is a means for notifying the user of the generated shelf optimization proposal.

[0613] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and provision of management advice and shelf layout optimization suggestions.

[0614] First, a user uses a smartphone or a dedicated image capture device to take a photo of a storage shelf or an area where products are displayed. For example, a user takes a photo of a beverage shelf in a supermarket with a smartphone. This image capture means is a smartphone or a high-performance camera device.

[0615] Next, the device sends the captured image data to a server. The device uploads the image data to the server using Wi-Fi or mobile data communication. For example, a smartphone sends the captured image to a server via Wi-Fi.

[0616] The server passes the received image data to an AI model for image recognition. The generative AI model identifies the product and determines the product number and quantity. This analysis includes the ability to recognize inventory stacked at the back. For example, the server analyzes the image it receives and identifies the type and quantity of beverage bottles. In this process, it also recognizes products hidden at the back.

[0617] The server then organizes the product information extracted by the generative AI model and stores it as digital data. Product numbers, quantities, and other necessary information are stored in a database. For example, the server stores the product numbers and quantities of beverages in a database.

[0618] The device notifies the user of the saved product information by voice. For example, the device may output a voice message saying, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can correct the incorrect judgment using the voice input function. For example, the user may input a voice message saying, "There are 4 units of product A." This voice output and input method is performed using a smart speaker or voice recognition software.

[0619] The server analyzes the user's voice instructions and updates the contents of the database. For example, if the server receives the instruction "There are four units of product A," it updates the quantity of product A in the database to four units.

[0620] Furthermore, the server analyzes the inventory data and generates ordering and discount suggestions. For example, a suggestion might be generated such as, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we suggest a 20% discount." The results of this suggestion are then notified to the user.

[0621] Finally, the server analyzes the data on best-selling products and generates shelf layout optimization suggestions. For example, specific advice such as "Product A should be placed in a prominent position" is provided. This makes it possible to optimize product display and placement.

[0622] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[0623] Example prompt sentence:

[0624] "Please recognize the product in this image and extract the product number and quantity."

[0625] "Please also recognize the inventory in the background of the image and provide the total number of items."

[0626] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0627] Step 1:

[0628] Input: Images of product shelves or display cases taken by users using smartphones or dedicated image capture devices

[0629] How it works: A user uses their device to take a photo of a shelf from a suitable angle. They take multiple images to ensure a clear view and to see inventory at the back.

[0630] Output: Image data saved on the device

[0631] Step 2:

[0632] Input: Image data stored on the device

[0633] How it works: The device uses Wi-Fi or mobile data to upload the captured image data to the server.

[0634] Output: Image data transferred to the server

[0635] Step 3:

[0636] Input: Image data transferred to the server

[0637] Operation: The server passes the received image data to the AI ​​model, which then performs image recognition. It identifies the product and determines the product number and quantity. It also recognizes inventory stacked in the back.

[0638] Output: Product information extracted by the generative AI model (product number, quantity, etc.)

[0639] Step 4:

[0640] Input: Product information extracted by the generative AI model

[0641] How it works: The server organizes product information and stores it in a database in digital format. The data stored includes the product number, quantity, and other necessary information for each product.

[0642] Output: Organized product information stored in a database

[0643] Step 5:

[0644] Input: Product information stored in the database

[0645] Operation: The device notifies the user of product information by voice. The device outputs a message such as, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0646] Output: Audio information notified to the user

[0647] Step 6:

[0648] Input: Audio information announced to the user

[0649] Operation: The user uses the voice input function to correct any incorrect judgments as necessary. For example, the user might say, "There are four units of product A."

[0650] Output: Corrected voice instructions entered into the device

[0651] Step 7:

[0652] Input: Modified voice instructions entered into the device

[0653] Operation: The server analyzes the voice instructions and modifies the contents of the database. The modified information is updated in the database.

[0654] Output: Database reflecting revised inventory information

[0655] Step 8:

[0656] Input: Database reflecting corrected inventory information

[0657] How it works: The server analyzes inventory data and generates ordering and discount suggestions. It generates suggestions like, "Product A is low in stock, so we recommend you order more. Product B is not selling well, so we suggest a 20% discount."

[0658] Output: Generated order proposals and discount proposals

[0659] Step 9:

[0660] Input: Generated order proposals and discount proposals

[0661] Operation: The server notifies the user of order suggestions and discount suggestions. For example, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we offer a 20% discount."

[0662] Output: Order proposals and discount proposals notified to the user

[0663] Step 10:

[0664] Input: Order proposals and discount proposals notified to the user

[0665] How it works: The server analyzes data on best-selling products and generates shelf layout optimization suggestions. It provides specific advice such as "Product A should be placed in a prominent position."

[0666] Output: Generated shelf optimization proposals

[0667] Step 11:

[0668] Input: Generated shelf optimization proposals

[0669] Operation: The server notifies the user of shelf optimization suggestions, such as "Product A should be placed in a prominent position."

[0670] Output: Shelf optimization suggestions communicated to the user

[0671] (Application example 1)

[0672] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0673] Modern retail businesses require efficient and accurate inventory management. However, traditional inventory management methods require a lot of manual checking and updating, which takes time and effort and is prone to human error. Additionally, product placement and display optimization often rely on experience and intuition, making management based on objective data difficult. To solve these problems, a system is needed that provides efficient and accurate inventory management and management advice based on objective data.

[0674] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0675] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an interface means for acquiring image data and voice data using a smart device and confirming and correcting the recognition results. This significantly reduces the effort and errors involved in conventional inventory management, enables efficient and accurate inventory management, and further enables optimization of ordering and display based on objective data.

[0676] "Image capturing means" refers to a device or function for capturing images of the arrangement and storage conditions of items and creating image data thereof.

[0677] The "means for transmitting image data to a server" refers to a function or protocol for transmitting captured image data to a server via a network.

[0678] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and automatically performs tasks such as image recognition and data analysis.

[0679] "Means for extracting product information" refers to analytical technology for extracting detailed product information (product number, quantity, etc.) from image data.

[0680] "Means for organizing and storing product information as digital data" refers to a system or method for organizing and storing extracted product information in a database or the like.

[0681] "Means for repeating product information aloud using voice output means" refers to a voice synthesis technology that outputs extracted and organized product information as voice and informs the user.

[0682] The "means for correcting erroneous determinations using a voice input means" refers to a voice recognition technology that accepts and analyzes a user's voice instructions and corrects erroneous data.

[0683] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to an algorithm or system that analyzes inventory data and proposes optimal order quantities and discount strategies.

[0684] The "means for notifying the user of the proposal results" refers to a messaging system or notification function for informing the user of the created order proposal or discount proposal.

[0685] "Interface means for acquiring image data and audio data using a smart device and confirming and correcting the recognition results" refers to a user interface for acquiring image and audio data using a device such as a smartphone or tablet and confirming and correcting the results.

[0686] MODE FOR CARRYING OUT THE INVENTION

[0687] The present invention is an efficient inventory management system using automated image recognition and a voice interface, and is configured as follows.

[0688] 1. Image capture method

[0689] Users use smart devices (smartphones or tablets) to take pictures of product shelves and storage areas, which generates image data that allows users to visually grasp the placement and quantity of items.

[0690] 2. Means of sending image data to the server

[0691] The terminal sends the captured image data to a server via the Internet, so the smart device must have network connectivity.

[0692] 3. A means of analyzing image data and extracting product information using generative AI models

[0693] The server inputs the received image data into a generative AI model for analysis. The generative AI model recognizes the product number and quantity contained in the image, and also analyzes the depth information of the image to recognize inventory that is hidden beneath the surface. This allows accurate product information to be extracted.

[0694] 4. A means of organizing and storing product information as digital data

[0695] The product information extracted by the server is organized as digital data by item, quantity, etc. The organized data is stored in a database and used for subsequent processing.

[0696] 5. A means for repeating product information aloud using a voice output means

[0697] The terminal has the ability to output the inventory information stored in the database aloud, for example, notifying the user in the form of "There are 10 units of product A and 5 units of product B."

[0698] 6. A method for correcting misjudgments using voice input

[0699] Users can use the voice input function to give instructions to the system, such as "I want 9 units of product A," which is then analyzed using voice recognition technology and the information in the database is updated.

[0700] 7. A means of analyzing inventory data and generating ordering and discounting proposals

[0701] The server analyzes the inventory data stored in the database to generate order requirements and discount suggestions, using algorithms to make suggestions based on sales trends and stock levels, and notifies the user.

[0702] 8. Means of notifying users of the proposed results

[0703] The generated ordering and discount proposals are notified to the user via the terminal. For example, the notification may say, "Product B is low in stock, so we recommend you order more. Also, product C is not selling well, so we suggest a 20% discount."

[0704] 9. Interface means for acquiring image data and voice data using a smart device, and for confirming and correcting the recognition results.

[0705] The system uses a smart device to acquire image and voice data and provides a user interface for confirming and correcting the recognition results. For example, a user can take a photo of a shelf with their smartphone and confirm the results by voice.

[0706] This significantly improves the efficiency of inventory management and reduces human error. Accurate product recognition using generative AI models, combined with the ease of a voice interface, improves the accuracy and efficiency of overall management.

[0707] Examples of specific examples and prompts

[0708] As a concrete example, a store clerk takes a photo of a shelf with their smartphone, and the app automatically reads the inventory information and notifies them via voice. The clerk can also give a voice instruction such as "There are 9 units of product A, not 10," and the app will correct the data and notify them again. Furthermore, the app will offer management advice such as "We recommend ordering product B. Also, please place product C in a prominent location."

[0709] Example prompt for a generative AI model:

[0710] "Inventor data: Product A: 10 pieces, Product B: 5 pieces, Product C: 2 pieces. Provide stock refill and display suggestions."

[0711] This prompt sentence generates appropriate management advice and display suggestions.

[0712] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0713] Step 1:

[0714] Users use their smart devices to take pictures of product shelves and storage areas, and then use the camera function to obtain image data that allows them to visually grasp the placement and quantity of items.

[0715] Input: Actual footage of shelves and storage areas

[0716] Output: Image data saved on the smart device

[0717] Step 2:

[0718] The device sends the acquired image data to a cloud server via the Internet, using the network connection function to send the image data to a specific server.

[0719] Input: Image data on a smart device

[0720] Output: Image data stored on a cloud server

[0721] Step 3:

[0722] The server inputs the received image data into the generative AI model for analysis. The generative AI model detects and recognizes the product numbers and quantities of the products contained in the image, and also analyzes the depth information of the image if necessary.

[0723] Input: Image data stored on a cloud server

[0724] Output: Specific product information such as product number and quantity

[0725] Step 4:

[0726] The server organizes the product information extracted by the generative AI model as digital data and stores it in a database. Through this process, information such as product numbers and quantities for each product is systematically organized.

[0727] Input: Product information extracted by the generative AI model

[0728] Output: Organized product information stored in a database

[0729] Step 5:

[0730] The terminal notifies the user of the inventory information stored in the database using a voice output function, for example, by repeating aloud, "There are 10 units of product A and 5 units of product B."

[0731] Input: Stock information stored in the database

[0732] Output: Voice output of stock information

[0733] Step 6:

[0734] The user can use the voice input function to give correction instructions to the system as needed, for example, "There are nine units of product A."

[0735] Input: User correction instructions via voice input

[0736] Output: Audio data received by the device

[0737] Step 7:

[0738] The server analyzes the received voice data and corrects any incorrect judgments. It uses voice recognition technology to analyze the instructions and corrects the relevant data in the database.

[0739] Input: Audio data received by the device

[0740] Output: Database with revised product data

[0741] Step 8:

[0742] The server analyzes the inventory data in the database and generates ordering and discount proposals, using algorithms to create optimal proposals based on sales trends, inventory levels, etc.

[0743] Input: Product information and inventory data stored in a database

[0744] Output: Generated order proposals and discount proposals

[0745] Step 9:

[0746] The generated recommendations are then sent to the user via their device. For example, they might say, "Product B is low in stock, so we recommend placing an additional order. Also, product C is not selling well, so we suggest a 20% discount."

[0747] Input: Generated order proposals and discount proposals

[0748] Output: The user is notified of the proposed results.

[0749] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0750] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[0751] 1. Image capture and data transmission

[0752] Subject: User and Device

[0753] A user starts the camera app on their device and takes an image of the display shelf. For example, they can take the image using a smartphone or a dedicated image capture device. The captured image is sent to the server by the device. The device then uploads the image data to the server via the Internet.

[0754] 2. Image Recognition and Product Information Extraction

[0755] Subject: Server and Generative AI Model

[0756] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[0757] 3. Organize and store product information

[0758] Subject: Server

[0759] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[0760] 4. Voice confirmation and correction

[0761] Subject: Terminal and User

[0762] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instruction. They can give instructions such as, "There are 4 units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[0763] 5. Correction and storage of data

[0764] Subject: Server

[0765] The server analyzes the user's voice commands and corrects the data. The database is updated based on the voice commands to maintain accurate inventory information. Voice-to-text conversion is performed within the device, and the converted text data is sent to the server.

[0766] 6. Providing management advice

[0767] Subject: Server

[0768] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine can adjust the suggestions based on the user's emotional state.

[0769] 7. Notification of Proposal

[0770] Subject: Server and terminal

[0771] The server notifies the user of the generated suggestions. For example, it may send a message such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount." Based on the emotion engine, the notification method and content can be adjusted according to the user's emotional state.

[0772] 8. Providing display proposals

[0773] Subject: Server

[0774] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. These include specific advice such as "Product A should be placed in a prominent position." The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[0775] The above is a specific embodiment of the present invention. This system uses image recognition and voice recognition technologies to enable efficient and accurate inventory management, as well as management advice that takes into account the user's emotions. This optimizes the inventory management process and enables more flexible and effective management decisions.

[0776] The processing flow will be explained below.

[0777] Step 1:

[0778] Subject: User

[0779] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[0780] Step 2:

[0781] Subject: Terminal

[0782] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[0783] Step 3:

[0784] Subject: Server

[0785] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[0786] Step 4:

[0787] Subject: Generative AI model

[0788] A generative AI model analyzes image data to extract the part number and quantity of each product. The model identifies products based on their visual characteristics and recognizes overlapping products and hidden inventory.

[0789] Step 5:

[0790] Subject: Server

[0791] The server receives product information from the generative AI model and organizes it as digital data, which is then stored in a database.

[0792] Step 6:

[0793] Subject: Terminal

[0794] The terminal repeats the inventory information received from the server by voice, using the voice output function to notify the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0795] Step 7:

[0796] Subject: User

[0797] The user confirms the result by voice and issues a correction instruction if there is an incorrect judgment. For example, the user can send a correction instruction by voice inputting "There are four units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[0798] Step 8:

[0799] Subject: Terminal

[0800] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[0801] Step 9:

[0802] Subject: Server

[0803] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is stored in the database.

[0804] Step 10:

[0805] Subject: Server

[0806] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine adjusts the suggestions based on the user's emotional state.

[0807] Step 11:

[0808] Subject: Server

[0809] The server notifies the user of the recommendation results. For example, it may send a message such as, "Product A is low in stock, so we recommend you place an additional order," or, "Product B is not selling well, so we suggest a 20% discount." The server adjusts the notification method and content according to the user's emotional state based on the emotion engine.

[0810] Step 12:

[0811] Subject: Server and terminal

[0812] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided. The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[0813] Step 13:

[0814] Subject: User

[0815] The user rearranges or orders products based on the suggestions received from the server. For example, the user may "order additional product A" or "apply a discount on product B."

[0816] This concludes the process flow of an inventory management system that combines image and voice recognition technologies with user emotion recognition. This system enables efficient and accurate inventory management and provides management advice that takes into account the user's emotional state.

[0817] Example 2

[0818] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] Conventional inventory management systems make it difficult to grasp accurate inventory information, often requiring manual confirmation. Furthermore, they do not provide management advice that takes into account the user's emotional state, limiting the effectiveness of management decisions. Furthermore, there is a lack of means to accurately identify hidden inventory and product part numbers and quantities. There is a need for a system that can solve these issues and provide efficient and accurate inventory management.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0821] In this invention, the server includes an image capturing means, a means for transmitting image data to an information processing device, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an emotion engine for analyzing the user's emotional state and adjusting the proposal content. This significantly improves the accuracy and efficiency of inventory management and makes it possible to provide management advice tailored to the user's emotional state.

[0822] "Image capturing means" refers to a device that allows a user to capture an image of a product display shelf, and includes a smartphone or a dedicated image capture device.

[0823] The "means for transmitting image data to an information processing device" is a process for transmitting captured image data to a server, and includes a function for uploading data via the Internet.

[0824] A "generative AI model" is an artificial intelligence model that analyzes image data, recognizes objects in the image, and extracts corresponding information.

[0825] "Means for extracting product information" includes a function to identify product part numbers and quantities in an image using a generative AI model.

[0826] "Means for organizing and storing product information as digital data" refers to the process of organizing the extracted product information into a specific format and storing it in a database.

[0827] "Audio output means" refers to a function that enables the terminal to convey stored product information to the user by voice, and includes a voice assistant and a speaker.

[0828] "Voice input means" refers to a function for receiving a user's voice instructions and analyzing the voice data, and includes a microphone and voice recognition software.

[0829] An "emotion engine" is a system that includes algorithms for analyzing voice data and understanding the user's emotional state.

[0830] The "means for analyzing inventory data and generating order proposals and discount proposals" is a process for automatically generating appropriate order and discount proposals based on inventory information.

[0831] "Means for notifying users of proposal results" refers to a function for informing users of the generated management advice and proposal content, and includes push notifications and message sending.

[0832] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[0833] In implementing this system, the following hardware and software is recommended:

[0834] 1. Hardware:

[0835] Users use smartphones or dedicated image capture devices to take images.

[0836] The terminals include computers and smartphones that can connect to the Internet.

[0837] 2. Software:

[0838] Generative AI models: For example, object detection algorithms such as "YOLO" and "Detectron2" are used.

[0839] Database: We use database management systems such as "MySQL" or "PostgreSQL" to store inventory information.

[0840] Speech recognition: Services such as "Google Cloud Speech-to-Text API" and "Amazon Alexa" are used to analyze voice data.

[0841] Emotion Engine: Implements emotion recognition algorithms to analyze the user's emotional state.

[0842] System processing overview

[0843] 1. Image capture and data transmission

[0844] Users use their smartphones to take pictures of the shelves, which are then compressed and uploaded to a server via the Internet.

[0845] 2. Image Recognition and Product Information Extraction

[0846] The server sends the received image data to the generative AI model, which analyzes the products in the image and identifies the product number and quantity. The detection results are returned to the server in JSON format.

[0847] 3. Organize and store product information

[0848] The server analyzes the data from the generative AI model and organizes the product number and quantity information for each product. The organized data is stored in a database and serves as the basis for the next process.

[0849] 4. Voice confirmation and correction

[0850] The device outputs inventory information stored in the database and repeats it back to the user. The user can use voice input to correct any incorrect judgments. The emotion engine analyzes the voice data and understands the user's emotional state.

[0851] 5. Correction and storage of data

[0852] The server analyzes the user's voice commands and updates the database with the corrected data. The voice input is converted into text on the device and sent to the server.

[0853] 6. Providing management advice

[0854] The server analyzes inventory data and generates ordering and discount offers, with an emotional engine adapting the offers based on the user's emotional state.

[0855] 7. Notification of Proposal

[0856] The server notifies the user of the generated suggestions using push notifications or messaging.

[0857] 8. Providing display proposals

[0858] The server analyzes best-selling products and generates optimization proposals for display and shelf layout. The emotion engine makes proposals that take into account the user's emotional state.

[0859] Specific examples

[0860] 1. A user takes a picture of a store's display shelves with their smartphone and sends it to the server.

[0861] 2. The server passes the image data to a generative AI model (e.g., YOLO) and analyzes the product number and quantity.

[0862] 3. The analysis results are saved in the database in the format of "5 units of product A, 3 units of product B, 8 units of product C."

[0863] 4. The device uses Google Assistant to tell the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0864] 5. The user inputs "Product A is 4 units" by voice and gives a correction instruction. The emotion engine analyzes the user's emotional state.

[0865] 6. The corrected data is updated in the database.

[0866] 7. The server uses TensorFlow to analyze the inventory data and notifies the customer that "Product A is low in stock, so we recommend placing an additional order."

[0867] 8. Generate display optimization suggestions for best-selling products, such as "Product A should be placed in a prominent position."

[0868] In this way, the present invention realizes a system in which the user, terminal, and server cooperate to provide efficient and accurate inventory management and management advice that takes into account the user's feelings.

[0869] Prompt Sentence Examples

[0870] "Upload image data of the display shelves and use the generative AI model to analyze the product part numbers and quantities. Then, read back the analysis results and correct any misidentifications with voice instructions. Finally, provide management advice based on the inventory status and generate display suggestions."

[0871] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0872] Step 1:

[0873] Image capture and data transmission

[0874] input:

[0875] A user picks up a smartphone, launches the camera app, and takes a picture of the display shelf.

[0876] Specific behavior:

[0877] The user taps the camera app on the smartphone's operation screen to launch it.

[0878] While looking at the camera preview screen, take a photo of the display shelf at the appropriate angle and distance.

[0879] The captured image file is saved in the device's local storage.

[0880] Data processing and output:

[0881] The image data captured by the device is compressed and saved in a file format such as JPEG.

[0882] The compressed image file is uploaded to a server via the Internet.

[0883] Step 2:

[0884] Image recognition and product information extraction

[0885] input:

[0886] The server receives the image data sent from the terminal.

[0887] Specific behavior:

[0888] The server receives the image data through an HTTP request.

[0889] The received image data is saved in a specific directory.

[0890] Data processing and output:

[0891] The server inputs the stored image data into a generative AI model (e.g., YOLO).

[0892] The generative AI model analyzes the products in the image, identifies the product number and quantity in JSON format, and returns it to the server.

[0893] Step 3:

[0894] Organizing and storing product information

[0895] input:

[0896] The server receives the JSON-formatted product information data returned from the generated AI model.

[0897] Specific behavior:

[0898] The server reads the JSON data and extracts the product number and quantity information.

[0899] The extracted information is then organized and presented in a digital format.

[0900] Data processing and output:

[0901] Save data for each product (e.g., "Product A: 5 units, Product B: 3 units") in the database.

[0902] Save the information in a database management system (e.g. MySQL) using the INSERT statement.

[0903] Step 4:

[0904] Voice confirmation and correction

[0905] input:

[0906] The server transmits the organized inventory information to the terminal, and the terminal starts outputting audio.

[0907] Specific behavior:

[0908] The device converts the inventory information from text to speech, and the voice assistant (e.g., Google Assistant) repeats, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[0909] The user can use the voice input function to give instructions to correct any incorrect judgments (e.g., "There are four units of product A").

[0910] Data processing and output:

[0911] The device converts the voice input into text and sends the text data to the server.

[0912] The emotion engine analyzes the voice data and understands the user's emotional state.

[0913] Step 5:

[0914] Data correction and storage

[0915] input:

[0916] The server receives the user's voice input data and begins analyzing it.

[0917] Specific behavior:

[0918] The server analyzes the received text data and identifies the incorrectly determined inventory information.

[0919] Generate the corresponding SQL statements to correct the misjudged inventory information.

[0920] Data processing and output:

[0921] Use the UPDATE statement to update the corrected inventory information in the database (e.g., "UPDATE inventory information table SET quantity=4 WHERE product ID='product A'").

[0922] The corrected accurate inventory information is saved in the database.

[0923] Step 6:

[0924] Providing management advice

[0925] input:

[0926] The server generates ordering suggestions and discount suggestions based on an up-to-date inventory database.

[0927] Specific behavior:

[0928] The server analyzes the inventory data and generates ordering and discount suggestions using data analysis algorithms (e.g., TensorFlow).

[0929] The emotion engine adjusts suggestions based on the user's emotional state.

[0930] Data processing and output:

[0931] Based on the detected trends, suggestions such as "Product A is low in stock, so we recommend placing an additional order" are generated.

[0932] The server sends the generated proposal to the terminal.

[0933] Step 7:

[0934] Proposal Notification

[0935] input:

[0936] The management advice and proposals generated by the server are sent to the terminal.

[0937] Specific behavior:

[0938] The server creates a notification message and sends a push notification to the user's smartphone using Firebase Cloud Messaging or similar.

[0939] The device receives the notification and displays it to the user.

[0940] Data processing and output:

[0941] The proposal is displayed to the user and management advice is provided (e.g., "Product A is low in stock, so we recommend placing an additional order").

[0942] Step 8:

[0943] Providing display proposals

[0944] input:

[0945] The server generates display suggestions based on data on best-selling items.

[0946] Specific behavior:

[0947] The server analyzes top-selling products and generates display optimization suggestions.

[0948] The emotion engine takes into account the user's emotional state and adjusts the suggestions accordingly.

[0949] Data processing and output:

[0950] The proposal is generated in the form of "Product A should be placed in a prominent position."

[0951] The proposal is sent to the terminal and displayed to the user.

[0952] In this way, the system of the present invention combines image recognition technology and voice recognition technology to provide efficient and accurate inventory management and management advice that takes into account the user's emotions.

[0953] (Application example 2)

[0954] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0955] Conventional inventory management systems have the problem that it is difficult to accurately grasp and manage product information, and they lack management advice that takes into account users' emotions and purchasing intentions, making it difficult to fully optimize inventory and promote sales.

[0956] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for capturing and analyzing the user's emotional state using a smart device, and a means for adjusting the content of suggestions and notification methods based on the user's emotional state. This enables accurate understanding of inventory information and flexible management advice based on the user's emotions.

[0957] "Image capturing means" refers to a device or system for capturing images of shelves and inventory.

[0958] "Means for transmitting image data to a server" refers to a function or device for transmitting captured images to a cloud or remote server via a network.

[0959] A "generative AI model" is a machine learning model that uses artificial intelligence technology to analyze image data and extract specific patterns and features.

[0960] "Means for organizing and storing product information as digital data" refers to a device or system for structuring extracted product information and storing it in a database.

[0961] The "audio output means" is a device or system for outputting the stored product information as audio.

[0962] The "voice input means" is a device or system for recognizing voice instructions from the user and converting the contents of the voice instructions into text data.

[0963] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to a device or system for analyzing the current inventory status and generating proposals for the need for additional orders and discounts.

[0964] The "means for notifying the user of the proposal results" is a device or system for notifying the user of the generated order proposal or discount proposal.

[0965] "Means for capturing and analyzing a user's emotional state using a smart device" refers to a device or system that uses smart glasses or other wearable devices to obtain and analyze a user's emotional state from voice and facial expressions.

[0966] "Means for adjusting the content of proposals and notification methods based on the user's emotional state" refers to a device or system for appropriately adjusting the content of order proposals and discount proposals and notification methods based on the analyzed emotional state of the user.

[0967] The present invention is a smart device-based system that combines efficient inventory management in brick-and-mortar stores with user emotion analysis. The system includes image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, provision of management advice, and user emotion recognition using an emotion engine. A specific embodiment of the system is described in detail below.

[0968] Hardware and software used

[0969] Hardware: Smart glasses (with camera, microphone, and display)

[0970] Software: generative AI models, emotion engines, speech recognition software (e.g., Google Cloud Speech-to-Text)

[0971] Image capture and data transmission

[0972] A user wears smart glasses and walks around the store scanning the inventory on the shelves. The smart glasses' camera captures images and sends them over the internet to a cloud server. The image data obtained through this process is then sent to the server via a high-speed data transmission channel.

[0973] Image recognition and product information extraction

[0974] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This makes it possible to determine the product's part number and quantity, as well as recognize inventory stacked deep within the image. This image recognition technology enables more accurate inventory information to be obtained.

[0975] Organizing and storing product information

[0976] The server then processes the product information extracted by the generative AI model and organizes it as digital data for each product. The information includes product numbers, quantities, and other necessary data. This information is then stored in a database and becomes the basis for the next process.

[0977] Voice confirmation and correction

[0978] The smart glasses' voice output function is used to repeat the organized inventory information to the user. For example, the voice output may say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments with voice instructions. During this process, the emotion engine analyzes the voice data and understands the user's emotional state.

[0979] Providing management advice

[0980] The server analyzes inventory data and generates ordering and discount suggestions. It uses analytical algorithms to make appropriate suggestions based on stock availability. The emotion engine can also adjust the suggestions based on the user's emotional state. For example, it can make specific ordering suggestions such as "Product A is low in stock, so we recommend you order more," or discount suggestions such as "Product B is not selling well, so we suggest a 20% discount."

[0981] Suggestions and notifications based on emotional state

[0982] The server notifies the user of the generated order proposals and discount proposals. Based on the emotion engine, the server adjusts the notification method and content according to the user's emotional state. This allows the server to provide information in a form that is psychologically easy for the user to accept.

[0983] Examples of specific examples and prompts

[0984] Examples:

[0985] A user wears smart glasses and takes pictures of the shelves while walking around the store.

[0986] Product information is sent to the server, and the generative AI model analyzes the inventory information.

[0987] When the user requests a correction by voice input, the user may give a voice instruction such as "There are four units of product A."

[0988] Example prompt sentence:

[0989] "Use the camera in your smart glasses to take pictures of your shelves and send the inventory information to the cloud."

[0990] "Capture your customers' facial expressions, analyze them with an emotion engine, and offer sales strategy advice."

[0991] "Propose additional orders or changes to shelf locations based on recognized inventory information."

[0992] This makes it possible to implement a system according to a specific embodiment of the present invention.

[0993] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0994] Step 1:

[0995] A user wears smart glasses and activates the smart glasses' camera to scan the inventory on the shelves while walking around the store. The smart glasses then take an image of the entire display shelf and capture the image data. The input here is the image captured by the camera, and the output is the image data stored in the smart glasses.

[0996] Step 2:

[0997] Image data captured by the device (smart glasses) is sent to a cloud server via the Internet. HTTP or HTTPS is used as the transmission protocol. The input is image data, and the output is image data stored on the cloud server.

[0998] Step 3:

[0999] The server passes the received image data to the generative AI model, which then performs image analysis. The generative AI model extracts the product's part number and quantity from the image data, and recognizes the inventory stacked in the back as well. The input is the transmitted image data, and the output is the extracted product information (part number, quantity).

[1000] Step 4:

[1001] The server organizes the product information extracted by the generative AI model, organizes it as digital data for each product, and stores it in a database. The input is the extracted product information, and the output is organized digital data stored in the database.

[1002] Step 5:

[1003] The server converts the organized inventory information into voice data and repeats it back to the user using the device's (smart glasses') voice output function. This uses a voice synthesis service such as Google Cloud Text-to-Speech. The input is organized digital data, and the output is voice data.

[1004] Step 6:

[1005] When a user wants to correct a false positive using the voice input function, they input the correction instructions through the microphone of the smart glasses. The input is the user's voice instruction, which is converted into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The output is text data.

[1006] Step 7:

[1007] The server analyzes the text data acquired through voice input and compares it with existing database information to correct misjudgments. The input is text data of the user's voice instructions, and the output is corrected inventory information.

[1008] Step 8:

[1009] The server analyzes the inventory data and generates ordering and discount proposals using an AI model, taking into account the user's emotional data using an emotion engine. The input is the revised inventory information and the user's emotional information, and the output is the generated ordering and discount proposals.

[1010] Step 9:

[1011] The server notifies the user of the generated order proposals and discount proposals. When notifying, the method and content of the notification are adjusted based on the user's emotional state using an emotion engine. The input is the proposal data, and the output is a notification to the user.

[1012] The above is a detailed description of the processing steps of the system for specifically implementing the present invention.

[1013] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1014] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1015] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1016] [Third embodiment]

[1017] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1018] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1019] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1020] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1021] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1022] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1023] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1024] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1025] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1026] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1027] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1028] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1029] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and management advice provision.

[1030] 1. Image capture and data transmission

[1031] Subject: User and Device

[1032] A user uses a terminal (e.g., a smartphone or dedicated image capture device) to take a photo of the storage shelf or the area where products are displayed. The captured image is sent to the server by the terminal. The terminal then uploads the image data to the server via the Internet.

[1033] 2. Image Recognition and Product Information Extraction

[1034] Subject: Server and Generative AI Model

[1035] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[1036] 3. Organize and store product information

[1037] Subject: Server

[1038] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[1039] 4. Voice confirmation and correction

[1040] Subject: Terminal and User

[1041] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instructions. They can give instructions such as, "There are 4 units of product A."

[1042] 5. Correction and storage of data

[1043] Subject: Server

[1044] The server analyzes the user's voice commands and modifies the data, updating the database based on the voice commands to maintain accurate inventory information.

[1045] 6. Providing management advice

[1046] Subject: Server

[1047] The server analyzes the inventory data and generates ordering suggestions and discount suggestions for inventory clearance. The server then notifies the user of the generated suggestions. For example, the notification may be in the form of "Product A is low in stock, so we recommend ordering more. Also, product B is not selling well, so we suggest a 20% discount."

[1048] 7. Providing display proposals

[1049] Subject: Server

[1050] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided.

[1051] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[1052] The processing flow will be explained below.

[1053] Step 1:

[1054] Subject: User

[1055] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[1056] Step 2:

[1057] Subject: Terminal

[1058] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[1059] Step 3:

[1060] Subject: Server

[1061] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[1062] Step 4:

[1063] Subject: Generative AI model

[1064] A generative AI model analyzes image data and extracts product information. The model identifies each product in the image, recognizes the product number and quantity associated with each product, and recognizes overlapping products and hidden inventory.

[1065] Step 5:

[1066] Subject: Server

[1067] The server receives product information from the generative AI model, organizes the received product information as digital data, and stores information such as the number of items for each product number in a database.

[1068] Step 6:

[1069] Subject: Terminal

[1070] The terminal reads back the inventory information obtained from the server by voice. Using the voice output function, it notifies the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1071] Step 7:

[1072] Subject: User

[1073] The user confirms the result by voice and issues instructions to correct any incorrect judgments. For example, the user can send a correction instruction by voice inputting "There are four units of product A."

[1074] Step 8:

[1075] Subject: Terminal

[1076] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[1077] Step 9:

[1078] Subject: Server

[1079] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is saved back into the database.

[1080] Step 10:

[1081] Subject: Server

[1082] The server analyzes inventory data and generates ordering and discount proposals. Data analysis algorithms are used to make appropriate proposals based on the inventory situation.

[1083] Step 11:

[1084] Subject: Server

[1085] The server notifies the user of the recommendation results, such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount."

[1086] Step 12:

[1087] Subject: Server

[1088] The server analyzes data on best-selling products and generates optimization proposals for display and shelf layout. The proposals include specific advice such as "Product A should be placed in a prominent position."

[1089] Step 13:

[1090] Subject: Server

[1091] The server notifies the user of the generated display proposal, who receives the proposal via app notification or email and makes changes to the product placement, etc.

[1092] This is the specific process flow of the inventory management system that uses image and voice recognition technology. This system enables efficient and accurate inventory management and management advice.

[1093] Example 1

[1094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1095] Conventional inventory management systems require a lot of manual checking, which makes them inefficient and prone to misjudgments and errors in updating information. They also lack the data needed to make management decisions based on inventory status, or advice on optimal product display methods, making comprehensive inventory management and sales promotion difficult.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1097] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, a means for generating shelf optimization proposals, and a means for notifying the user of the shelf optimization proposals. This not only enables efficient and accurate inventory management, but also enables the provision of data useful for management decisions and specific advice on optimal product display methods.

[1098] "Image capturing means" refers to a device or mechanism for capturing images of product shelves or display shelves.

[1099] The "means for transmitting image data to a server" is a communication means for uploading captured image data to a server via the Internet.

[1100] "Means for analyzing image data using a generative AI model and extracting product information" refers to analytical means that uses a generative AI model to recognize the product number and quantity of products in the image data.

[1101] "Means for organizing and storing product information as digital data" refers to means for systematizing product information extracted by a generative AI model and storing it in digital data format.

[1102] The "means for repeating product information by voice using voice output means" refers to voice output means for outputting product information as voice and notifying the user.

[1103] The "means for correcting erroneous determination using a voice input means" is a means for receiving voice input from the user and correcting erroneously determined information.

[1104] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to analytical means for analyzing inventory data and generating product ordering and discount proposals.

[1105] The "means for notifying the user of the proposal results" is a means for notifying the user of the generated order proposal or discount proposal.

[1106] The "means for generating shelf optimization proposals" is an analytical means for generating proposals for optimizing the display and placement of products.

[1107] The "means for notifying the user of the shelf optimization proposal" is a means for notifying the user of the generated shelf optimization proposal.

[1108] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and provision of management advice and shelf layout optimization suggestions.

[1109] First, a user uses a smartphone or a dedicated image capture device to take a photo of a storage shelf or an area where products are displayed. For example, a user takes a photo of a beverage shelf in a supermarket with a smartphone. This image capture means is a smartphone or a high-performance camera device.

[1110] Next, the device sends the captured image data to a server. The device uploads the image data to the server using Wi-Fi or mobile data communication. For example, a smartphone sends the captured image to a server via Wi-Fi.

[1111] The server passes the received image data to an AI model for image recognition. The generative AI model identifies the product and determines the product number and quantity. This analysis includes the ability to recognize inventory stacked at the back. For example, the server analyzes the image it receives and identifies the type and quantity of beverage bottles. In this process, it also recognizes products hidden at the back.

[1112] The server then organizes the product information extracted by the generative AI model and stores it as digital data. Product numbers, quantities, and other necessary information are stored in a database. For example, the server stores the product numbers and quantities of beverages in a database.

[1113] The device notifies the user of the saved product information by voice. For example, the device may output a voice message saying, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can correct the incorrect judgment using the voice input function. For example, the user may input a voice message saying, "There are 4 units of product A." This voice output and input method is performed using a smart speaker or voice recognition software.

[1114] The server analyzes the user's voice instructions and updates the contents of the database. For example, if the server receives the instruction "There are four units of product A," it updates the quantity of product A in the database to four units.

[1115] Furthermore, the server analyzes the inventory data and generates ordering and discount suggestions. For example, a suggestion might be generated such as, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we suggest a 20% discount." The results of this suggestion are then notified to the user.

[1116] Finally, the server analyzes the data on best-selling products and generates shelf layout optimization suggestions. For example, specific advice such as "Product A should be placed in a prominent position" is provided. This makes it possible to optimize product display and placement.

[1117] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[1118] Example prompt sentence:

[1119] "Please recognize the product in this image and extract the product number and quantity."

[1120] "Please also recognize the inventory in the background of the image and provide the total number of items."

[1121] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1122] Step 1:

[1123] Input: Images of product shelves or display cases taken by users using smartphones or dedicated image capture devices

[1124] How it works: A user uses their device to take a photo of a shelf from a suitable angle. They take multiple images to ensure a clear view and to see inventory at the back.

[1125] Output: Image data saved on the device

[1126] Step 2:

[1127] Input: Image data stored on the device

[1128] How it works: The device uses Wi-Fi or mobile data to upload the captured image data to the server.

[1129] Output: Image data transferred to the server

[1130] Step 3:

[1131] Input: Image data transferred to the server

[1132] Operation: The server passes the received image data to the AI ​​model, which then performs image recognition. It identifies the product and determines the product number and quantity. It also recognizes inventory stacked in the back.

[1133] Output: Product information extracted by the generative AI model (product number, quantity, etc.)

[1134] Step 4:

[1135] Input: Product information extracted by the generative AI model

[1136] How it works: The server organizes product information and stores it in a database in digital format. The data stored includes the product number, quantity, and other necessary information for each product.

[1137] Output: Organized product information stored in a database

[1138] Step 5:

[1139] Input: Product information stored in the database

[1140] Operation: The device notifies the user of product information by voice. The device outputs a message such as, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1141] Output: Audio information notified to the user

[1142] Step 6:

[1143] Input: Audio information announced to the user

[1144] Operation: The user uses the voice input function to correct any incorrect judgments as necessary. For example, the user might say, "There are four units of product A."

[1145] Output: Corrected voice instructions entered into the device

[1146] Step 7:

[1147] Input: Modified voice instructions entered into the device

[1148] Operation: The server analyzes the voice instructions and modifies the contents of the database. The modified information is updated in the database.

[1149] Output: Database reflecting revised inventory information

[1150] Step 8:

[1151] Input: Database reflecting corrected inventory information

[1152] How it works: The server analyzes inventory data and generates ordering and discount suggestions. It generates suggestions like, "Product A is low in stock, so we recommend you order more. Product B is not selling well, so we suggest a 20% discount."

[1153] Output: Generated order proposals and discount proposals

[1154] Step 9:

[1155] Input: Generated order proposals and discount proposals

[1156] Operation: The server notifies the user of order suggestions and discount suggestions. For example, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we offer a 20% discount."

[1157] Output: Order proposals and discount proposals notified to the user

[1158] Step 10:

[1159] Input: Order proposals and discount proposals notified to the user

[1160] How it works: The server analyzes data on best-selling products and generates shelf layout optimization suggestions. It provides specific advice such as "Product A should be placed in a prominent position."

[1161] Output: Generated shelf optimization proposals

[1162] Step 11:

[1163] Input: Generated shelf optimization proposals

[1164] Operation: The server notifies the user of shelf optimization suggestions, such as "Product A should be placed in a prominent position."

[1165] Output: Shelf optimization suggestions communicated to the user

[1166] (Application example 1)

[1167] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1168] Modern retail businesses require efficient and accurate inventory management. However, traditional inventory management methods require a lot of manual checking and updating, which takes time and effort and is prone to human error. Additionally, product placement and display optimization often rely on experience and intuition, making management based on objective data difficult. To solve these problems, a system is needed that provides efficient and accurate inventory management and management advice based on objective data.

[1169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1170] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an interface means for acquiring image data and voice data using a smart device and confirming and correcting the recognition results. This significantly reduces the effort and errors involved in conventional inventory management, enables efficient and accurate inventory management, and further enables optimization of ordering and display based on objective data.

[1171] "Image capturing means" refers to a device or function for capturing images of the arrangement and storage conditions of items and creating image data thereof.

[1172] The "means for transmitting image data to a server" refers to a function or protocol for transmitting captured image data to a server via a network.

[1173] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and automatically performs tasks such as image recognition and data analysis.

[1174] "Means for extracting product information" refers to analytical technology for extracting detailed product information (product number, quantity, etc.) from image data.

[1175] "Means for organizing and storing product information as digital data" refers to a system or method for organizing and storing extracted product information in a database or the like.

[1176] "Means for repeating product information aloud using voice output means" refers to a voice synthesis technology that outputs extracted and organized product information as voice and informs the user.

[1177] The "means for correcting erroneous determinations using a voice input means" refers to a voice recognition technology that accepts and analyzes a user's voice instructions and corrects erroneous data.

[1178] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to an algorithm or system that analyzes inventory data and proposes optimal order quantities and discount strategies.

[1179] The "means for notifying the user of the proposal results" refers to a messaging system or notification function for informing the user of the created order proposal or discount proposal.

[1180] "Interface means for acquiring image data and audio data using a smart device and confirming and correcting the recognition results" refers to a user interface for acquiring image and audio data using a device such as a smartphone or tablet and confirming and correcting the results.

[1181] MODE FOR CARRYING OUT THE INVENTION

[1182] The present invention is an efficient inventory management system using automated image recognition and a voice interface, and is configured as follows.

[1183] 1. Image capture method

[1184] Users use smart devices (smartphones or tablets) to take pictures of product shelves and storage areas, which generates image data that allows users to visually grasp the placement and quantity of items.

[1185] 2. Means of sending image data to the server

[1186] The terminal sends the captured image data to a server via the Internet, so the smart device must have network connectivity.

[1187] 3. A means of analyzing image data and extracting product information using generative AI models

[1188] The server inputs the received image data into a generative AI model for analysis. The generative AI model recognizes the product number and quantity contained in the image, and also analyzes the depth information of the image to recognize inventory that is hidden beneath the surface. This allows accurate product information to be extracted.

[1189] 4. A means of organizing and storing product information as digital data

[1190] The product information extracted by the server is organized as digital data by item, quantity, etc. The organized data is stored in a database and used for subsequent processing.

[1191] 5. A means for repeating product information aloud using a voice output means

[1192] The terminal has the ability to output the inventory information stored in the database aloud, for example, notifying the user in the form of "There are 10 units of product A and 5 units of product B."

[1193] 6. A method for correcting misjudgments using voice input

[1194] Users can use the voice input function to give instructions to the system, such as "I want 9 units of product A," which is then analyzed using voice recognition technology and the information in the database is updated.

[1195] 7. A means of analyzing inventory data and generating ordering and discounting proposals

[1196] The server analyzes the inventory data stored in the database to generate order requirements and discount suggestions, using algorithms to make suggestions based on sales trends and stock levels, and notifies the user.

[1197] 8. Means of notifying users of the proposed results

[1198] The generated ordering and discount proposals are notified to the user via the terminal. For example, the notification may say, "Product B is low in stock, so we recommend you order more. Also, product C is not selling well, so we suggest a 20% discount."

[1199] 9. Interface means for acquiring image data and voice data using a smart device, and for confirming and correcting the recognition results.

[1200] The system uses a smart device to acquire image and voice data and provides a user interface for confirming and correcting the recognition results. For example, a user can take a photo of a shelf with their smartphone and confirm the results by voice.

[1201] This significantly improves the efficiency of inventory management and reduces human error. Accurate product recognition using generative AI models, combined with the ease of a voice interface, improves the accuracy and efficiency of overall management.

[1202] Examples of specific examples and prompts

[1203] As a concrete example, a store clerk takes a photo of a shelf with their smartphone, and the app automatically reads the inventory information and notifies them via voice. The clerk can also give a voice instruction such as "There are 9 units of product A, not 10," and the app will correct the data and notify them again. Furthermore, the app will offer management advice such as "We recommend ordering product B. Also, please place product C in a prominent location."

[1204] Example prompt for a generative AI model:

[1205] "Inventor data: Product A: 10 pieces, Product B: 5 pieces, Product C: 2 pieces. Provide stock refill and display suggestions."

[1206] This prompt sentence generates appropriate management advice and display suggestions.

[1207] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1208] Step 1:

[1209] Users use their smart devices to take pictures of product shelves and storage areas, and then use the camera function to obtain image data that allows them to visually grasp the placement and quantity of items.

[1210] Input: Actual footage of shelves and storage areas

[1211] Output: Image data saved on the smart device

[1212] Step 2:

[1213] The device sends the acquired image data to a cloud server via the Internet, using the network connection function to send the image data to a specific server.

[1214] Input: Image data on a smart device

[1215] Output: Image data stored on a cloud server

[1216] Step 3:

[1217] The server inputs the received image data into the generative AI model for analysis. The generative AI model detects and recognizes the product numbers and quantities of the products contained in the image, and also analyzes the depth information of the image if necessary.

[1218] Input: Image data stored on a cloud server

[1219] Output: Specific product information such as product number and quantity

[1220] Step 4:

[1221] The server organizes the product information extracted by the generative AI model as digital data and stores it in a database. Through this process, information such as product numbers and quantities for each product is systematically organized.

[1222] Input: Product information extracted by the generative AI model

[1223] Output: Organized product information stored in a database

[1224] Step 5:

[1225] The terminal notifies the user of the inventory information stored in the database using a voice output function, for example, by repeating aloud, "There are 10 units of product A and 5 units of product B."

[1226] Input: Stock information stored in the database

[1227] Output: Voice output of stock information

[1228] Step 6:

[1229] The user can use the voice input function to give correction instructions to the system as needed, for example, "There are nine units of product A."

[1230] Input: User correction instructions via voice input

[1231] Output: Audio data received by the device

[1232] Step 7:

[1233] The server analyzes the received voice data and corrects any incorrect judgments. It uses voice recognition technology to analyze the instructions and corrects the relevant data in the database.

[1234] Input: Audio data received by the device

[1235] Output: Database with revised product data

[1236] Step 8:

[1237] The server analyzes the inventory data in the database and generates ordering and discount proposals, using algorithms to create optimal proposals based on sales trends, inventory levels, etc.

[1238] Input: Product information and inventory data stored in a database

[1239] Output: Generated order proposals and discount proposals

[1240] Step 9:

[1241] The generated recommendations are then sent to the user via their device. For example, they might say, "Product B is low in stock, so we recommend placing an additional order. Also, product C is not selling well, so we suggest a 20% discount."

[1242] Input: Generated order proposals and discount proposals

[1243] Output: The user is notified of the proposed results.

[1244] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1245] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[1246] 1. Image capture and data transmission

[1247] Subject: User and Device

[1248] A user starts the camera app on their device and takes an image of the display shelf. For example, they can take the image using a smartphone or a dedicated image capture device. The captured image is sent to the server by the device. The device then uploads the image data to the server via the Internet.

[1249] 2. Image Recognition and Product Information Extraction

[1250] Subject: Server and Generative AI Model

[1251] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[1252] 3. Organize and store product information

[1253] Subject: Server

[1254] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[1255] 4. Voice confirmation and correction

[1256] Subject: Terminal and User

[1257] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instruction. They can give instructions such as, "There are 4 units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[1258] 5. Correction and storage of data

[1259] Subject: Server

[1260] The server analyzes the user's voice commands and corrects the data. The database is updated based on the voice commands to maintain accurate inventory information. Voice-to-text conversion is performed within the device, and the converted text data is sent to the server.

[1261] 6. Providing management advice

[1262] Subject: Server

[1263] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine can adjust the suggestions based on the user's emotional state.

[1264] 7. Notification of Proposal

[1265] Subject: Server and terminal

[1266] The server notifies the user of the generated suggestions. For example, it may send a message such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount." Based on the emotion engine, the notification method and content can be adjusted according to the user's emotional state.

[1267] 8. Providing display proposals

[1268] Subject: Server

[1269] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. These include specific advice such as "Product A should be placed in a prominent position." The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[1270] The above is a specific embodiment of the present invention. This system uses image recognition and voice recognition technologies to enable efficient and accurate inventory management, as well as management advice that takes into account the user's emotions. This optimizes the inventory management process and enables more flexible and effective management decisions.

[1271] The processing flow will be explained below.

[1272] Step 1:

[1273] Subject: User

[1274] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[1275] Step 2:

[1276] Subject: Terminal

[1277] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[1278] Step 3:

[1279] Subject: Server

[1280] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[1281] Step 4:

[1282] Subject: Generative AI model

[1283] A generative AI model analyzes image data to extract the part number and quantity of each product. The model identifies products based on their visual characteristics and recognizes overlapping products and hidden inventory.

[1284] Step 5:

[1285] Subject: Server

[1286] The server receives product information from the generative AI model and organizes it as digital data, which is then stored in a database.

[1287] Step 6:

[1288] Subject: Terminal

[1289] The terminal repeats the inventory information received from the server by voice, using the voice output function to notify the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1290] Step 7:

[1291] Subject: User

[1292] The user confirms the result by voice and issues a correction instruction if there is an incorrect judgment. For example, the user can send a correction instruction by voice inputting "There are four units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[1293] Step 8:

[1294] Subject: Terminal

[1295] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[1296] Step 9:

[1297] Subject: Server

[1298] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is stored in the database.

[1299] Step 10:

[1300] Subject: Server

[1301] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine adjusts the suggestions based on the user's emotional state.

[1302] Step 11:

[1303] Subject: Server

[1304] The server notifies the user of the recommendation results. For example, it may send a message such as, "Product A is low in stock, so we recommend you place an additional order," or, "Product B is not selling well, so we suggest a 20% discount." The server adjusts the notification method and content according to the user's emotional state based on the emotion engine.

[1305] Step 12:

[1306] Subject: Server and terminal

[1307] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided. The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[1308] Step 13:

[1309] Subject: User

[1310] The user rearranges or orders products based on the suggestions received from the server. For example, the user may "order additional product A" or "apply a discount on product B."

[1311] This concludes the process flow of an inventory management system that combines image and voice recognition technologies with user emotion recognition. This system enables efficient and accurate inventory management and provides management advice that takes into account the user's emotional state.

[1312] Example 2

[1313] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1314] Conventional inventory management systems make it difficult to grasp accurate inventory information, often requiring manual confirmation. Furthermore, they do not provide management advice that takes into account the user's emotional state, limiting the effectiveness of management decisions. Furthermore, there is a lack of means to accurately identify hidden inventory and product part numbers and quantities. There is a need for a system that can solve these issues and provide efficient and accurate inventory management.

[1315] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1316] In this invention, the server includes an image capturing means, a means for transmitting image data to an information processing device, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an emotion engine for analyzing the user's emotional state and adjusting the proposal content. This significantly improves the accuracy and efficiency of inventory management and makes it possible to provide management advice tailored to the user's emotional state.

[1317] "Image capturing means" refers to a device that allows a user to capture an image of a product display shelf, and includes a smartphone or a dedicated image capture device.

[1318] The "means for transmitting image data to an information processing device" is a process for transmitting captured image data to a server, and includes a function for uploading data via the Internet.

[1319] A "generative AI model" is an artificial intelligence model that analyzes image data, recognizes objects in the image, and extracts corresponding information.

[1320] "Means for extracting product information" includes a function to identify product part numbers and quantities in an image using a generative AI model.

[1321] "Means for organizing and storing product information as digital data" refers to the process of organizing the extracted product information into a specific format and storing it in a database.

[1322] "Audio output means" refers to a function that enables the terminal to convey stored product information to the user by voice, and includes a voice assistant and a speaker.

[1323] "Voice input means" refers to a function for receiving a user's voice instructions and analyzing the voice data, and includes a microphone and voice recognition software.

[1324] An "emotion engine" is a system that includes algorithms for analyzing voice data and understanding the user's emotional state.

[1325] The "means for analyzing inventory data and generating order proposals and discount proposals" is a process for automatically generating appropriate order and discount proposals based on inventory information.

[1326] "Means for notifying users of proposal results" refers to a function for informing users of the generated management advice and proposal content, and includes push notifications and message sending.

[1327] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[1328] In implementing this system, the following hardware and software is recommended:

[1329] 1. Hardware:

[1330] Users use smartphones or dedicated image capture devices to take images.

[1331] The terminals include computers and smartphones that can connect to the Internet.

[1332] 2. Software:

[1333] Generative AI models: For example, object detection algorithms such as "YOLO" and "Detectron2" are used.

[1334] Database: We use database management systems such as "MySQL" or "PostgreSQL" to store inventory information.

[1335] Speech recognition: Services such as "Google Cloud Speech-to-Text API" and "Amazon Alexa" are used to analyze voice data.

[1336] Emotion Engine: Implements emotion recognition algorithms to analyze the user's emotional state.

[1337] System processing overview

[1338] 1. Image capture and data transmission

[1339] Users use their smartphones to take pictures of the shelves, which are then compressed and uploaded to a server via the Internet.

[1340] 2. Image Recognition and Product Information Extraction

[1341] The server sends the received image data to the generative AI model, which analyzes the products in the image and identifies the product number and quantity. The detection results are returned to the server in JSON format.

[1342] 3. Organize and store product information

[1343] The server analyzes the data from the generative AI model and organizes the product number and quantity information for each product. The organized data is stored in a database and serves as the basis for the next process.

[1344] 4. Voice confirmation and correction

[1345] The device outputs inventory information stored in the database and repeats it back to the user. The user can use voice input to correct any incorrect judgments. The emotion engine analyzes the voice data and understands the user's emotional state.

[1346] 5. Correction and storage of data

[1347] The server analyzes the user's voice commands and updates the database with the corrected data. The voice input is converted into text on the device and sent to the server.

[1348] 6. Providing management advice

[1349] The server analyzes inventory data and generates ordering and discount offers, with an emotional engine adapting the offers based on the user's emotional state.

[1350] 7. Notification of Proposal

[1351] The server notifies the user of the generated suggestions using push notifications or messaging.

[1352] 8. Providing display proposals

[1353] The server analyzes best-selling products and generates optimization proposals for display and shelf layout. The emotion engine makes proposals that take into account the user's emotional state.

[1354] Specific examples

[1355] 1. A user takes a picture of a store's display shelves with their smartphone and sends it to the server.

[1356] 2. The server passes the image data to a generative AI model (e.g., YOLO) and analyzes the product number and quantity.

[1357] 3. The analysis results are saved in the database in the format of "5 units of product A, 3 units of product B, 8 units of product C."

[1358] 4. The device uses Google Assistant to tell the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1359] 5. The user inputs "Product A is 4 units" by voice and gives a correction instruction. The emotion engine analyzes the user's emotional state.

[1360] 6. The corrected data is updated in the database.

[1361] 7. The server uses TensorFlow to analyze the inventory data and notifies the customer that "Product A is low in stock, so we recommend placing an additional order."

[1362] 8. Generate display optimization suggestions for best-selling products, such as "Product A should be placed in a prominent position."

[1363] In this way, the present invention realizes a system in which the user, terminal, and server cooperate to provide efficient and accurate inventory management and management advice that takes into account the user's feelings.

[1364] Prompt Sentence Examples

[1365] "Upload image data of the display shelves and use the generative AI model to analyze the product part numbers and quantities. Then, read back the analysis results and correct any misidentifications with voice instructions. Finally, provide management advice based on the inventory status and generate display suggestions."

[1366] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1367] Step 1:

[1368] Image capture and data transmission

[1369] input:

[1370] A user picks up a smartphone, launches the camera app, and takes a picture of the display shelf.

[1371] Specific behavior:

[1372] The user taps the camera app on the smartphone's operation screen to launch it.

[1373] While looking at the camera preview screen, take a photo of the display shelf at the appropriate angle and distance.

[1374] The captured image file is saved in the device's local storage.

[1375] Data processing and output:

[1376] The image data captured by the device is compressed and saved in a file format such as JPEG.

[1377] The compressed image file is uploaded to a server via the Internet.

[1378] Step 2:

[1379] Image recognition and product information extraction

[1380] input:

[1381] The server receives the image data sent from the terminal.

[1382] Specific behavior:

[1383] The server receives the image data through an HTTP request.

[1384] The received image data is saved in a specific directory.

[1385] Data processing and output:

[1386] The server inputs the stored image data into a generative AI model (e.g., YOLO).

[1387] The generative AI model analyzes the products in the image, identifies the product number and quantity in JSON format, and returns it to the server.

[1388] Step 3:

[1389] Organizing and storing product information

[1390] input:

[1391] The server receives the JSON-formatted product information data returned from the generated AI model.

[1392] Specific behavior:

[1393] The server reads the JSON data and extracts the product number and quantity information.

[1394] The extracted information is then organized and presented in a digital format.

[1395] Data processing and output:

[1396] Save data for each product (e.g., "Product A: 5 units, Product B: 3 units") in the database.

[1397] Save the information in a database management system (e.g. MySQL) using the INSERT statement.

[1398] Step 4:

[1399] Voice confirmation and correction

[1400] input:

[1401] The server transmits the organized inventory information to the terminal, and the terminal starts outputting audio.

[1402] Specific behavior:

[1403] The device converts the inventory information from text to speech, and the voice assistant (e.g., Google Assistant) repeats, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1404] The user can use the voice input function to give instructions to correct any incorrect judgments (e.g., "There are four units of product A").

[1405] Data processing and output:

[1406] The device converts the voice input into text and sends the text data to the server.

[1407] The emotion engine analyzes the voice data and understands the user's emotional state.

[1408] Step 5:

[1409] Data correction and storage

[1410] input:

[1411] The server receives the user's voice input data and begins analyzing it.

[1412] Specific behavior:

[1413] The server analyzes the received text data and identifies the incorrectly determined inventory information.

[1414] Generate the corresponding SQL statements to correct the misjudged inventory information.

[1415] Data processing and output:

[1416] Use the UPDATE statement to update the corrected inventory information in the database (e.g., "UPDATE inventory information table SET quantity=4 WHERE product ID='product A'").

[1417] The corrected accurate inventory information is saved in the database.

[1418] Step 6:

[1419] Providing management advice

[1420] input:

[1421] The server generates ordering suggestions and discount suggestions based on an up-to-date inventory database.

[1422] Specific behavior:

[1423] The server analyzes the inventory data and generates ordering and discount suggestions using data analysis algorithms (e.g., TensorFlow).

[1424] The emotion engine adjusts suggestions based on the user's emotional state.

[1425] Data processing and output:

[1426] Based on the detected trends, suggestions such as "Product A is low in stock, so we recommend placing an additional order" are generated.

[1427] The server sends the generated proposal to the terminal.

[1428] Step 7:

[1429] Proposal Notification

[1430] input:

[1431] The management advice and proposals generated by the server are sent to the terminal.

[1432] Specific behavior:

[1433] The server creates a notification message and sends a push notification to the user's smartphone using Firebase Cloud Messaging or similar.

[1434] The device receives the notification and displays it to the user.

[1435] Data processing and output:

[1436] The proposal is displayed to the user and management advice is provided (e.g., "Product A is low in stock, so we recommend placing an additional order").

[1437] Step 8:

[1438] Providing display proposals

[1439] input:

[1440] The server generates display suggestions based on data on best-selling items.

[1441] Specific behavior:

[1442] The server analyzes top-selling products and generates display optimization suggestions.

[1443] The emotion engine takes into account the user's emotional state and adjusts the suggestions accordingly.

[1444] Data processing and output:

[1445] The proposal is generated in the form of "Product A should be placed in a prominent position."

[1446] The proposal is sent to the terminal and displayed to the user.

[1447] In this way, the system of the present invention combines image recognition technology and voice recognition technology to provide efficient and accurate inventory management and management advice that takes into account the user's emotions.

[1448] (Application example 2)

[1449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1450] Conventional inventory management systems have the problem that it is difficult to accurately grasp and manage product information, and they lack management advice that takes into account users' emotions and purchasing intentions, making it difficult to fully optimize inventory and promote sales.

[1451] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for capturing and analyzing the user's emotional state using a smart device, and a means for adjusting the content of suggestions and notification methods based on the user's emotional state. This enables accurate understanding of inventory information and flexible management advice based on the user's emotions.

[1452] "Image capturing means" refers to a device or system for capturing images of shelves and inventory.

[1453] "Means for transmitting image data to a server" refers to a function or device for transmitting captured images to a cloud or remote server via a network.

[1454] A "generative AI model" is a machine learning model that uses artificial intelligence technology to analyze image data and extract specific patterns and features.

[1455] "Means for organizing and storing product information as digital data" refers to a device or system for structuring extracted product information and storing it in a database.

[1456] The "audio output means" is a device or system for outputting the stored product information as audio.

[1457] The "voice input means" is a device or system for recognizing voice instructions from the user and converting the contents of the voice instructions into text data.

[1458] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to a device or system for analyzing the current inventory status and generating proposals for the need for additional orders and discounts.

[1459] The "means for notifying the user of the proposal results" is a device or system for notifying the user of the generated order proposal or discount proposal.

[1460] "Means for capturing and analyzing a user's emotional state using a smart device" refers to a device or system that uses smart glasses or other wearable devices to obtain and analyze a user's emotional state from voice and facial expressions.

[1461] "Means for adjusting the content of proposals and notification methods based on the user's emotional state" refers to a device or system for appropriately adjusting the content of order proposals and discount proposals and notification methods based on the analyzed emotional state of the user.

[1462] The present invention is a smart device-based system that combines efficient inventory management in brick-and-mortar stores with user emotion analysis. The system includes image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, provision of management advice, and user emotion recognition using an emotion engine. A specific embodiment of the system is described in detail below.

[1463] Hardware and software used

[1464] Hardware: Smart glasses (with camera, microphone, and display)

[1465] Software: generative AI models, emotion engines, speech recognition software (e.g., Google Cloud Speech-to-Text)

[1466] Image capture and data transmission

[1467] A user wears smart glasses and walks around the store scanning the inventory on the shelves. The smart glasses' camera captures images and sends them over the internet to a cloud server. The image data obtained through this process is then sent to the server via a high-speed data transmission channel.

[1468] Image recognition and product information extraction

[1469] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This makes it possible to determine the product's part number and quantity, as well as recognize inventory stacked deep within the image. This image recognition technology enables more accurate inventory information to be obtained.

[1470] Organizing and storing product information

[1471] The server then processes the product information extracted by the generative AI model and organizes it as digital data for each product. The information includes product numbers, quantities, and other necessary data. This information is then stored in a database and becomes the basis for the next process.

[1472] Voice confirmation and correction

[1473] The smart glasses' voice output function is used to repeat the organized inventory information to the user. For example, the voice output may say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments with voice instructions. During this process, the emotion engine analyzes the voice data and understands the user's emotional state.

[1474] Providing management advice

[1475] The server analyzes inventory data and generates ordering and discount suggestions. It uses analytical algorithms to make appropriate suggestions based on stock availability. The emotion engine can also adjust the suggestions based on the user's emotional state. For example, it can make specific ordering suggestions such as "Product A is low in stock, so we recommend you order more," or discount suggestions such as "Product B is not selling well, so we suggest a 20% discount."

[1476] Suggestions and notifications based on emotional state

[1477] The server notifies the user of the generated order proposals and discount proposals. Based on the emotion engine, the server adjusts the notification method and content according to the user's emotional state. This allows the server to provide information in a form that is psychologically easy for the user to accept.

[1478] Examples of specific examples and prompts

[1479] Examples:

[1480] A user wears smart glasses and takes pictures of the shelves while walking around the store.

[1481] Product information is sent to the server, and the generative AI model analyzes the inventory information.

[1482] When the user requests a correction by voice input, the user may give a voice instruction such as "There are four units of product A."

[1483] Example prompt sentence:

[1484] "Use the camera in your smart glasses to take pictures of your shelves and send the inventory information to the cloud."

[1485] "Capture your customers' facial expressions, analyze them with an emotion engine, and offer sales strategy advice."

[1486] "Propose additional orders or changes to shelf locations based on recognized inventory information."

[1487] This makes it possible to implement a system according to a specific embodiment of the present invention.

[1488] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1489] Step 1:

[1490] A user wears smart glasses and activates the smart glasses' camera to scan the inventory on the shelves while walking around the store. The smart glasses then take an image of the entire display shelf and capture the image data. The input here is the image captured by the camera, and the output is the image data stored in the smart glasses.

[1491] Step 2:

[1492] Image data captured by the device (smart glasses) is sent to a cloud server via the Internet. HTTP or HTTPS is used as the transmission protocol. The input is image data, and the output is image data stored on the cloud server.

[1493] Step 3:

[1494] The server passes the received image data to the generative AI model, which then performs image analysis. The generative AI model extracts the product's part number and quantity from the image data, and recognizes the inventory stacked in the back as well. The input is the transmitted image data, and the output is the extracted product information (part number, quantity).

[1495] Step 4:

[1496] The server organizes the product information extracted by the generative AI model, organizes it as digital data for each product, and stores it in a database. The input is the extracted product information, and the output is organized digital data stored in the database.

[1497] Step 5:

[1498] The server converts the organized inventory information into voice data and repeats it back to the user using the device's (smart glasses') voice output function. This uses a voice synthesis service such as Google Cloud Text-to-Speech. The input is organized digital data, and the output is voice data.

[1499] Step 6:

[1500] When a user wants to correct a false positive using the voice input function, they input the correction instructions through the microphone of the smart glasses. The input is the user's voice instruction, which is converted into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The output is text data.

[1501] Step 7:

[1502] The server analyzes the text data acquired through voice input and compares it with existing database information to correct misjudgments. The input is text data of the user's voice instructions, and the output is corrected inventory information.

[1503] Step 8:

[1504] The server analyzes the inventory data and generates ordering and discount proposals using an AI model, taking into account the user's emotional data using an emotion engine. The input is the revised inventory information and the user's emotional information, and the output is the generated ordering and discount proposals.

[1505] Step 9:

[1506] The server notifies the user of the generated order proposals and discount proposals. When notifying, the method and content of the notification are adjusted based on the user's emotional state using an emotion engine. The input is the proposal data, and the output is a notification to the user.

[1507] The above is a detailed description of the processing steps of the system for specifically implementing the present invention.

[1508] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1509] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1510] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1511] [Fourth embodiment]

[1512] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1513] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1514] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1515] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1516] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1517] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1518] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1519] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1520] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1521] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1522] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1523] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1524] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1525] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and management advice provision.

[1526] 1. Image capture and data transmission

[1527] Subject: User and Device

[1528] A user uses a terminal (e.g., a smartphone or dedicated image capture device) to take a photo of the storage shelf or the area where products are displayed. The captured image is sent to the server by the terminal. The terminal then uploads the image data to the server via the Internet.

[1529] 2. Image Recognition and Product Information Extraction

[1530] Subject: Server and Generative AI Model

[1531] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[1532] 3. Organize and store product information

[1533] Subject: Server

[1534] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[1535] 4. Voice confirmation and correction

[1536] Subject: Terminal and User

[1537] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instructions. They can give instructions such as, "There are 4 units of product A."

[1538] 5. Correction and storage of data

[1539] Subject: Server

[1540] The server analyzes the user's voice commands and modifies the data, updating the database based on the voice commands to maintain accurate inventory information.

[1541] 6. Providing management advice

[1542] Subject: Server

[1543] The server analyzes the inventory data and generates ordering suggestions and discount suggestions for inventory clearance. The server then notifies the user of the generated suggestions. For example, the notification may be in the form of "Product A is low in stock, so we recommend ordering more. Also, product B is not selling well, so we suggest a 20% discount."

[1544] 7. Providing display proposals

[1545] Subject: Server

[1546] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided.

[1547] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[1548] The processing flow will be explained below.

[1549] Step 1:

[1550] Subject: User

[1551] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[1552] Step 2:

[1553] Subject: Terminal

[1554] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[1555] Step 3:

[1556] Subject: Server

[1557] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[1558] Step 4:

[1559] Subject: Generative AI model

[1560] A generative AI model analyzes image data and extracts product information. The model identifies each product in the image, recognizes the product number and quantity associated with each product, and recognizes overlapping products and hidden inventory.

[1561] Step 5:

[1562] Subject: Server

[1563] The server receives product information from the generative AI model, organizes the received product information as digital data, and stores information such as the number of items for each product number in a database.

[1564] Step 6:

[1565] Subject: Terminal

[1566] The terminal reads back the inventory information obtained from the server by voice. Using the voice output function, it notifies the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1567] Step 7:

[1568] Subject: User

[1569] The user confirms the result by voice and issues instructions to correct any incorrect judgments. For example, the user can send a correction instruction by voice inputting "There are four units of product A."

[1570] Step 8:

[1571] Subject: Terminal

[1572] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[1573] Step 9:

[1574] Subject: Server

[1575] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is saved back into the database.

[1576] Step 10:

[1577] Subject: Server

[1578] The server analyzes inventory data and generates ordering and discount proposals. Data analysis algorithms are used to make appropriate proposals based on the inventory situation.

[1579] Step 11:

[1580] Subject: Server

[1581] The server notifies the user of the recommendation results, such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount."

[1582] Step 12:

[1583] Subject: Server

[1584] The server analyzes data on best-selling products and generates optimization proposals for display and shelf layout. The proposals include specific advice such as "Product A should be placed in a prominent position."

[1585] Step 13:

[1586] Subject: Server

[1587] The server notifies the user of the generated display proposal, who receives the proposal via app notification or email and makes changes to the product placement, etc.

[1588] This is the specific process flow of the inventory management system that uses image and voice recognition technology. This system enables efficient and accurate inventory management and management advice.

[1589] Example 1

[1590] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1591] Conventional inventory management systems require a lot of manual checking, which makes them inefficient and prone to misjudgments and errors in updating information. They also lack the data needed to make management decisions based on inventory status, or advice on optimal product display methods, making comprehensive inventory management and sales promotion difficult.

[1592] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1593] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, a means for generating shelf optimization proposals, and a means for notifying the user of the shelf optimization proposals. This not only enables efficient and accurate inventory management, but also enables the provision of data useful for management decisions and specific advice on optimal product display methods.

[1594] "Image capturing means" refers to a device or mechanism for capturing images of product shelves or display shelves.

[1595] The "means for transmitting image data to a server" is a communication means for uploading captured image data to a server via the Internet.

[1596] "Means for analyzing image data using a generative AI model and extracting product information" refers to analytical means that uses a generative AI model to recognize the product number and quantity of products in the image data.

[1597] "Means for organizing and storing product information as digital data" refers to means for systematizing product information extracted by a generative AI model and storing it in digital data format.

[1598] The "means for repeating product information by voice using voice output means" refers to voice output means for outputting product information as voice and notifying the user.

[1599] The "means for correcting erroneous determination using a voice input means" is a means for receiving voice input from the user and correcting erroneously determined information.

[1600] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to analytical means for analyzing inventory data and generating product ordering and discount proposals.

[1601] The "means for notifying the user of the proposal results" is a means for notifying the user of the generated order proposal or discount proposal.

[1602] The "means for generating shelf optimization proposals" is an analytical means for generating proposals for optimizing the display and placement of products.

[1603] The "means for notifying the user of the shelf optimization proposal" is a means for notifying the user of the generated shelf optimization proposal.

[1604] The present invention provides an efficient inventory management system for retail businesses, which includes a series of functions: image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, and provision of management advice and shelf layout optimization suggestions.

[1605] First, a user uses a smartphone or a dedicated image capture device to take a photo of a storage shelf or an area where products are displayed. For example, a user takes a photo of a beverage shelf in a supermarket with a smartphone. This image capture means is a smartphone or a high-performance camera device.

[1606] Next, the device sends the captured image data to a server. The device uploads the image data to the server using Wi-Fi or mobile data communication. For example, a smartphone sends the captured image to a server via Wi-Fi.

[1607] The server passes the received image data to an AI model for image recognition. The generative AI model identifies the product and determines the product number and quantity. This analysis includes the ability to recognize inventory stacked at the back. For example, the server analyzes the image it receives and identifies the type and quantity of beverage bottles. In this process, it also recognizes products hidden at the back.

[1608] The server then organizes the product information extracted by the generative AI model and stores it as digital data. Product numbers, quantities, and other necessary information are stored in a database. For example, the server stores the product numbers and quantities of beverages in a database.

[1609] The device notifies the user of the saved product information by voice. For example, the device may output a voice message saying, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can correct the incorrect judgment using the voice input function. For example, the user may input a voice message saying, "There are 4 units of product A." This voice output and input method is performed using a smart speaker or voice recognition software.

[1610] The server analyzes the user's voice instructions and updates the contents of the database. For example, if the server receives the instruction "There are four units of product A," it updates the quantity of product A in the database to four units.

[1611] Furthermore, the server analyzes the inventory data and generates ordering and discount suggestions. For example, a suggestion might be generated such as, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we suggest a 20% discount." The results of this suggestion are then notified to the user.

[1612] Finally, the server analyzes the data on best-selling products and generates shelf layout optimization suggestions. For example, specific advice such as "Product A should be placed in a prominent position" is provided. This makes it possible to optimize product display and placement.

[1613] The above is a specific embodiment of the present invention. This system realizes efficient and accurate inventory management using image recognition and voice confirmation. It also provides management advice, enabling optimal management decisions based on inventory status.

[1614] Example prompt sentence:

[1615] "Please recognize the product in this image and extract the product number and quantity."

[1616] "Please also recognize the inventory in the background of the image and provide the total number of items."

[1617] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1618] Step 1:

[1619] Input: Images of product shelves or display cases taken by users using smartphones or dedicated image capture devices

[1620] How it works: A user uses their device to take a photo of a shelf from a suitable angle. They take multiple images to ensure a clear view and to see inventory at the back.

[1621] Output: Image data saved on the device

[1622] Step 2:

[1623] Input: Image data stored on the device

[1624] How it works: The device uses Wi-Fi or mobile data to upload the captured image data to the server.

[1625] Output: Image data transferred to the server

[1626] Step 3:

[1627] Input: Image data transferred to the server

[1628] Operation: The server passes the received image data to the AI ​​model, which then performs image recognition. It identifies the product and determines the product number and quantity. It also recognizes inventory stacked in the back.

[1629] Output: Product information extracted by the generative AI model (product number, quantity, etc.)

[1630] Step 4:

[1631] Input: Product information extracted by the generative AI model

[1632] How it works: The server organizes product information and stores it in a database in digital format. The data stored includes the product number, quantity, and other necessary information for each product.

[1633] Output: Organized product information stored in a database

[1634] Step 5:

[1635] Input: Product information stored in the database

[1636] Operation: The device notifies the user of product information by voice. The device outputs a message such as, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1637] Output: Audio information notified to the user

[1638] Step 6:

[1639] Input: Audio information announced to the user

[1640] Operation: The user uses the voice input function to correct any incorrect judgments as necessary. For example, the user might say, "There are four units of product A."

[1641] Output: Corrected voice instructions entered into the device

[1642] Step 7:

[1643] Input: Modified voice instructions entered into the device

[1644] Operation: The server analyzes the voice instructions and modifies the contents of the database. The modified information is updated in the database.

[1645] Output: Database reflecting revised inventory information

[1646] Step 8:

[1647] Input: Database reflecting corrected inventory information

[1648] How it works: The server analyzes inventory data and generates ordering and discount suggestions. It generates suggestions like, "Product A is low in stock, so we recommend you order more. Product B is not selling well, so we suggest a 20% discount."

[1649] Output: Generated order proposals and discount proposals

[1650] Step 9:

[1651] Input: Generated order proposals and discount proposals

[1652] Operation: The server notifies the user of order suggestions and discount suggestions. For example, "Product A is low in stock, so we recommend you order more. Also, product B is not selling well, so we offer a 20% discount."

[1653] Output: Order proposals and discount proposals notified to the user

[1654] Step 10:

[1655] Input: Order proposals and discount proposals notified to the user

[1656] How it works: The server analyzes data on best-selling products and generates shelf layout optimization suggestions. It provides specific advice such as "Product A should be placed in a prominent position."

[1657] Output: Generated shelf optimization proposals

[1658] Step 11:

[1659] Input: Generated shelf optimization proposals

[1660] Operation: The server notifies the user of shelf optimization suggestions, such as "Product A should be placed in a prominent position."

[1661] Output: Shelf optimization suggestions communicated to the user

[1662] (Application example 1)

[1663] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1664] Modern retail businesses require efficient and accurate inventory management. However, traditional inventory management methods require a lot of manual checking and updating, which takes time and effort and is prone to human error. Additionally, product placement and display optimization often rely on experience and intuition, making management based on objective data difficult. To solve these problems, a system is needed that provides efficient and accurate inventory management and management advice based on objective data.

[1665] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1666] In this invention, the server includes an image capturing means, a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an interface means for acquiring image data and voice data using a smart device and confirming and correcting the recognition results. This significantly reduces the effort and errors involved in conventional inventory management, enables efficient and accurate inventory management, and further enables optimization of ordering and display based on objective data.

[1667] "Image capturing means" refers to a device or function for capturing images of the arrangement and storage conditions of items and creating image data thereof.

[1668] The "means for transmitting image data to a server" refers to a function or protocol for transmitting captured image data to a server via a network.

[1669] A "generative AI model" is an artificial intelligence model that learns from large amounts of data and automatically performs tasks such as image recognition and data analysis.

[1670] "Means for extracting product information" refers to analytical technology for extracting detailed product information (product number, quantity, etc.) from image data.

[1671] "Means for organizing and storing product information as digital data" refers to a system or method for organizing and storing extracted product information in a database or the like.

[1672] "Means for repeating product information aloud using voice output means" refers to a voice synthesis technology that outputs extracted and organized product information as voice and informs the user.

[1673] The "means for correcting erroneous determinations using a voice input means" refers to a voice recognition technology that accepts and analyzes a user's voice instructions and corrects erroneous data.

[1674] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to an algorithm or system that analyzes inventory data and proposes optimal order quantities and discount strategies.

[1675] The "means for notifying the user of the proposal results" refers to a messaging system or notification function for informing the user of the created order proposal or discount proposal.

[1676] "Interface means for acquiring image data and audio data using a smart device and confirming and correcting the recognition results" refers to a user interface for acquiring image and audio data using a device such as a smartphone or tablet and confirming and correcting the results.

[1677] MODE FOR CARRYING OUT THE INVENTION

[1678] The present invention is an efficient inventory management system using automated image recognition and a voice interface, and is configured as follows.

[1679] 1. Image capture method

[1680] Users use smart devices (smartphones or tablets) to take pictures of product shelves and storage areas, which generates image data that allows users to visually grasp the placement and quantity of items.

[1681] 2. Means of sending image data to the server

[1682] The terminal sends the captured image data to a server via the Internet, so the smart device must have network connectivity.

[1683] 3. A means of analyzing image data and extracting product information using generative AI models

[1684] The server inputs the received image data into a generative AI model for analysis. The generative AI model recognizes the product number and quantity contained in the image, and also analyzes the depth information of the image to recognize inventory that is hidden beneath the surface. This allows accurate product information to be extracted.

[1685] 4. A means of organizing and storing product information as digital data

[1686] The product information extracted by the server is organized as digital data by item, quantity, etc. The organized data is stored in a database and used for subsequent processing.

[1687] 5. A means for repeating product information aloud using a voice output means

[1688] The terminal has the ability to output the inventory information stored in the database aloud, for example, notifying the user in the form of "There are 10 units of product A and 5 units of product B."

[1689] 6. A method for correcting misjudgments using voice input

[1690] Users can use the voice input function to give instructions to the system, such as "I want 9 units of product A," which is then analyzed using voice recognition technology and the information in the database is updated.

[1691] 7. A means of analyzing inventory data and generating ordering and discounting proposals

[1692] The server analyzes the inventory data stored in the database to generate order requirements and discount suggestions, using algorithms to make suggestions based on sales trends and stock levels, and notifies the user.

[1693] 8. Means of notifying users of the proposed results

[1694] The generated ordering and discount proposals are notified to the user via the terminal. For example, the notification may say, "Product B is low in stock, so we recommend you order more. Also, product C is not selling well, so we suggest a 20% discount."

[1695] 9. Interface means for acquiring image data and voice data using a smart device, and for confirming and correcting the recognition results.

[1696] The system uses a smart device to acquire image and voice data and provides a user interface for confirming and correcting the recognition results. For example, a user can take a photo of a shelf with their smartphone and confirm the results by voice.

[1697] This significantly improves the efficiency of inventory management and reduces human error. Accurate product recognition using generative AI models, combined with the ease of a voice interface, improves the accuracy and efficiency of overall management.

[1698] Examples of specific examples and prompts

[1699] As a concrete example, a store clerk takes a photo of a shelf with their smartphone, and the app automatically reads the inventory information and notifies them via voice. The clerk can also give a voice instruction such as "There are 9 units of product A, not 10," and the app will correct the data and notify them again. Furthermore, the app will offer management advice such as "We recommend ordering product B. Also, please place product C in a prominent location."

[1700] Example prompt for a generative AI model:

[1701] "Inventor data: Product A: 10 pieces, Product B: 5 pieces, Product C: 2 pieces. Provide stock refill and display suggestions."

[1702] This prompt sentence generates appropriate management advice and display suggestions.

[1703] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1704] Step 1:

[1705] Users use their smart devices to take pictures of product shelves and storage areas, and then use the camera function to obtain image data that allows them to visually grasp the placement and quantity of items.

[1706] Input: Actual footage of shelves and storage areas

[1707] Output: Image data saved on the smart device

[1708] Step 2:

[1709] The device sends the acquired image data to a cloud server via the Internet, using the network connection function to send the image data to a specific server.

[1710] Input: Image data on a smart device

[1711] Output: Image data stored on a cloud server

[1712] Step 3:

[1713] The server inputs the received image data into the generative AI model for analysis. The generative AI model detects and recognizes the product numbers and quantities of the products contained in the image, and also analyzes the depth information of the image if necessary.

[1714] Input: Image data stored on a cloud server

[1715] Output: Specific product information such as product number and quantity

[1716] Step 4:

[1717] The server organizes the product information extracted by the generative AI model as digital data and stores it in a database. Through this process, information such as product numbers and quantities for each product is systematically organized.

[1718] Input: Product information extracted by the generative AI model

[1719] Output: Organized product information stored in a database

[1720] Step 5:

[1721] The terminal notifies the user of the inventory information stored in the database using a voice output function, for example, by repeating aloud, "There are 10 units of product A and 5 units of product B."

[1722] Input: Stock information stored in the database

[1723] Output: Voice output of stock information

[1724] Step 6:

[1725] The user can use the voice input function to give correction instructions to the system as needed, for example, "There are nine units of product A."

[1726] Input: User correction instructions via voice input

[1727] Output: Audio data received by the device

[1728] Step 7:

[1729] The server analyzes the received voice data and corrects any incorrect judgments. It uses voice recognition technology to analyze the instructions and corrects the relevant data in the database.

[1730] Input: Audio data received by the device

[1731] Output: Database with revised product data

[1732] Step 8:

[1733] The server analyzes the inventory data in the database and generates ordering and discount proposals, using algorithms to create optimal proposals based on sales trends, inventory levels, etc.

[1734] Input: Product information and inventory data stored in a database

[1735] Output: Generated order proposals and discount proposals

[1736] Step 9:

[1737] The generated recommendations are then sent to the user via their device. For example, they might say, "Product B is low in stock, so we recommend placing an additional order. Also, product C is not selling well, so we suggest a 20% discount."

[1738] Input: Generated order proposals and discount proposals

[1739] Output: The user is notified of the proposed results.

[1740] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1741] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[1742] 1. Image capture and data transmission

[1743] Subject: User and Device

[1744] A user starts the camera app on their device and takes an image of the display shelf. For example, they can take the image using a smartphone or a dedicated image capture device. The captured image is sent to the server by the device. The device then uploads the image data to the server via the Internet.

[1745] 2. Image Recognition and Product Information Extraction

[1746] Subject: Server and Generative AI Model

[1747] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This analysis includes determining the product's part number and quantity. Furthermore, the AI ​​can recognize inventory stacked deep inside, which cannot be seen on the surface, resulting in more accurate inventory information.

[1748] 3. Organize and store product information

[1749] Subject: Server

[1750] The server then compiles the product information extracted by the AI ​​model and organizes it as digital data for each product. This includes product numbers, quantities, and other necessary information. The organized data is stored in a database and serves as the basis for the next process.

[1751] 4. Voice confirmation and correction

[1752] Subject: Terminal and User

[1753] The device will repeat the saved inventory information to the user by voice. For example, the voice output will say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments by voice instruction. They can give instructions such as, "There are 4 units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[1754] 5. Correction and storage of data

[1755] Subject: Server

[1756] The server analyzes the user's voice commands and corrects the data. The database is updated based on the voice commands to maintain accurate inventory information. Voice-to-text conversion is performed within the device, and the converted text data is sent to the server.

[1757] 6. Providing management advice

[1758] Subject: Server

[1759] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine can adjust the suggestions based on the user's emotional state.

[1760] 7. Notification of Proposal

[1761] Subject: Server and terminal

[1762] The server notifies the user of the generated suggestions. For example, it may send a message such as "Product A is low in stock, so we recommend you place an additional order" or "Product B is not selling well, so we suggest a 20% discount." Based on the emotion engine, the notification method and content can be adjusted according to the user's emotional state.

[1763] 8. Providing display proposals

[1764] Subject: Server

[1765] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. These include specific advice such as "Product A should be placed in a prominent position." The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[1766] The above is a specific embodiment of the present invention. This system uses image recognition and voice recognition technologies to enable efficient and accurate inventory management, as well as management advice that takes into account the user's emotions. This optimizes the inventory management process and enables more flexible and effective management decisions.

[1767] The processing flow will be explained below.

[1768] Step 1:

[1769] Subject: User

[1770] The user launches the camera app on their device and takes a picture of the shelf or product display area. The user then uses a smartphone or dedicated device to capture the required area with the camera. Once the capture is complete, the image is saved.

[1771] Step 2:

[1772] Subject: Terminal

[1773] The device sends the captured image to a server, where the image data is uploaded to the server via the Internet and transferred to the server's data processing unit.

[1774] Step 3:

[1775] Subject: Server

[1776] The server passes the received image data to the generative AI model, which is used to analyze the product information in the image. The server provides the image as input to the AI ​​model.

[1777] Step 4:

[1778] Subject: Generative AI model

[1779] A generative AI model analyzes image data to extract the part number and quantity of each product. The model identifies products based on their visual characteristics and recognizes overlapping products and hidden inventory.

[1780] Step 5:

[1781] Subject: Server

[1782] The server receives product information from the generative AI model and organizes it as digital data, which is then stored in a database.

[1783] Step 6:

[1784] Subject: Terminal

[1785] The terminal repeats the inventory information received from the server by voice, using the voice output function to notify the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1786] Step 7:

[1787] Subject: User

[1788] The user confirms the result by voice and issues a correction instruction if there is an incorrect judgment. For example, the user can send a correction instruction by voice inputting "There are four units of product A." At this time, the emotion engine analyzes the voice data and understands the user's emotional state.

[1789] Step 8:

[1790] Subject: Terminal

[1791] The terminal converts the user's voice instructions into text and sends the correction instructions to the server. The voice-to-text conversion process is performed within the terminal, and the converted text data is sent to the server.

[1792] Step 9:

[1793] Subject: Server

[1794] The server modifies the inventory information in the database based on the user's voice instructions, and the modified digital data is stored in the database.

[1795] Step 10:

[1796] Subject: Server

[1797] The server analyzes inventory data and generates ordering and discount suggestions. Data analysis algorithms are used to make appropriate suggestions based on the inventory status. An emotion engine adjusts the suggestions based on the user's emotional state.

[1798] Step 11:

[1799] Subject: Server

[1800] The server notifies the user of the recommendation results. For example, it may send a message such as, "Product A is low in stock, so we recommend you place an additional order," or, "Product B is not selling well, so we suggest a 20% discount." The server adjusts the notification method and content according to the user's emotional state based on the emotion engine.

[1801] Step 12:

[1802] Subject: Server and terminal

[1803] The server analyzes data on best-selling products and generates optimization suggestions for display and shelf layout. The suggestions are notified to the user, and specific advice such as "It would be good to place product A in a prominent position" is provided. The emotion engine makes it possible to make suggestions that take into account the user's emotional state.

[1804] Step 13:

[1805] Subject: User

[1806] The user rearranges or orders products based on the suggestions received from the server. For example, the user may "order additional product A" or "apply a discount on product B."

[1807] This concludes the process flow of an inventory management system that combines image and voice recognition technologies with user emotion recognition. This system enables efficient and accurate inventory management and provides management advice that takes into account the user's emotional state.

[1808] Example 2

[1809] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1810] Conventional inventory management systems make it difficult to grasp accurate inventory information, often requiring manual confirmation. Furthermore, they do not provide management advice that takes into account the user's emotional state, limiting the effectiveness of management decisions. Furthermore, there is a lack of means to accurately identify hidden inventory and product part numbers and quantities. There is a need for a system that can solve these issues and provide efficient and accurate inventory management.

[1811] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1812] In this invention, the server includes an image capturing means, a means for transmitting image data to an information processing device, a means for analyzing the image data using a generative AI model and extracting product information, a means for organizing and saving the product information as digital data, a means for audibly repeating the product information using a voice output means, a means for correcting erroneous judgments using a voice input means, a means for analyzing inventory data and generating order proposals and discount proposals, a means for notifying the user of the proposal results, and an emotion engine for analyzing the user's emotional state and adjusting the proposal content. This significantly improves the accuracy and efficiency of inventory management and makes it possible to provide management advice tailored to the user's emotional state.

[1813] "Image capturing means" refers to a device that allows a user to capture an image of a product display shelf, and includes a smartphone or a dedicated image capture device.

[1814] The "means for transmitting image data to an information processing device" is a process for transmitting captured image data to a server, and includes a function for uploading data via the Internet.

[1815] A "generative AI model" is an artificial intelligence model that analyzes image data, recognizes objects in the image, and extracts corresponding information.

[1816] "Means for extracting product information" includes a function to identify product part numbers and quantities in an image using a generative AI model.

[1817] "Means for organizing and storing product information as digital data" refers to the process of organizing the extracted product information into a specific format and storing it in a database.

[1818] "Audio output means" refers to a function that enables the terminal to convey stored product information to the user by voice, and includes a voice assistant and a speaker.

[1819] "Voice input means" refers to a function for receiving a user's voice instructions and analyzing the voice data, and includes a microphone and voice recognition software.

[1820] An "emotion engine" is a system that includes algorithms for analyzing voice data and understanding the user's emotional state.

[1821] The "means for analyzing inventory data and generating order proposals and discount proposals" is a process for automatically generating appropriate order and discount proposals based on inventory information.

[1822] "Means for notifying users of proposal results" refers to a function for informing users of the generated management advice and proposal content, and includes push notifications and message sending.

[1823] This invention provides a system that combines efficient inventory management and user sentiment analysis in the retail industry. The system includes image capture, data transfer, recognition by a generative AI model, voice confirmation and correction, provision of management advice, and user sentiment recognition by an emotion engine.

[1824] In implementing this system, the following hardware and software is recommended:

[1825] 1. Hardware:

[1826] Users use smartphones or dedicated image capture devices to take images.

[1827] The terminals include computers and smartphones that can connect to the Internet.

[1828] 2. Software:

[1829] Generative AI models: For example, object detection algorithms such as "YOLO" and "Detectron2" are used.

[1830] Database: We use database management systems such as "MySQL" or "PostgreSQL" to store inventory information.

[1831] Speech recognition: Services such as "Google Cloud Speech-to-Text API" and "Amazon Alexa" are used to analyze voice data.

[1832] Emotion Engine: Implements emotion recognition algorithms to analyze the user's emotional state.

[1833] System processing overview

[1834] 1. Image capture and data transmission

[1835] Users use their smartphones to take pictures of the shelves, which are then compressed and uploaded to a server via the Internet.

[1836] 2. Image Recognition and Product Information Extraction

[1837] The server sends the received image data to the generative AI model, which analyzes the products in the image and identifies the product number and quantity. The detection results are returned to the server in JSON format.

[1838] 3. Organize and store product information

[1839] The server analyzes the data from the generative AI model and organizes the product number and quantity information for each product. The organized data is stored in a database and serves as the basis for the next process.

[1840] 4. Voice confirmation and correction

[1841] The device outputs inventory information stored in the database and repeats it back to the user. The user can use voice input to correct any incorrect judgments. The emotion engine analyzes the voice data and understands the user's emotional state.

[1842] 5. Correction and storage of data

[1843] The server analyzes the user's voice commands and updates the database with the corrected data. The voice input is converted into text on the device and sent to the server.

[1844] 6. Providing management advice

[1845] The server analyzes inventory data and generates ordering and discount offers, with an emotional engine adapting the offers based on the user's emotional state.

[1846] 7. Notification of Proposal

[1847] The server notifies the user of the generated suggestions using push notifications or messaging.

[1848] 8. Providing display proposals

[1849] The server analyzes best-selling products and generates optimization proposals for display and shelf layout. The emotion engine makes proposals that take into account the user's emotional state.

[1850] Specific examples

[1851] 1. A user takes a picture of a store's display shelves with their smartphone and sends it to the server.

[1852] 2. The server passes the image data to a generative AI model (e.g., YOLO) and analyzes the product number and quantity.

[1853] 3. The analysis results are saved in the database in the format of "5 units of product A, 3 units of product B, 8 units of product C."

[1854] 4. The device uses Google Assistant to tell the user, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1855] 5. The user inputs "Product A is 4 units" by voice and gives a correction instruction. The emotion engine analyzes the user's emotional state.

[1856] 6. The corrected data is updated in the database.

[1857] 7. The server uses TensorFlow to analyze the inventory data and notifies the customer that "Product A is low in stock, so we recommend placing an additional order."

[1858] 8. Generate display optimization suggestions for best-selling products, such as "Product A should be placed in a prominent position."

[1859] In this way, the present invention realizes a system in which the user, terminal, and server cooperate to provide efficient and accurate inventory management and management advice that takes into account the user's feelings.

[1860] Prompt Sentence Examples

[1861] "Upload image data of the display shelves and use the generative AI model to analyze the product part numbers and quantities. Then, read back the analysis results and correct any misidentifications with voice instructions. Finally, provide management advice based on the inventory status and generate display suggestions."

[1862] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1863] Step 1:

[1864] Image capture and data transmission

[1865] input:

[1866] A user picks up a smartphone, launches the camera app, and takes a picture of the display shelf.

[1867] Specific behavior:

[1868] The user taps the camera app on the smartphone's operation screen to launch it.

[1869] While looking at the camera preview screen, take a photo of the display shelf at the appropriate angle and distance.

[1870] The captured image file is saved in the device's local storage.

[1871] Data processing and output:

[1872] The image data captured by the device is compressed and saved in a file format such as JPEG.

[1873] The compressed image file is uploaded to a server via the Internet.

[1874] Step 2:

[1875] Image recognition and product information extraction

[1876] input:

[1877] The server receives the image data sent from the terminal.

[1878] Specific behavior:

[1879] The server receives the image data through an HTTP request.

[1880] The received image data is saved in a specific directory.

[1881] Data processing and output:

[1882] The server inputs the stored image data into a generative AI model (e.g., YOLO).

[1883] The generative AI model analyzes the products in the image, identifies the product number and quantity in JSON format, and returns it to the server.

[1884] Step 3:

[1885] Organizing and storing product information

[1886] input:

[1887] The server receives the JSON-formatted product information data returned from the generated AI model.

[1888] Specific behavior:

[1889] The server reads the JSON data and extracts the product number and quantity information.

[1890] The extracted information is then organized and presented in a digital format.

[1891] Data processing and output:

[1892] Save data for each product (e.g., "Product A: 5 units, Product B: 3 units") in the database.

[1893] Save the information in a database management system (e.g. MySQL) using the INSERT statement.

[1894] Step 4:

[1895] Voice confirmation and correction

[1896] input:

[1897] The server transmits the organized inventory information to the terminal, and the terminal starts outputting audio.

[1898] Specific behavior:

[1899] The device converts the inventory information from text to speech, and the voice assistant (e.g., Google Assistant) repeats, "There are 5 units of product A, 3 units of product B, and 8 units of product C."

[1900] The user can use the voice input function to give instructions to correct any incorrect judgments (e.g., "There are four units of product A").

[1901] Data processing and output:

[1902] The device converts the voice input into text and sends the text data to the server.

[1903] The emotion engine analyzes the voice data and understands the user's emotional state.

[1904] Step 5:

[1905] Data correction and storage

[1906] input:

[1907] The server receives the user's voice input data and begins analyzing it.

[1908] Specific behavior:

[1909] The server analyzes the received text data and identifies the incorrectly determined inventory information.

[1910] Generate the corresponding SQL statements to correct the misjudged inventory information.

[1911] Data processing and output:

[1912] Use the UPDATE statement to update the corrected inventory information in the database (e.g., "UPDATE inventory information table SET quantity=4 WHERE product ID='product A'").

[1913] The corrected accurate inventory information is saved in the database.

[1914] Step 6:

[1915] Providing management advice

[1916] input:

[1917] The server generates ordering suggestions and discount suggestions based on an up-to-date inventory database.

[1918] Specific behavior:

[1919] The server analyzes the inventory data and generates ordering and discount suggestions using data analysis algorithms (e.g., TensorFlow).

[1920] The emotion engine adjusts suggestions based on the user's emotional state.

[1921] Data processing and output:

[1922] Based on the detected trends, suggestions such as "Product A is low in stock, so we recommend placing an additional order" are generated.

[1923] The server sends the generated proposal to the terminal.

[1924] Step 7:

[1925] Proposal Notification

[1926] input:

[1927] The management advice and proposals generated by the server are sent to the terminal.

[1928] Specific behavior:

[1929] The server creates a notification message and sends a push notification to the user's smartphone using Firebase Cloud Messaging or similar.

[1930] The device receives the notification and displays it to the user.

[1931] Data processing and output:

[1932] The proposal is displayed to the user and management advice is provided (e.g., "Product A is low in stock, so we recommend placing an additional order").

[1933] Step 8:

[1934] Providing display proposals

[1935] input:

[1936] The server generates display suggestions based on data on best-selling items.

[1937] Specific behavior:

[1938] The server analyzes top-selling products and generates display optimization suggestions.

[1939] The emotion engine takes into account the user's emotional state and adjusts the suggestions accordingly.

[1940] Data processing and output:

[1941] The proposal is generated in the form of "Product A should be placed in a prominent position."

[1942] The proposal is sent to the terminal and displayed to the user.

[1943] In this way, the system of the present invention combines image recognition technology and voice recognition technology to provide efficient and accurate inventory management and management advice that takes into account the user's emotions.

[1944] (Application example 2)

[1945] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1946] Conventional inventory management systems have the problem that it is difficult to accurately grasp and manage product information, and they lack management advice that takes into account users' emotions and purchasing intentions, making it difficult to fully optimize inventory and promote sales.

[1947] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for transmitting image data to the server, a means for analyzing the image data using a generative AI model and extracting product information, a means for capturing and analyzing the user's emotional state using a smart device, and a means for adjusting the content of suggestions and notification methods based on the user's emotional state. This enables accurate understanding of inventory information and flexible management advice based on the user's emotions.

[1948] "Image capturing means" refers to a device or system for capturing images of shelves and inventory.

[1949] "Means for transmitting image data to a server" refers to a function or device for transmitting captured images to a cloud or remote server via a network.

[1950] A "generative AI model" is a machine learning model that uses artificial intelligence technology to analyze image data and extract specific patterns and features.

[1951] "Means for organizing and storing product information as digital data" refers to a device or system for structuring extracted product information and storing it in a database.

[1952] The "audio output means" is a device or system for outputting the stored product information as audio.

[1953] The "voice input means" is a device or system for recognizing voice instructions from the user and converting the contents of the voice instructions into text data.

[1954] The "means for analyzing inventory data and generating order proposals and discount proposals" refers to a device or system for analyzing the current inventory status and generating proposals for the need for additional orders and discounts.

[1955] The "means for notifying the user of the proposal results" is a device or system for notifying the user of the generated order proposal or discount proposal.

[1956] "Means for capturing and analyzing a user's emotional state using a smart device" refers to a device or system that uses smart glasses or other wearable devices to obtain and analyze a user's emotional state from voice and facial expressions.

[1957] "Means for adjusting the content of proposals and notification methods based on the user's emotional state" refers to a device or system for appropriately adjusting the content of order proposals and discount proposals and notification methods based on the analyzed emotional state of the user.

[1958] The present invention is a smart device-based system that combines efficient inventory management in brick-and-mortar stores with user emotion analysis. The system includes image capture, data transfer, recognition using a generative AI model, voice confirmation and correction, provision of management advice, and user emotion recognition using an emotion engine. A specific embodiment of the system is described in detail below.

[1959] Hardware and software used

[1960] Hardware: Smart glasses (with camera, microphone, and display)

[1961] Software: generative AI models, emotion engines, speech recognition software (e.g., Google Cloud Speech-to-Text)

[1962] Image capture and data transmission

[1963] A user wears smart glasses and walks around the store scanning the inventory on the shelves. The smart glasses' camera captures images and sends them over the internet to a cloud server. The image data obtained through this process is then sent to the server via a high-speed data transmission channel.

[1964] Image recognition and product information extraction

[1965] The server passes the received image data to the generative AI model, which then analyzes the products in the image. This makes it possible to determine the product's part number and quantity, as well as recognize inventory stacked deep within the image. This image recognition technology enables more accurate inventory information to be obtained.

[1966] Organizing and storing product information

[1967] The server then processes the product information extracted by the generative AI model and organizes it as digital data for each product. The information includes product numbers, quantities, and other necessary data. This information is then stored in a database and becomes the basis for the next process.

[1968] Voice confirmation and correction

[1969] The smart glasses' voice output function is used to repeat the organized inventory information to the user. For example, the voice output may say, "There are 5 units of product A, 3 units of product B, and 8 units of product C." The user can use the voice input function to correct any incorrect judgments with voice instructions. During this process, the emotion engine analyzes the voice data and understands the user's emotional state.

[1970] Providing management advice

[1971] The server analyzes inventory data and generates ordering and discount suggestions. It uses analytical algorithms to make appropriate suggestions based on stock availability. The emotion engine can also adjust the suggestions based on the user's emotional state. For example, it can make specific ordering suggestions such as "Product A is low in stock, so we recommend you order more," or discount suggestions such as "Product B is not selling well, so we suggest a 20% discount."

[1972] Suggestions and notifications based on emotional state

[1973] The server notifies the user of the generated order proposals and discount proposals. Based on the emotion engine, the server adjusts the notification method and content according to the user's emotional state. This allows the server to provide information in a form that is psychologically easy for the user to accept.

[1974] Examples of specific examples and prompts

[1975] Examples:

[1976] A user wears smart glasses and takes pictures of the shelves while walking around the store.

[1977] Product information is sent to the server, and the generative AI model analyzes the inventory information.

[1978] When the user requests a correction by voice input, the user may give a voice instruction such as "There are four units of product A."

[1979] Example prompt sentence:

[1980] "Use the camera in your smart glasses to take pictures of your shelves and send the inventory information to the cloud."

[1981] "Capture your customers' facial expressions, analyze them with an emotion engine, and offer sales strategy advice."

[1982] "Propose additional orders or changes to shelf locations based on recognized inventory information."

[1983] This makes it possible to implement a system according to a specific embodiment of the present invention.

[1984] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1985] Step 1:

[1986] A user wears smart glasses and activates the smart glasses' camera to scan the inventory on the shelves while walking around the store. The smart glasses then take an image of the entire display shelf and capture the image data. The input here is the image captured by the camera, and the output is the image data stored in the smart glasses.

[1987] Step 2:

[1988] Image data captured by the device (smart glasses) is sent to a cloud server via the Internet. HTTP or HTTPS is used as the transmission protocol. The input is image data, and the output is image data stored on the cloud server.

[1989] Step 3:

[1990] The server passes the received image data to the generative AI model, which then performs image analysis. The generative AI model extracts the product's part number and quantity from the image data, and recognizes the inventory stacked in the back as well. The input is the transmitted image data, and the output is the extracted product information (part number, quantity).

[1991] Step 4:

[1992] The server organizes the product information extracted by the generative AI model, organizes it as digital data for each product, and stores it in a database. The input is the extracted product information, and the output is organized digital data stored in the database.

[1993] Step 5:

[1994] The server converts the organized inventory information into voice data and repeats it back to the user using the device's (smart glasses') voice output function. This uses a voice synthesis service such as Google Cloud Text-to-Speech. The input is organized digital data, and the output is voice data.

[1995] Step 6:

[1996] When a user wants to correct a false positive using the voice input function, they input the correction instructions through the microphone of the smart glasses. The input is the user's voice instruction, which is converted into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The output is text data.

[1997] Step 7:

[1998] The server analyzes the text data acquired through voice input and compares it with existing database information to correct misjudgments. The input is text data of the user's voice instructions, and the output is corrected inventory information.

[1999] Step 8:

[2000] The server analyzes the inventory data and generates ordering and discount proposals using an AI model, taking into account the user's emotional data using an emotion engine. The input is the revised inventory information and the user's emotional information, and the output is the generated ordering and discount proposals.

[2001] Step 9:

[2002] The server notifies the user of the generated order proposals and discount proposals. When notifying, the method and content of the notification are adjusted based on the user's emotional state using an emotion engine. The input is the proposal data, and the output is a notification to the user.

[2003] The above is a detailed description of the processing steps of the system for specifically implementing the present invention.

[2004] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2005] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2006] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2007] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2008] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2009] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2010] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2011] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2012] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2013] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2014] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2015] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2016] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2017] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2018] It is not necessary to store all of the specific processing program 56 in a storage device such as...

Claims

1. Image capturing means; means for transmitting image data to a server; A means for analyzing image data using a generative AI model and extracting product information; A means of organizing and storing product information as digital data, a means for repeating the product information by voice using a voice output means; A means for correcting an erroneous determination using a voice input means; means for analyzing inventory data and generating ordering suggestions and discount suggestions; a means for notifying the user of the proposal result; A system including:

2. The system of claim 1 , wherein the generative AI model further comprises means for analyzing depth information of the image and recognizing understacked inventory.

3. The system of claim 1 , further comprising means for recognizing the product number and quantity when the generative AI model extracts product information.

Images