System

Abstract

A system is provided.SOLUTION: A system comprising: photographing means for a user to photograph a receipt; transmitting means for transmitting the photographed receipt image to a server; analyzing means for converting the received receipt image into text data using an optical character recognition technique; storing means for extracting product information from the analyzed text data and storing the product information in association with an account of the user; proposal generating means for managing and proposing various products owned by the user based on the stored product information; notifying means for notifying the user of the generated proposal; and action processing means for receiving an action of the user and updating data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, many people find it difficult to efficiently manage the goods they have purchased and the items they own. In particular, there is no system that can centrally manage a wide range of goods, such as food, clothing, household goods, hobby goods, and financial products, and provide appropriate recommendations and support for actions. As a result, people end up overpurchasing and discarding goods, wasting time by having to compare prices, and using multiple platforms. Furthermore, young people in particular suffer from a chronic sense of financial shortage and problems with the disposal of goods. There is a need for a system that can solve these issues, streamline daily life, and optimize consumer behavior. [Means for solving the problem]

[0005] This system includes a camera for users to take a photo of a receipt, a transmitter for transmitting the captured receipt image to a server, an analyzer for converting the received receipt image into text data using optical character recognition technology, a storage unit for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generator for managing and proposing various products owned by the user based on the stored product information, a notification unit for notifying the user of the generated proposals, and an action processor for receiving user actions and updating the data. This system allows users to easily manage their purchases and belongings and optimize their consumption behavior by receiving efficient proposals. Specifically, the system provides a wide range of lifestyle support, including recipe suggestions based on ingredients owned, purchasing suggestions for ingredients in short supply, managing owned clothing and suggesting flea market prices, comparing prices for detergents and beauty products, managing hobby and luxury items and linking them to flea market sites, and advice on buying and selling financial products.

[0006] "Capture means" refers to a device or function that a user uses to capture an image of a receipt or other data.

[0007] The "transmission means" refers to a protocol or function for transmitting image data acquired by the image capture means to a server.

[0008] "Analysis means" refers to a system or software that uses the transmitted image data and applies optical character recognition techniques to extract text information from the image.

[0009] The "storage means" is a function that associates product information extracted from the analyzed text data with the user's account and stores it in a database.

[0010] The "proposal generation means" is an algorithm or engine that generates appropriate proposals based on the stored product information and in accordance with the user's possessions and purchase history.

[0011] The "notification means" is a mechanism for instantly notifying the user of generated suggestions and actions.

[0012] The "action processing means" is a function that supports the user in taking action based on the suggestions and updates data in accordance with the action.

[0013] A "receipt image" is image data of a piece of paper on which information about purchased items is written, captured by a user using a photographing device.

[0014] "Optical character recognition technology" is a technology that identifies characters in an image and extracts them as text data.

[0015] "Product information" is data related to purchased items, and includes information such as product name, price, quantity, and purchase date.

[0016] "Suggestions" are support information such as recipes, price comparisons, flea market listings, and consumer behavior advice provided to the user via the storage means and suggestion generation means.

[0017] A "notification" is a message or alert that conveys information about a suggestion or action to the user. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. The system includes a terminal on which a dedicated app is installed and a server connected via a network.

[0040] System Configuration

[0041] The system includes the following main components:

[0042] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[0043] 2. Transmission method: The captured receipt image is sent to the server.

[0044] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[0045] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[0046] 5. Proposal generation means: Based on the stored product information, manages the user's possessions and generates various proposals.

[0047] 6. Notification means: The generated proposal is notified to the user's terminal.

[0048] 7. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[0049] Program processing and specific examples

[0050] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[0051] Capture and send receipt

[0052] User:

[0053] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[0054] Device:

[0055] The device temporarily saves the captured receipt image. The user checks the image and, if there are no problems, presses the "Start Analysis" button. The device then sends the receipt image to the server.

[0056] Receipt image analysis and product data extraction

[0057] server:

[0058] The server passes the received receipt image to the OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[0059] Proposal generation and distribution

[0060] server:

[0061] The system updates the management data for various products owned by the user based on the saved data. For example, for food, it generates recipe suggestions based on the ingredients owned and suggests purchasing the necessary ingredients. For clothing, it suggests flea market prices and new fashions. For household goods, it compares prices of detergents and beauty products and suggests the lowest prices.

[0062] Device:

[0063] Once a proposal is generated, the server notifies the user's device, which then displays the proposal to the user for further confirmation.

[0064] Executing a user action

[0065] User:

[0066] The user can check the suggestions and take action such as creating a recipe, purchasing missing ingredients, listing clothing, purchasing daily necessities, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy the basil they are running low on.

[0067] Device:

[0068] Depending on the action selected by the user, the device will display an appropriate screen, such as supporting the process of selling clothes at a flea market, or providing links to purchase ingredients that are in short supply.

[0069] Save and notify results

[0070] server:

[0071] Once the user completes their action, the server updates the data based on that behavior and records it to improve the accuracy of future suggestions.

[0072] Device:

[0073] The user is notified that the transaction is complete. For example, a notification saying "Basil purchase complete" is displayed on the device.

[0074] Specific examples

[0075] Example 1: Food suggestion implementation

[0076] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[0077] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0078] 3. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[0079] 4. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[0080] 5. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[0081] In this way, a system is realized that allows users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[0082] The processing flow will be explained below.

[0083] Step 1:

[0084] User:

[0085] The user launches the app, takes a photo of the receipt using the camera, presses the capture button, and adjusts the image so that the entire receipt is visible.

[0086] Step 2:

[0087] Device:

[0088] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[0089] Step 3:

[0090] server:

[0091] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[0092] Step 4:

[0093] server:

[0094] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[0095] Step 5:

[0096] server:

[0097] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[0098] Step 6:

[0099] server:

[0100] The system updates inventory management data based on the saved product information. For example, for food, it generates recipe suggestions based on the ingredients you have. It also creates a list of ingredients you need or are lacking and suggests purchases.

[0101] Step 7:

[0102] server:

[0103] Based on the user's profile and past data, the system generates flea market price suggestions and new product recommendations. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[0104] Step 8:

[0105] server:

[0106] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products.

[0107] Step 9:

[0108] Device:

[0109] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[0110] Step 10:

[0111] User:

[0112] After checking the suggested content, the user selects an action such as creating a recipe, purchasing ingredients, listing clothing, purchasing a product, etc. For example, if the user accepts the recipe suggestion, they select "View Recipe."

[0113] Step 11:

[0114] Device:

[0115] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[0116] Step 12:

[0117] server:

[0118] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[0119] Step 13:

[0120] Device:

[0121] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[0122] Example 1

[0123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] In conventional item management systems, it was difficult to efficiently manage information about items owned by users. Furthermore, the technology for making optimal suggestions to users was insufficient, so users had to search for and select information themselves, which resulted in issues such as reduced user convenience and complicated item management.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0126] In this invention, the server includes a photographing means for a user to photograph a certificate, a transmitting means for transmitting the photographed certificate image to an information processing device, an analyzing means for converting the received certificate image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing the information in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, a notifying means for notifying the user of the generated proposals, and an action processing means for executing an action selected by the user based on the proposal received and updating the data. This allows the user to automatically manage product information and receive optimal proposals simply by photographing the certificate.

[0127] "Photographing means" refers to a camera function that allows a user to photograph a document (such as a receipt) and an application function that supports such photographing.

[0128] "Transmission means" refers to a communication function or protocol for transmitting the photographed certificate image to an information processing device (such as a server).

[0129] "Analysis means" refers to software or algorithms for converting received certificate images into text data using optical character recognition (OCR) technology.

[0130] "Storage means" refers to a database or storage function for storing product information extracted from the analyzed text data in association with the user's account.

[0131] "Proposal generation means" refers to an algorithm or generative AI model for managing various products owned by a user and generating optimal proposals based on stored product information.

[0132] "Notification means" refers to a notification function (e.g., push notification or email notification) or its implementation for notifying the user of the generated proposal.

[0133] "Action processing means" refers to a processing function for executing an action selected by the user based on the proposal received and for reflecting the result in the database.

[0134] "Voucher" refers to a document issued as a result of a transaction (e.g., receipt, invoice).

[0135] "Information processing device" refers to a computer system or server for receiving, analyzing, storing data, and generating suggestions.

[0136] "Optical character recognition technology (OCR)" refers to technology for extracting character information from an image and converting it into text data.

[0137] "Generative AI model" refers to an artificial intelligence model (e.g., GPT-3, GPT-4) that generates optimal suggestions based on user data.

[0138] MODE FOR CARRYING OUT THE INVENTION

[0139] System Configuration

[0140] This invention is a system that allows users to photograph documents (receipts, etc.) and efficiently manage and display the information. The system includes a terminal with a dedicated app installed and an information processing device (server) connected via a network.

[0141] Hardware and Software

[0142] Device:

[0143] The device is a mobile device such as a smartphone or tablet that has a camera and internet connection, and a dedicated app installed.

[0144] server:

[0145] A server is a computer system equipped with a high-performance processor and a large amount of memory. The following software is required for a server:

[0146] OCR engine: Optical character recognition software such as ABBYY FineReader or Tesseract.

[0147] Database: A relational database management system such as MySQL or PostgreSQL.

[0148] Generative AI models: Use artificial intelligence models such as GPT-3 and GPT-4.

[0149] Explaining program processing in natural language

[0150] The user takes a photo of the certificate for an item purchased in daily life using a dedicated app. During this process, the image of the certificate taken by the user is temporarily saved in the device.

[0151] Transmission method:

[0152] When the user presses the "Start Analysis" button, the device sends the certificate image to the server via an internet connection.

[0153] Analysis method:

[0154] The server passes the received certificate image to an OCR engine, extracts the text information from the image, and converts the image into text data using ABBYY FineReader or Tesseract.

[0155] Storage method:

[0156] The server extracts product names, prices, quantities, purchase dates, etc. from the analyzed text data and stores them in a database. At this time, product names are standardized to accommodate differences in notation at different stores.

[0157] Suggestion generation means:

[0158] The server updates the user's inventory based on the stored product data. For food, it generates recipe suggestions based on the ingredients the user has in their possession. It uses a generative AI model to suggest optimal recipes based on the user's preferences and past selection history.

[0159] For example, if a user purchases tomatoes, pasta, and cheese at a supermarket, the server will suggest a recipe for "tomato pasta" based on this information and suggest purchasing any basil that is in short supply.

[0160] Notification method:

[0161] Once a suggestion is generated, the server notifies the device of its contents, and upon receiving the notification, the device displays the suggestion to the user.

[0162] Action processing method:

[0163] After the user checks the suggestions and selects the necessary action, the device will provide a screen and link for carrying out the action, such as providing a link to purchase missing ingredients so the user can purchase them from an online shop.

[0164] Save the results:

[0165] The server records the results of the user's actions in a database and improves the accuracy of suggestions from the next time onwards, thereby making the user's life more efficient.

[0166] Specific examples

[0167] Example 1: Food suggestion implementation

[0168] A user takes a photo of a receipt from a supermarket purchase, and the device sends the image to the server.

[0169] The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0170] The server uses a generative AI model to suggest a recipe for "tomato pasta" and suggests purchasing the missing basil.

[0171] The terminal notifies the user of the proposed content, and the user decides to purchase the basil.

[0172] The device assists with the purchase process, and the server records the updated data, allowing users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[0173] Prompt Sentence Examples

[0174] A specific example of a prompt sentence that a user might enter into a generative AI model is as follows:

[0175] Example prompt: "I took a photo of the receipt for the groceries I bought at the supermarket. Can you suggest a meal based on the ingredients I have on hand?"

[0176] By entering such prompts, the generative AI model will suggest appropriate recipes and notify you of any missing ingredients.

[0177] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0178] Step 1: User takes a photo of the certificate

[0179] User:

[0180] The user launches the dedicated app and takes a photo of the receipt using the camera on their smartphone or tablet. Specifically, they make sure the receipt fits on the screen and press the capture button.

[0181] input:

[0182] Physical voucher.

[0183] output:

[0184] A voucher image in digital image format.

[0185] Specific behavior:

[0186] The smartphone camera takes a photo of the certificate and saves the image in a dedicated app.

[0187] Step 2: Send the certificate image to the server

[0188] Device:

[0189] The terminal temporarily stores the captured certificate image. When the user checks the image and presses the "Start analysis" button, the terminal sends the certificate image to the server.

[0190] input:

[0191] A voucher image in digital image format.

[0192] output:

[0193] The voucher image sent to the server.

[0194] Specific behavior:

[0195] Using the sending function within the dedicated app, the certificate image is uploaded to the server via the Internet.

[0196] Step 3: Analysis of the voucher image

[0197] server:

[0198] The server inputs the received certificate image into an OCR engine and extracts text information from the image.

[0199] input:

[0200] The voucher image sent to the server.

[0201] output:

[0202] Extracted text data (product name, price, quantity, purchase date, etc.).

[0203] Specific behavior:

[0204] An OCR engine (such as ABBYY FineReader or Tesseract) is used to convert the text information in the certificate image into text data.

[0205] Step 4: Store and standardize text data

[0206] server:

[0207] The server extracts product information from the extracted text data, standardizes the product name, associates it with the user's account, and stores it in a database.

[0208] input:

[0209] The extracted text data.

[0210] output:

[0211] Standardized product information stored in a database.

[0212] Specific behavior:

[0213] Product information (product name, price, quantity, purchase date, etc.) is inserted into the database using an SQL query, and the product is recognized as the same product according to product name standardization rules.

[0214] Step 5: Generate proposals

[0215] server:

[0216] The server uses the stored product information to update the user's inventory using a generative AI model and generate optimal suggestions, such as recipe suggestions based on the ingredients the user owns.

[0217] input:

[0218] Standardized product information stored in a database.

[0219] output:

[0220] Suggestions for users.

[0221] Specific behavior:

[0222] Using generative AI models (such as GPT-3 and GPT-4), it analyzes the user's inventory and generates optimal suggestions based on the user's preferences and past selection history.

[0223] Step 6: Proposal Notification

[0224] server:

[0225] The generated proposal is sent to the device for notification.

[0226] input:

[0227] The generated proposal.

[0228] output:

[0229] A notification message that is displayed on the user's terminal.

[0230] Specific behavior:

[0231] The generated suggestions are sent to the user's device using push notifications or email notifications.

[0232] Step 7: User Actions

[0233] User:

[0234] The user reviews the suggestions and selects and executes the necessary action, for example, purchasing ingredients for the suggested recipe.

[0235] input:

[0236] User selection based on suggestions.

[0237] output:

[0238] The result of the action taken (e.g., checkout).

[0239] Specific behavior:

[0240] From within the dedicated app, users can access links and follow procedures to purchase the necessary ingredients from online shops.

[0241] Step 8: Save and notify results

[0242] server:

[0243] The results of the user's actions are recorded in a database to improve the accuracy of the next suggestion, and the user is notified when the action is complete.

[0244] input:

[0245] The result of the action taken.

[0246] output:

[0247] Action results and user notification messages stored in the database.

[0248] Specific behavior:

[0249] Update the database using SQL queries to save the action results, and use the notification function to send an action completion message to the user's device.

[0250] (Application example 1)

[0251] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0252] In modern society, it remains difficult for users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations based on that information. In particular, efficiently managing ingredients and purchasing the ingredients they need requires a lot of effort, placing a heavy burden on users. For this reason, there is a need for a system that allows users to efficiently manage their owned items, receive optimal recommendations, optimize their lives, and quickly order ingredients they need through food delivery services.

[0253] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0254] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmission means for transmitting the photographed receipt image to the server, an analysis means for converting the received receipt image into text data using optical character recognition technology, a storage means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generation means for managing and proposing various products owned by the user based on the stored product information, a notification means for notifying the user of the generated proposals, an action processing means for receiving user actions and updating data, and an order processing means for allowing the user to order missing ingredients via a food delivery service. This enables efficient management of the user's purchased items, optimal proposals, and rapid ordering of missing ingredients.

[0255] "Photographing means" refers to a device or method that allows a user to photograph a receipt.

[0256] "Transmission means" refers to a device or method for transmitting a captured receipt image to a server.

[0257] The "analysis means" is a device or method for converting a received receipt image into text data using optical character recognition technology.

[0258] The "storage means" refers to a device or method for extracting product information from the analyzed text data and storing it in association with the user's account.

[0259] The "proposal generating means" is a device or method for managing and proposing various products owned by a user based on the stored product information.

[0260] The "notification means" is a device or method for notifying the user of the generated proposal.

[0261] An "action processing means" is a device or method for receiving a user's action and updating data.

[0262] "Order processing means" refers to a device or method that allows a user to order missing ingredients through a food delivery service.

[0263] "Optical character recognition technology" is a technology for extracting text data from images.

[0264] This invention provides a system for enabling a user to efficiently manage the goods purchased or items owned in daily life and receive optimal suggestions. This system includes a photographing means, a transmitting means, an analyzing means, a storing means, a suggestion generating means, a notifying means, an action processing means, and an order processing means.

[0265] System Program

[0266] The system is programmed as follows:

[0267] First, the user takes a photo of the receipt using a device such as a smartphone or tablet. The image capturing unit captures this receipt image and temporarily stores it on the device. The user checks the captured image, and if there are no particular problems, they press the "Start Analysis" button, which causes the receipt image to be sent to the server by the transmission unit.

[0268] The server uses optical character recognition (OCR) technology to analyze the received receipt image and extract text data from the image. This analysis is performed using software such as PIL (Python Imaging Library) and Pytesseract. Product information is extracted from the analyzed text data and stored in a database in association with the user's account using a storage method. This stored data includes the product name, price, quantity, purchase date, etc.

[0269] The proposal generation means updates the management information for various products owned by the user based on the saved data and generates optimal proposals. For example, for food, it suggests recipes based on the ingredients possessed. It also suggests purchasing ingredients that are in short supply. A generative AI model is used to make proposals based on the user's purchase history and preferences.

[0270] The notification means notifies the user's device of the suggested recipe and information about missing ingredients. The user can receive the notification and check the suggested content. For example, a notification saying "Basil is needed to make tomato pasta" may be displayed.

[0271] When a user takes some action, for example, when ordering missing ingredients, the order processing means comes into play. The user can order the ingredients they need through an order link linked to a food delivery service. Once the order is confirmed, the data is updated in the server's database.

[0272] Hardware and software used

[0273] The hardware used includes smartphones and tablets, which act as the image capture and transmission devices. The server handles optical character recognition analysis, data storage, suggestion generation, notifications, and order processing. The software used includes PIL, Pytesseract, and a REST API.

[0274] Specific examples

[0275] A specific example is when a user buys "tomatoes, pasta, and cheese" at a supermarket and then takes a photo of the receipt and analyzes it. After analysis, "tomato pasta" is suggested as a recipe, but the necessary "basil" is missing, so a link to purchase basil via food delivery is provided. Through this link, the user can quickly order the missing ingredients.

[0276] Prompt Sentence Examples

[0277] "Please take a photo of the receipt for the items purchased at the supermarket and analyze it using OCR. Based on the analyzed content, please suggest the most suitable recipe based on the stored ingredient information, and allow the user to order any missing ingredients through food delivery. For example, if a user purchases 'tomatoes, pasta, and cheese,' the system will suggest a recipe for 'tomato pasta,' and allow the user to order the necessary 'basil.'"

[0278] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0279] Step 1:

[0280] The user takes a photo of the receipt they received at the supermarket using their smartphone. The input is the receipt image, and the output is image data temporarily saved on the smartphone. This operation allows the user to digitally record the product information they purchased.

[0281] Step 2:

[0282] The terminal checks the captured receipt image, and when the user presses the "Start analysis" button, the terminal uses a transmission means to send the receipt image to the server. The input is the temporarily saved receipt image data, and the output is the image data sent to the server via the Internet.

[0283] Step 3:

[0284] The server receives the sent receipt image and analyzes it using optical character recognition (OCR) technology. The input is the receipt image data, and the output is the extracted text data. The server uses PIL and Pytesseract to extract text information such as product name, price, quantity, and purchase date from the image data.

[0285] Step 4:

[0286] The server extracts product information from the parsed text data and stores it in a database. The input is the text data obtained by OCR analysis, and the output is product information stored in association with the user's account. This data includes the product name, price, quantity, purchase date, etc.

[0287] Step 5:

[0288] Based on the stored product information, the server uses a generative AI model to generate suggestions. The input is the product information stored in the database, and the output is suggestions based on the user's preferences and purchase history. For example, it can suggest recipes based on ingredients the user has on hand and suggest purchasing ingredients they are running low on.

[0289] Step 6:

[0290] The server notifies the user's smartphone of the proposed content using a notification means. The input is the generated proposal content, and the output is a proposal notification displayed on the user's device. For example, a notification saying "Basil is needed to make tomato pasta" is displayed.

[0291] Step 7:

[0292] The user checks the notification and, if there are any missing ingredients, places an order through a food delivery service via their smartphone. The input is information about the missing ingredients, and the output is an order link generated by the order processing means and the food order via that link.

[0293] Step 8:

[0294] The server receives the user's completed order and updates the database. The input is the user's order information obtained through the order processing means, and the output is the updated property information and data to be reflected in the next proposal.

[0295] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0296] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[0297] System Configuration

[0298] The system includes the following main components:

[0299] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[0300] 2. Transmission method: The captured receipt image is sent to the server.

[0301] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[0302] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[0303] 5. Proposal generation means: Based on the stored product information, manages the user's belongings and generates various proposals.

[0304] 6. Emotion engine: Recognizes and analyzes user emotions and adjusts suggestions accordingly.

[0305] 7. Notification means: The generated proposal is notified to the user's terminal.

[0306] 8. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[0307] Program processing and specific examples

[0308] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[0309] Capture and send receipt

[0310] User:

[0311] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[0312] Device:

[0313] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[0314] Receipt image analysis and product data extraction

[0315] server:

[0316] The server passes the received receipt image to an OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[0317] Suggestion generation and emotion regulation

[0318] server:

[0319] Based on the saved data, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed or in short supply and suggests purchases.

[0320] Emotion Engine:

[0321] The emotion engine analyzes the user's voice input and facial expression data and compares it with past emotional data. For example, it recognizes emotions from facial expressions and voice when the user takes a photo of a receipt with the camera and adjusts the suggestions based on that emotional state.

[0322] Proposal generation and distribution

[0323] server:

[0324] Based on the user's profile, past data, and the results of analysis by the emotion engine, the system generates suggestions for flea market prices and new fashions. It also compares prices for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[0325] Distributing proposals and taking action

[0326] server:

[0327] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[0328] Device:

[0329] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[0330] Executing a user action

[0331] User:

[0332] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[0333] Device:

[0334] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[0335] Save and notify results

[0336] server:

[0337] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[0338] Device:

[0339] The user is notified that the process is complete. For example, a notification saying "Basil purchase complete" is displayed on the device. This allows the user to proceed to the next step.

[0340] Specific examples

[0341] Example 1: Food suggestion and emotion regulation implementation

[0342] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[0343] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0344] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. If the user is tired, it will prioritize suggesting easy-to-make recipes.

[0345] 4. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[0346] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[0347] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[0348] In this way, more personalized services can be provided by making suggestions that take the user's emotions into consideration.

[0349] The processing flow will be explained below.

[0350] Step 1:

[0351] User:

[0352] The user launches the app, uses the camera to take a photo of the receipt, adjusts the image so that the entire receipt fits in the frame, and presses the capture button.

[0353] Step 2:

[0354] Device:

[0355] The device displays a preview of the captured receipt image to the user. When the user confirms the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[0356] Step 3:

[0357] server:

[0358] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[0359] Step 4:

[0360] server:

[0361] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[0362] Step 5:

[0363] server:

[0364] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[0365] Step 6:

[0366] server:

[0367] Based on the saved product information, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed or missing and suggests purchases.

[0368] Step 7:

[0369] Emotion Engine:

[0370] The emotion engine analyzes voice input and facial expressions while the user is using the app, and recognizes emotions in real time. The recognized emotion data is sent to the server.

[0371] Step 8:

[0372] server:

[0373] The server receives emotional data from the emotion engine and adjusts the suggestions accordingly. For example, if the user is tired, it will prioritize easy-to-make recipes.

[0374] Step 9:

[0375] server:

[0376] Based on the user's profile, past data, and the results of the emotion engine analysis, the system generates suggestions for flea market prices and new fashions. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[0377] Step 10:

[0378] server:

[0379] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[0380] Step 11:

[0381] Device:

[0382] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[0383] Step 12:

[0384] User:

[0385] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[0386] Step 13:

[0387] Device:

[0388] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[0389] Step 14:

[0390] server:

[0391] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[0392] Step 15:

[0393] Device:

[0394] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[0395] Example 2

[0396] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] Conventional systems have had difficulty effectively managing the products purchased and items owned by users and providing appropriate suggestions based on the user's situation. They also lacked the ability to provide personalized suggestions that take into account the user's emotional state. As a result, the suggestions users received were often not sufficiently useful.

[0398] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0399] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmitting means for transmitting the photographed receipt image to the server, an analyzing means for converting the received receipt image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, an emotion recognition means for analyzing the user's voice and facial expression and adjusting the content of the proposals, a notifying means for notifying the user of the generated proposals, and an action processing means for receiving user actions and updating data. This enables the user to efficiently manage the products purchased and items owned in their daily lives and receive optimal proposals according to the user's emotional state.

[0400] "Photographing means" refers to a device or function that allows a user to photograph a receipt.

[0401] "Transmission means" refers to a device or function for transmitting a captured receipt image to a server.

[0402] The term "analysis means" refers to a device or function that converts the received receipt image into text data using optical character recognition technology.

[0403] "Storage means" refers to a device or function that extracts product information from the analyzed text data and stores the information in association with the user's account.

[0404] The "proposal generating means" refers to a device or function that manages and proposes various products owned by the user based on the stored product information.

[0405] "Emotion recognition means" refers to a device or function that analyzes the user's voice and facial expressions and adjusts the content of suggestions.

[0406] "Notification means" refers to a device or function for notifying the user of the generated suggestions.

[0407] "Action processing means" refers to a device or function that receives a user action and updates data.

[0408] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal on which a dedicated application is installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[0409] System Configuration

[0410] The system includes the following main components:

[0411] Filming method

[0412] Users take a photo of the receipt using a dedicated application, which utilizes the camera function of their smartphone or tablet.

[0413] Transmission method

[0414] The device then sends the captured receipt image to a server via the internet, typically using Wi-Fi or mobile data.

[0415] Analysis means

[0416] The server passes the received receipt image to an optical character recognition (OCR) engine to extract text data from the image, using commonly available commercial or open source software (e.g., Tesseract OCR).

[0417] Preservation means

[0418] The server extracts product information from the analyzed text data, associates it with the user's account, and stores it in a database. The database management system can be MySQL or PostgreSQL.

[0419] Proposal generation means

[0420] The server manages the various products owned by the user based on the stored product information. Specifically, it suggests recipes based on the ingredients owned and suggests purchasing ingredients that are in short supply. It also manages the clothing owned, suggests flea market prices after a certain period of time, and supports the listing of items.

[0421] emotion recognition means

[0422] The emotion engine analyzes the user's voice input and facial expression data to tailor its suggestions, using machine learning models (e.g., generative AI models built using TensorFlow or PyTorch) to analyze the voice and facial expressions.

[0423] Notification means

[0424] The server notifies the user of the generated suggestions via push notification, SMS, email, or other methods.

[0425] Action Processing Means

[0426] It receives user actions (e.g., purchasing a product or putting it up for sale at a flea market), sends related data to the server, and updates the database. This means includes a communication function that receives operational inputs from the terminal and sends that information to the server.

[0427] Specific examples

[0428] Example 1: Food suggestion and emotion regulation implementation

[0429] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[0430] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0431] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is tired, it will prioritize and suggest easy-to-make recipes.

[0432] 4. Suggestion Generation: The server suggests a recipe for "Tomato Pasta" and suggests purchasing the missing basil.

[0433] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[0434] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[0435] Example prompts for generative AI models

[0436] "Please explain the entire process of using this system, from taking a photo of a receipt for groceries purchased at a supermarket to receiving suggestions."

[0437] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0438] Step 1:

[0439] Take a photo of the receipt and confirm the input

[0440] User:

[0441] The user launches the dedicated app and uses the camera function to take a photo of the receipt.

[0442] Input: Receipt image

[0443] Device:

[0444] The device displays the captured receipt image and allows the user to check it on the preview screen. After checking it, the user taps the "Start analysis" button.

[0445] Output: Image of receipt approved by user

[0446] Step 2:

[0447] Sending images

[0448] Device:

[0449] The terminal receives a user's operation and sends the receipt image to the server, using an internet connection for image transmission.

[0450] Input: User approved receipt image

[0451] Output: Receipt image sent to the server

[0452] Step 3:

[0453] Image reception and OCR analysis

[0454] server:

[0455] The server receives the receipt image sent from the terminal and passes it to the OCR engine, which extracts the text data from the image.

[0456] Input: Receipt image

[0457] Output: Text data (product name, price, quantity, purchase date, etc.)

[0458] Step 4:

[0459] Extracting and storing product data

[0460] server:

[0461] The server analyzes the extracted text data, extracts product information, converts the data into fields such as product name, price, quantity, and purchase date, associates it with the user's account, and stores it in a database.

[0462] Input: Text data

[0463] Output: Saved product information

[0464] Step 5:

[0465] Proposal Generation

[0466] server:

[0467] The server updates the management data for the various products owned by the user based on the saved product information. It then suggests recipes based on the ingredients owned and suggests purchasing any ingredients that are lacking. For owned clothing, it also suggests a flea market price after a certain period of time has passed and supports the user in listing the items.

[0468] Input: Saved product information

[0469] Output: Suggestions (recipe suggestions, suggestions for purchasing missing ingredients, flea market suggestions, etc.)

[0470] Step 6:

[0471] Emotion recognition and suggestion adjustment

[0472] Emotion Engine:

[0473] The emotion engine analyzes the user's voice input and facial expression data to determine their current emotional state, and adjusts the suggestions accordingly. For example, if the user is tired, it will suggest easy-to-make recipes.

[0474] Input: User voice input and facial expression data

[0475] Output: Adjusted proposal

[0476] Step 7:

[0477] Proposal Notification

[0478] server:

[0479] The server sends the adjusted proposal to the user's terminal.

[0480] Input: Adjusted proposal

[0481] Output: Suggestions sent to the device

[0482] Device:

[0483] The device will notify the user of the suggestions it receives from the server via a pop-up or notification bar, and will include a link for the user to view more details.

[0484] Input: Adjusted proposal

[0485] Output: Proposal notification to user

[0486] Step 8:

[0487] Executing a user action

[0488] User:

[0489] The user checks the proposed information and selects an action to take, such as accepting the recipe suggestion and purchasing the missing ingredients.

[0490] Input: Proposal

[0491] Output: User's action selection

[0492] Device:

[0493] The device responds to the user's selection and supports specific actions, such as providing information about the nearest store or a link to purchase from an online store.

[0494] Input: User action selection

[0495] Output: Action support (store information, purchase link, etc.)

[0496] Step 9:

[0497] Save and notify results

[0498] server:

[0499] Once a user completes an action, the server updates the database based on that action. For example, if a user purchases basil, that purchase is recorded.

[0500] Input: User action completion information

[0501] Output: Updated database

[0502] Device:

[0503] Notify the user that the transaction is complete. For example, a notification might say, "Your basil purchase is complete."

[0504] Input: Completion notification information from the server

[0505] Output: Notification of completion to the user

[0506] (Application example 2)

[0507] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0508] In recent years, there has been a demand for systems that can efficiently manage users' purchasing behavior and possessions in their daily lives and provide optimal suggestions to users. However, current systems lack the ability to provide suggestions that take into account the user's emotional state, and the suggestions provided are often inappropriate for the user's current situation or emotional state. This has led to problems such as users feeling indifferent or dissatisfied with the suggestions provided.

[0509] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a photographing means for the user to photograph a receipt; a transmitting means for transmitting the photographed receipt image to the server; an analyzing means for converting the received receipt image into text data using optical character recognition technology; a storing means for extracting product information from the analyzed text data and storing it in association with the user's account; a proposal generating means for managing and proposing various products owned by the user based on the stored product information; an emotion analyzing means for recognizing the user's emotional state and adjusting the content of the proposals; a notifying means for notifying the user of the generated proposals; and an action processing means for receiving user actions and updating data. This makes it possible to provide proposals tailored to the user's current emotional state, thereby realizing a more personalized service.

[0510] "User" refers to an individual or corporation that uses this system.

[0511] A "receipt" is a paper or digital data containing purchase information issued by a store when a user purchases a product.

[0512] "Photographing means" refers to a device or application with a camera function that allows a user to photograph a receipt or other object.

[0513] "Transmission means" refers to a communication function for transmitting the captured receipt image to an external system such as a server.

[0514] A "server" is a central processing system that manages, analyzes, stores, and generates proposals on data.

[0515] "Optical character recognition technology" is a technology that extracts text data from images, and is called Optical Character Recognition (OCR) in English.

[0516] "Analysis means" refers to the function of converting the received receipt image into text data using optical character recognition technology and analyzing the content.

[0517] "Text data" is character information extracted by the analysis means.

[0518] "Product information" refers to purchase data such as the product name, price, quantity, and purchase date and time listed on the receipt.

[0519] "Storage means" refers to a function for storing analyzed product information in a database and associating it with a user's account.

[0520] The "proposal generating means" is a function that manages and proposes various products owned by the user based on the stored product information.

[0521] "Emotion analysis means" refers to the function of recognizing and analyzing the user's emotional state from their voice, facial expressions, etc., and adjusting the content of suggestions based on the results.

[0522] The "notification means" is a function for notifying the user of the generated proposal.

[0523] The "action processing means" refers to a function that receives an action in response to a user's suggestion and updates data based on that action.

[0524] "Content" refers to entertainment or information media that can be viewed or subscribed to, such as movies, dramas, music, and books.

[0525] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. This system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[0526] System Configuration

[0527] The system includes the following main components:

[0528] 1. Photography Method:

[0529] The user uses a dedicated app to take a photo of the receipt using the camera function of their smartphone, smart glasses, or head-mounted display, and the captured image of the receipt is saved on the device.

[0530] 2. Means of transmission:

[0531] After the user takes a photo of the receipt, the image is sent to the server via a dedicated app, using an internet connection.

[0532] 3. Analysis method:

[0533] The server converts the received receipt image into text data using optical character recognition (OCR) technology. The OCR engine used is "Tesseract OCR."

[0534] 4. Preservation means:

[0535] Product information is extracted from the analyzed text data, associated with the user's account, and stored in a database (MongoDB).

[0536] 5. Proposal generation means:

[0537] Based on the stored product information, the system manages the user's inventory and generates tailored suggestions based on the user's emotional state, including recipe suggestions, product purchases, and entertainment content suggestions.

[0538] 6. Emotion analysis means:

[0539] The system analyzes the user's voice input and facial expression data to recognize the user's emotional state. This analysis is performed using Microsoft Azure Cognitive Services.

[0540] 7. Means of notification:

[0541] The generated suggestions are notified to the user using Firebase Cloud Messaging (FCM). Users can receive notifications via their smartphones, smart glasses, or head-mounted displays.

[0542] 8. Action Processing Means:

[0543] Receive an action from the user to accept or decline the suggestion and update the data accordingly.

[0544] Specific examples

[0545] Example: Movie proposal

[0546] 1. Shooting and sending:

[0547] The user takes a photo of the movie theater receipt with their smartphone and sends the image to the server via a dedicated app.

[0548] 2. Image Analysis:

[0549] The server uses Tesseract OCR to analyze the receipt image, extract product information such as the movie title, viewing date and time, and ticket purchase amount, and stores this information in MongoDB.

[0550] 3. Emotion recognition:

[0551] The emotion engine analyzes the user's facial expressions and voice to recognize their current emotional state. For example, the analysis may determine that the user is "tired."

[0552] 4. Proposal generation:

[0553] The server suggests relaxing movies (e.g., comedy movies) to the user based on the stored product information and emotional state.

[0554] 5. Notice and Execution:

[0555] Firebase Cloud Messaging is used to notify the user of the suggestion, and the user watches a relaxing movie.

[0556] Prompt Sentence Examples

[0557] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[0558] This system configuration and process makes it possible to provide suggestions tailored to the user's current emotional state, enabling more personalized services.

[0559] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0560] Step 1:

[0561] Capture and send receipt

[0562] After receiving a receipt at the movie theater, the user launches a dedicated app on their smartphone or smart glasses and takes a photo of the receipt. The captured image is previewed on the app, and after the user confirms it, they press the send button to send the image to the server. The input here is the captured receipt image, and the output is the image data sent to the server.

[0563] Step 2:

[0564] Receipt image analysis

[0565] The server retrieves the received receipt image and uses the OCR engine, Tesseract OCR, to extract text data from the image. The specific operation of image analysis is to read the image file, detect the text area, and convert it into text data. The input here is the receipt image data, and the output is extracted product information (movie title, viewing date and time, purchase amount, etc.).

[0566] Step 3:

[0567] Saving product information

[0568] The server organizes the product information analyzed by OCR, associates it with the user's account, and stores it in a database (MongoDB). The input here is the analyzed text data, and the output is the completed save to the database. Specifically, the product information is stored, associated with the user's account as a key.

[0569] Step 4:

[0570] Emotion analysis

[0571] While the user is taking a photo of the receipt, a dedicated app records video and audio, capturing the user's facial expressions and voice. The emotion engine (Microsoft Azure Cognitive Services) analyzes this data to recognize and determine the user's emotional state. The input here is the user's voice and facial expression data, and the output is the recognized emotional state. Specific operations involve the use of facial expression analysis algorithms and voice analysis algorithms.

[0572] Step 5:

[0573] Proposal Generation

[0574] The server generates recommendations appropriate for the user based on the stored product information and emotional state. In this case, if the user is determined to be "tired," it will suggest relaxing movies (e.g., comedy movies). The input here is product information and emotional state, and the output is a list of suggested content. Specific operations include filtering and a recommendation engine based on the emotional state.

[0575] Step 6:

[0576] Proposal Notification

[0577] The server sends the generated suggestions to the user's device using Firebase Cloud Messaging (FCM). The user receives the notification on their smartphone, smart glasses, or head-mounted display. The input here is the suggestion content, and the output is the notification sent to the device. Specifically, the notification is sent in real time via the FCM API.

[0578] Step 7:

[0579] Handling User Actions

[0580] The user receives a notification and confirms the content suggestion. If the user chooses to watch the suggested movie, the server records the action and updates the database. The input is the user's action, and the output is the updated database. Specific operations include adding a viewing record and saving it as training data for the recommendation algorithm.

[0581] Prompt Sentence Examples

[0582] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[0583] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0584] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0585] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0586] [Second embodiment]

[0587] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0588] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0589] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0590] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0591] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0592] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0593] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0594] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0597] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0598] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0599] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. The system includes a terminal on which a dedicated app is installed and a server connected via a network.

[0600] System Configuration

[0601] The system includes the following main components:

[0602] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[0603] 2. Transmission method: The captured receipt image is sent to the server.

[0604] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[0605] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[0606] 5. Proposal generation means: Based on the stored product information, manages the user's possessions and generates various proposals.

[0607] 6. Notification means: The generated proposal is notified to the user's terminal.

[0608] 7. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[0609] Program processing and specific examples

[0610] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[0611] Capture and send receipt

[0612] User:

[0613] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[0614] Device:

[0615] The device temporarily saves the captured receipt image. The user checks the image and, if there are no problems, presses the "Start Analysis" button. The device then sends the receipt image to the server.

[0616] Receipt image analysis and product data extraction

[0617] server:

[0618] The server passes the received receipt image to the OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[0619] Proposal generation and distribution

[0620] server:

[0621] The system updates the management data for various products owned by the user based on the saved data. For example, for food, it generates recipe suggestions based on the ingredients owned and suggests purchasing the necessary ingredients. For clothing, it suggests flea market prices and new fashions. For household goods, it compares prices of detergents and beauty products and suggests the lowest prices.

[0622] Device:

[0623] Once a proposal is generated, the server notifies the user's device, which then displays the proposal to the user for further confirmation.

[0624] Executing a user action

[0625] User:

[0626] The user can check the suggestions and take action such as creating a recipe, purchasing missing ingredients, listing clothing, purchasing daily necessities, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy the basil they are running low on.

[0627] Device:

[0628] Depending on the action selected by the user, the device will display an appropriate screen, such as supporting the process of selling clothes at a flea market, or providing links to purchase ingredients that are in short supply.

[0629] Save and notify results

[0630] server:

[0631] Once the user completes their action, the server updates the data based on that behavior and records it to improve the accuracy of future suggestions.

[0632] Device:

[0633] The user is notified that the transaction is complete. For example, a notification saying "Basil purchase complete" is displayed on the device.

[0634] Specific examples

[0635] Example 1: Food suggestion implementation

[0636] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[0637] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0638] 3. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[0639] 4. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[0640] 5. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[0641] In this way, a system is realized that allows users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[0642] The processing flow will be explained below.

[0643] Step 1:

[0644] User:

[0645] The user launches the app, takes a photo of the receipt using the camera, presses the capture button, and adjusts the image so that the entire receipt is visible.

[0646] Step 2:

[0647] Device:

[0648] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[0649] Step 3:

[0650] server:

[0651] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[0652] Step 4:

[0653] server:

[0654] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[0655] Step 5:

[0656] server:

[0657] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[0658] Step 6:

[0659] server:

[0660] The system updates inventory management data based on the saved product information. For example, for food, it generates recipe suggestions based on the ingredients you have. It also creates a list of ingredients you need or are lacking and suggests purchases.

[0661] Step 7:

[0662] server:

[0663] Based on the user's profile and past data, the system generates flea market price suggestions and new product recommendations. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[0664] Step 8:

[0665] server:

[0666] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products.

[0667] Step 9:

[0668] Device:

[0669] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[0670] Step 10:

[0671] User:

[0672] After checking the suggested content, the user selects an action such as creating a recipe, purchasing ingredients, listing clothing, purchasing a product, etc. For example, if the user accepts the recipe suggestion, they select "View Recipe."

[0673] Step 11:

[0674] Device:

[0675] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[0676] Step 12:

[0677] server:

[0678] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[0679] Step 13:

[0680] Device:

[0681] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[0682] Example 1

[0683] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0684] In conventional item management systems, it was difficult to efficiently manage information about items owned by users. Furthermore, the technology for making optimal suggestions to users was insufficient, so users had to search for and select information themselves, which resulted in issues such as reduced user convenience and complicated item management.

[0685] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0686] In this invention, the server includes a photographing means for a user to photograph a certificate, a transmitting means for transmitting the photographed certificate image to an information processing device, an analyzing means for converting the received certificate image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing the information in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, a notifying means for notifying the user of the generated proposals, and an action processing means for executing an action selected by the user based on the proposal received and updating the data. This allows the user to automatically manage product information and receive optimal proposals simply by photographing the certificate.

[0687] "Photographing means" refers to a camera function that allows a user to photograph a document (such as a receipt) and an application function that supports such photographing.

[0688] "Transmission means" refers to a communication function or protocol for transmitting the photographed certificate image to an information processing device (such as a server).

[0689] "Analysis means" refers to software or algorithms for converting received certificate images into text data using optical character recognition (OCR) technology.

[0690] "Storage means" refers to a database or storage function for storing product information extracted from the analyzed text data in association with the user's account.

[0691] "Proposal generation means" refers to an algorithm or generative AI model for managing various products owned by a user and generating optimal proposals based on stored product information.

[0692] "Notification means" refers to a notification function (e.g., push notification or email notification) or its implementation for notifying the user of the generated proposal.

[0693] "Action processing means" refers to a processing function for executing an action selected by the user based on the proposal received and for reflecting the result in the database.

[0694] "Voucher" refers to a document issued as a result of a transaction (e.g., receipt, invoice).

[0695] "Information processing device" refers to a computer system or server for receiving, analyzing, storing data, and generating suggestions.

[0696] "Optical character recognition technology (OCR)" refers to technology for extracting character information from an image and converting it into text data.

[0697] "Generative AI model" refers to an artificial intelligence model (e.g., GPT-3, GPT-4) that generates optimal suggestions based on user data.

[0698] MODE FOR CARRYING OUT THE INVENTION

[0699] System Configuration

[0700] This invention is a system that allows users to photograph documents (receipts, etc.) and efficiently manage and display the information. The system includes a terminal with a dedicated app installed and an information processing device (server) connected via a network.

[0701] Hardware and Software

[0702] Device:

[0703] The device is a mobile device such as a smartphone or tablet that has a camera and internet connection, and a dedicated app installed.

[0704] server:

[0705] A server is a computer system equipped with a high-performance processor and a large amount of memory. The following software is required for a server:

[0706] OCR engine: Optical character recognition software such as ABBYY FineReader or Tesseract.

[0707] Database: A relational database management system such as MySQL or PostgreSQL.

[0708] Generative AI models: Use artificial intelligence models such as GPT-3 and GPT-4.

[0709] Explaining program processing in natural language

[0710] The user takes a photo of the certificate for an item purchased in daily life using a dedicated app. During this process, the image of the certificate taken by the user is temporarily saved in the device.

[0711] Transmission method:

[0712] When the user presses the "Start Analysis" button, the device sends the certificate image to the server via an internet connection.

[0713] Analysis method:

[0714] The server passes the received certificate image to an OCR engine, extracts the text information from the image, and converts the image into text data using ABBYY FineReader or Tesseract.

[0715] Storage method:

[0716] The server extracts product names, prices, quantities, purchase dates, etc. from the analyzed text data and stores them in a database. At this time, product names are standardized to accommodate differences in notation at different stores.

[0717] Suggestion generation means:

[0718] The server updates the user's inventory based on the stored product data. For food, it generates recipe suggestions based on the ingredients the user has in their possession. It uses a generative AI model to suggest optimal recipes based on the user's preferences and past selection history.

[0719] For example, if a user purchases tomatoes, pasta, and cheese at a supermarket, the server will suggest a recipe for "tomato pasta" based on this information and suggest purchasing any basil that is in short supply.

[0720] Means of notification:

[0721] Once a suggestion is generated, the server notifies the device of its contents, and upon receiving the notification, the device displays the suggestion to the user.

[0722] Action processing method:

[0723] After the user checks the suggestions and selects the necessary action, the device will provide a screen and link for carrying out that action, such as providing a link to purchase missing ingredients so the user can purchase them from an online shop.

[0724] Save the results:

[0725] The server records the results of the user's actions in a database and improves the accuracy of suggestions from the next time onwards, thereby making the user's life more efficient.

[0726] Specific examples

[0727] Example 1: Food suggestion implementation

[0728] A user takes a photo of a receipt from a supermarket purchase, and the device sends the image to the server.

[0729] The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0730] The server uses a generative AI model to suggest a recipe for "tomato pasta" and suggests purchasing the missing basil.

[0731] The terminal notifies the user of the proposed content, and the user decides to purchase the basil.

[0732] The device assists with the purchase process, and the server records the updated data, allowing users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[0733] Prompt Sentence Examples

[0734] A specific example of a prompt sentence that a user might enter into a generative AI model is as follows:

[0735] Example prompt: "I took a photo of the receipt for the groceries I bought at the supermarket. Can you suggest a meal based on the ingredients I have on hand?"

[0736] By entering such prompts, the generative AI model will suggest appropriate recipes and notify you of any missing ingredients.

[0737] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0738] Step 1: User takes a photo of the certificate

[0739] User:

[0740] The user launches the dedicated app and takes a photo of the receipt using the camera on their smartphone or tablet. Specifically, they make sure the receipt fits on the screen and press the capture button.

[0741] input:

[0742] Physical voucher.

[0743] output:

[0744] A voucher image in digital image format.

[0745] Specific behavior:

[0746] The smartphone camera takes a photo of the certificate and saves the image in a dedicated app.

[0747] Step 2: Send the certificate image to the server

[0748] Device:

[0749] The terminal temporarily stores the captured certificate image. When the user checks the image and presses the "Start Analysis" button, the terminal sends the certificate image to the server.

[0750] input:

[0751] A voucher image in digital image format.

[0752] output:

[0753] The voucher image sent to the server.

[0754] Specific behavior:

[0755] Using the sending function within the dedicated app, the certificate image is uploaded to the server via the Internet.

[0756] Step 3: Analysis of the voucher image

[0757] server:

[0758] The server inputs the received certificate image into an OCR engine and extracts text information from the image.

[0759] input:

[0760] The voucher image sent to the server.

[0761] output:

[0762] Extracted text data (product name, price, quantity, purchase date, etc.).

[0763] Specific behavior:

[0764] An OCR engine (such as ABBYY FineReader or Tesseract) is used to convert the text information in the certificate image into text data.

[0765] Step 4: Store and standardize text data

[0766] server:

[0767] The server extracts product information from the extracted text data, standardizes the product name, associates it with the user's account, and stores it in a database.

[0768] input:

[0769] The extracted text data.

[0770] output:

[0771] Standardized product information stored in a database.

[0772] Specific behavior:

[0773] Product information (product name, price, quantity, purchase date, etc.) is inserted into the database using an SQL query, and the product is recognized as the same product according to product name standardization rules.

[0774] Step 5: Generate proposals

[0775] server:

[0776] The server uses the stored product information to update the user's inventory using a generative AI model and generate optimal suggestions, such as recipe suggestions based on the ingredients the user owns.

[0777] input:

[0778] Standardized product information stored in a database.

[0779] output:

[0780] Suggestions for users.

[0781] Specific behavior:

[0782] Using generative AI models (such as GPT-3 and GPT-4), it analyzes the user's inventory and generates optimal suggestions based on the user's preferences and past selection history.

[0783] Step 6: Proposal Notification

[0784] server:

[0785] The generated proposal is sent to the device for notification.

[0786] input:

[0787] The generated proposal.

[0788] output:

[0789] A notification message that is displayed on the user's terminal.

[0790] Specific behavior:

[0791] The generated suggestions are sent to the user's device using push notifications or email notifications.

[0792] Step 7: User Actions

[0793] User:

[0794] The user reviews the suggestions and selects and executes the necessary action, for example, purchasing ingredients for the suggested recipe.

[0795] input:

[0796] User selection based on suggestions.

[0797] output:

[0798] The result of the action taken (e.g., checkout).

[0799] Specific behavior:

[0800] From within the dedicated app, users can access links and follow procedures to purchase the necessary ingredients from online shops.

[0801] Step 8: Save and notify results

[0802] server:

[0803] The results of the user's actions are recorded in a database to improve the accuracy of the next suggestion, and the user is notified when the action is complete.

[0804] input:

[0805] The result of the action taken.

[0806] output:

[0807] Action results and user notification messages stored in the database.

[0808] Specific behavior:

[0809] Update the database using SQL queries to save the action results, and use the notification function to send an action completion message to the user's device.

[0810] (Application example 1)

[0811] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0812] In modern society, it remains difficult for users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations based on that information. In particular, efficiently managing ingredients and purchasing the ingredients they need requires a lot of effort, placing a heavy burden on users. For this reason, there is a need for a system that allows users to efficiently manage their owned items, receive optimal recommendations to optimize their lives, and quickly order ingredients they need through food delivery services.

[0813] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0814] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmitting means for transmitting the photographed receipt image to the server, an analyzing means for converting the received receipt image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, a notifying means for notifying the user of the generated proposals, an action processing means for receiving user actions and updating the data, and an order processing means for allowing the user to order missing ingredients via a food delivery service. This enables efficient management of the user's purchased items, optimal proposals, and rapid ordering of missing ingredients.

[0815] "Photographing means" refers to a device or method that allows a user to photograph a receipt.

[0816] "Transmission means" refers to a device or method for transmitting a captured receipt image to a server.

[0817] The "analysis means" is a device or method for converting a received receipt image into text data using optical character recognition technology.

[0818] The "storage means" refers to a device or method for extracting product information from the analyzed text data and storing it in association with the user's account.

[0819] The "proposal generating means" is a device or method for managing and proposing various products owned by a user based on the stored product information.

[0820] The "notification means" is a device or method for notifying the user of the generated proposal.

[0821] An "action processing means" is a device or method for receiving a user's action and updating data.

[0822] "Order processing means" refers to a device or method that allows a user to order missing ingredients through a food delivery service.

[0823] "Optical character recognition technology" is a technology for extracting text data from images.

[0824] This invention provides a system for enabling a user to efficiently manage the goods purchased or items owned in daily life and receive optimal suggestions. This system includes a photographing means, a transmitting means, an analyzing means, a storing means, a suggestion generating means, a notifying means, an action processing means, and an order processing means.

[0825] System Program

[0826] The system is programmed as follows:

[0827] First, the user takes a photo of the receipt using a device such as a smartphone or tablet. The image capturing unit captures this receipt image and temporarily stores it on the device. The user checks the captured image, and if there are no particular problems, they press the "Start Analysis" button, which causes the receipt image to be sent to the server by the transmission unit.

[0828] The server uses optical character recognition (OCR) technology to analyze the received receipt image and extract text data from the image. This analysis is performed using software such as PIL (Python Imaging Library) and Pytesseract. Product information is extracted from the analyzed text data and stored in a database in association with the user's account using a storage method. This stored data includes the product name, price, quantity, purchase date, etc.

[0829] The proposal generation means updates the management information for various products owned by the user based on the saved data and generates optimal proposals. For example, for food, it suggests recipes based on the ingredients possessed. It also suggests purchasing ingredients that are in short supply. A generative AI model is used to make proposals based on the user's purchase history and preferences.

[0830] The notification means notifies the user's device of the suggested recipe and information about missing ingredients. The user can receive the notification and check the suggested content. For example, a notification saying "Basil is needed to make tomato pasta" may be displayed.

[0831] When a user takes some action, for example, when ordering missing ingredients, the order processing means comes into play. The user can order the ingredients they need through an order link linked to a food delivery service. Once the order is confirmed, the data is updated in the server's database.

[0832] Hardware and software used

[0833] The hardware used includes smartphones and tablets, which act as the image capture and transmission devices. The server handles optical character recognition analysis, data storage, suggestion generation, notifications, and order processing. The software used includes PIL, Pytesseract, and a REST API.

[0834] Specific examples

[0835] A specific example is when a user buys "tomatoes, pasta, and cheese" at a supermarket and then takes a photo of the receipt and analyzes it. After analysis, "tomato pasta" is suggested as a recipe, but the necessary "basil" is missing, so a link to purchase basil via food delivery is provided. Through this link, the user can quickly order the missing ingredients.

[0836] Prompt Sentence Examples

[0837] "Please take a photo of the receipt for the items purchased at the supermarket and analyze it using OCR. Based on the analyzed content, please suggest the most suitable recipe based on the stored ingredient information, and allow the user to order any missing ingredients through food delivery. For example, if a user purchases 'tomatoes, pasta, and cheese,' the system will suggest a recipe for 'tomato pasta,' and allow the user to order the necessary 'basil.'"

[0838] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0839] Step 1:

[0840] The user takes a photo of the receipt they received at the supermarket using their smartphone. The input is the receipt image, and the output is image data temporarily saved on the smartphone. This operation allows the user to digitally record the product information they purchased.

[0841] Step 2:

[0842] The terminal checks the captured receipt image, and when the user presses the "Start analysis" button, the terminal uses a transmission means to send the receipt image to the server. The input is the temporarily saved receipt image data, and the output is the image data sent to the server via the Internet.

[0843] Step 3:

[0844] The server receives the sent receipt image and analyzes it using optical character recognition (OCR) technology. The input is the receipt image data, and the output is the extracted text data. The server uses PIL and Pytesseract to extract text information such as product name, price, quantity, and purchase date from the image data.

[0845] Step 4:

[0846] The server extracts product information from the parsed text data and stores it in a database. The input is the text data obtained by OCR analysis, and the output is product information stored in association with the user's account. This data includes the product name, price, quantity, purchase date, etc.

[0847] Step 5:

[0848] Based on the stored product information, the server uses a generative AI model to generate suggestions. The input is the product information stored in the database, and the output is suggestions based on the user's preferences and purchase history. For example, it can suggest recipes based on ingredients the user has on hand and suggest purchasing ingredients they are running low on.

[0849] Step 6:

[0850] The server notifies the user's smartphone of the proposed content using a notification means. The input is the generated proposal content, and the output is a proposal notification displayed on the user's device. For example, a notification saying "Basil is needed to make tomato pasta" is displayed.

[0851] Step 7:

[0852] The user checks the notification and, if there are any missing ingredients, places an order through a food delivery service via their smartphone. The input is information about the missing ingredients, and the output is an order link generated by the order processing means and the food order via that link.

[0853] Step 8:

[0854] The server receives the user's completed order and updates the database. The input is the user's order information obtained through the order processing means, and the output is the updated property information and data to be reflected in the next proposal.

[0855] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0856] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[0857] System Configuration

[0858] The system includes the following main components:

[0859] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[0860] 2. Transmission method: The captured receipt image is sent to the server.

[0861] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[0862] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[0863] 5. Proposal generation means: Based on the stored product information, manages the user's belongings and generates various proposals.

[0864] 6. Emotion engine: Recognizes and analyzes user emotions and adjusts suggestions accordingly.

[0865] 7. Notification means: The generated proposal is notified to the user's terminal.

[0866] 8. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[0867] Program processing and specific examples

[0868] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[0869] Capture and send receipt

[0870] User:

[0871] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[0872] Device:

[0873] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[0874] Receipt image analysis and product data extraction

[0875] server:

[0876] The server passes the received receipt image to an OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[0877] Suggestion generation and emotion regulation

[0878] server:

[0879] Based on the saved data, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed or in short supply and suggests purchases.

[0880] Emotion Engine:

[0881] The emotion engine analyzes the user's voice input and facial expression data and compares it with past emotional data. For example, it recognizes emotions from facial expressions and voice when the user takes a photo of a receipt with the camera and adjusts the suggestions based on that emotional state.

[0882] Proposal generation and distribution

[0883] server:

[0884] Based on the user's profile, past data, and the results of analysis by the emotion engine, the system generates suggestions for flea market prices and new fashions. It also compares prices for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[0885] Distributing proposals and taking action

[0886] server:

[0887] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[0888] Device:

[0889] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[0890] Executing a user action

[0891] User:

[0892] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[0893] Device:

[0894] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[0895] Save and notify results

[0896] server:

[0897] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[0898] Device:

[0899] The user is notified that the process is complete. For example, a notification saying "Basil purchase complete" is displayed on the device. This allows the user to proceed to the next step.

[0900] Specific examples

[0901] Example 1: Food suggestion and emotion regulation implementation

[0902] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[0903] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0904] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. If the user is tired, it will prioritize suggesting easy-to-make recipes.

[0905] 4. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[0906] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[0907] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[0908] In this way, more personalized services can be provided by making suggestions that take the user's emotions into consideration.

[0909] The processing flow will be explained below.

[0910] Step 1:

[0911] User:

[0912] The user launches the app, uses the camera to take a photo of the receipt, adjusts the image so that the entire receipt fits in the frame, and presses the capture button.

[0913] Step 2:

[0914] Device:

[0915] The device displays a preview of the captured receipt image to the user. When the user confirms the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[0916] Step 3:

[0917] server:

[0918] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[0919] Step 4:

[0920] server:

[0921] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[0922] Step 5:

[0923] server:

[0924] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[0925] Step 6:

[0926] server:

[0927] Based on the saved product information, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed and those in short supply and suggests purchases.

[0928] Step 7:

[0929] Emotion Engine:

[0930] The emotion engine analyzes voice input and facial expressions while the user is using the app, and recognizes emotions in real time. The recognized emotion data is sent to the server.

[0931] Step 8:

[0932] server:

[0933] The server receives emotional data from the emotion engine and adjusts the suggestions accordingly. For example, if the user is tired, it will prioritize easy-to-make recipes.

[0934] Step 9:

[0935] server:

[0936] Based on the user's profile, past data, and the results of the emotion engine analysis, the system generates suggestions for flea market prices and new fashions. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[0937] Step 10:

[0938] server:

[0939] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[0940] Step 11:

[0941] Device:

[0942] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[0943] Step 12:

[0944] User:

[0945] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[0946] Step 13:

[0947] Device:

[0948] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[0949] Step 14:

[0950] server:

[0951] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[0952] Step 15:

[0953] Device:

[0954] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[0955] Example 2

[0956] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0957] Conventional systems have had difficulty effectively managing the products purchased and items owned by users and providing appropriate suggestions based on the user's situation. They also lacked the ability to provide personalized suggestions that take into account the user's emotional state. As a result, the suggestions users received were often not sufficiently useful.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0959] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmitting means for transmitting the photographed receipt image to the server, an analyzing means for converting the received receipt image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, an emotion recognition means for analyzing the user's voice and facial expression and adjusting the content of the proposals, a notifying means for notifying the user of the generated proposals, and an action processing means for receiving user actions and updating data. This enables the user to efficiently manage the products purchased and items owned in their daily lives and receive optimal proposals according to the user's emotional state.

[0960] "Photographing means" refers to a device or function that allows a user to photograph a receipt.

[0961] "Transmission means" refers to a device or function for transmitting a captured receipt image to a server.

[0962] The term "analysis means" refers to a device or function that converts the received receipt image into text data using optical character recognition technology.

[0963] "Storage means" refers to a device or function that extracts product information from the analyzed text data and stores the information in association with the user's account.

[0964] The "proposal generating means" refers to a device or function that manages and proposes various products owned by the user based on the stored product information.

[0965] "Emotion recognition means" refers to a device or function that analyzes the user's voice and facial expressions and adjusts the content of suggestions.

[0966] "Notification means" refers to a device or function for notifying the user of the generated suggestions.

[0967] "Action processing means" refers to a device or function that receives a user action and updates data.

[0968] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal on which a dedicated application is installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[0969] System Configuration

[0970] The system includes the following main components:

[0971] Filming method

[0972] Users take a photo of the receipt using a dedicated application, which utilizes the camera function of their smartphone or tablet.

[0973] Transmission method

[0974] The device then sends the captured receipt image to a server via the internet, typically using Wi-Fi or mobile data.

[0975] Analysis means

[0976] The server passes the received receipt image to an optical character recognition (OCR) engine to extract text data from the image, using commonly available commercial or open source software (e.g., Tesseract OCR).

[0977] Preservation means

[0978] The server extracts product information from the analyzed text data, associates it with the user's account, and stores it in a database. The database management system can be MySQL or PostgreSQL.

[0979] Proposal generation means

[0980] The server manages the various products owned by the user based on the stored product information. Specifically, it suggests recipes based on the ingredients owned and suggests purchasing ingredients that are in short supply. It also manages the clothing owned, suggests flea market prices after a certain period of time, and supports the listing of items.

[0981] emotion recognition means

[0982] The emotion engine analyzes the user's voice input and facial expression data to tailor its suggestions, using machine learning models (e.g., generative AI models built using TensorFlow or PyTorch) to analyze the voice and facial expressions.

[0983] Notification means

[0984] The server notifies the user of the generated suggestions via push notification, SMS, email, or other methods.

[0985] Action Processing Means

[0986] It receives user actions (e.g., purchasing a product or putting it up for sale at a flea market), sends related data to the server, and updates the database. This means includes a communication function that receives operational inputs from the terminal and sends that information to the server.

[0987] Specific examples

[0988] Example 1: Food suggestion and emotion regulation implementation

[0989] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[0990] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[0991] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is tired, it will prioritize and suggest easy-to-make recipes.

[0992] 4. Suggestion Generation: The server suggests a recipe for "Tomato Pasta" and suggests purchasing the missing basil.

[0993] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[0994] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[0995] Example prompts for generative AI models

[0996] "Please explain the entire process of using this system, from taking a photo of a receipt for groceries purchased at a supermarket to receiving suggestions."

[0997] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0998] Step 1:

[0999] Take a photo of the receipt and confirm the input

[1000] User:

[1001] The user launches the dedicated app and uses the camera function to take a photo of the receipt.

[1002] Input: Receipt image

[1003] Device:

[1004] The device displays the captured receipt image and allows the user to check it on the preview screen. After checking it, the user taps the "Start analysis" button.

[1005] Output: Image of receipt approved by user

[1006] Step 2:

[1007] Sending images

[1008] Device:

[1009] The terminal receives a user's operation and sends the receipt image to the server, using an internet connection for image transmission.

[1010] Input: User approved receipt image

[1011] Output: Receipt image sent to the server

[1012] Step 3:

[1013] Image reception and OCR analysis

[1014] server:

[1015] The server receives the receipt image sent from the terminal and passes it to the OCR engine, which extracts the text data from the image.

[1016] Input: Receipt image

[1017] Output: Text data (product name, price, quantity, purchase date, etc.)

[1018] Step 4:

[1019] Extracting and storing product data

[1020] server:

[1021] The server analyzes the extracted text data, extracts product information, converts the data into fields such as product name, price, quantity, and purchase date, associates it with the user's account, and stores it in a database.

[1022] Input: Text data

[1023] Output: Saved product information

[1024] Step 5:

[1025] Proposal Generation

[1026] server:

[1027] The server updates the management data for the various products owned by the user based on the saved product information. It then suggests recipes based on the ingredients owned and suggests purchasing any ingredients that are lacking. For owned clothing, it also suggests a flea market price after a certain period of time has passed and supports the user in listing the items.

[1028] Input: Saved product information

[1029] Output: Suggestions (recipe suggestions, suggestions for purchasing missing ingredients, flea market suggestions, etc.)

[1030] Step 6:

[1031] Emotion recognition and suggestion adjustment

[1032] Emotion Engine:

[1033] The emotion engine analyzes the user's voice input and facial expression data to determine their current emotional state, and adjusts the suggestions accordingly. For example, if the user is tired, it will suggest easy-to-make recipes.

[1034] Input: User voice input and facial expression data

[1035] Output: Adjusted proposal

[1036] Step 7:

[1037] Proposal Notification

[1038] server:

[1039] The server sends the adjusted proposal to the user's terminal.

[1040] Input: Adjusted proposal

[1041] Output: Suggestions sent to the device

[1042] Device:

[1043] The device will notify the user of the suggestions it receives from the server via a pop-up or notification bar, and will include a link for the user to view more details.

[1044] Input: Adjusted proposal

[1045] Output: Proposal notification to user

[1046] Step 8:

[1047] Executing a user action

[1048] User:

[1049] The user checks the proposed information and selects an action to take, such as accepting the recipe suggestion and purchasing the missing ingredients.

[1050] Input: Proposal

[1051] Output: User action selection

[1052] Device:

[1053] The device responds to the user's selection and supports specific actions, such as providing information about the nearest store or a link to purchase from an online store.

[1054] Input: User action selection

[1055] Output: Action support (store information, purchase link, etc.)

[1056] Step 9:

[1057] Save and notify results

[1058] server:

[1059] Once a user completes an action, the server updates the database based on that action. For example, if a user purchases basil, that purchase is recorded.

[1060] Input: User action completion information

[1061] Output: Updated database

[1062] Device:

[1063] Notify the user that the transaction is complete. For example, a notification might say, "Your basil purchase is complete."

[1064] Input: Completion notification information from the server

[1065] Output: Notification of completion to the user

[1066] (Application example 2)

[1067] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1068] In recent years, there has been a demand for systems that can efficiently manage users' purchasing behavior and possessions in their daily lives and provide optimal suggestions to users. However, current systems lack the ability to provide suggestions that take into account the user's emotional state, and the suggestions provided are often inappropriate for the user's current situation or emotional state. This has led to problems such as users feeling indifferent or dissatisfied with the suggestions provided.

[1069] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a photographing means for the user to photograph a receipt; a transmitting means for transmitting the photographed receipt image to the server; an analyzing means for converting the received receipt image into text data using optical character recognition technology; a storing means for extracting product information from the analyzed text data and storing it in association with the user's account; a proposal generating means for managing and proposing various products owned by the user based on the stored product information; an emotion analyzing means for recognizing the user's emotional state and adjusting the content of the proposals; a notifying means for notifying the user of the generated proposals; and an action processing means for receiving user actions and updating data. This makes it possible to provide proposals tailored to the user's current emotional state, thereby realizing a more personalized service.

[1070] "User" refers to an individual or corporation that uses this system.

[1071] A "receipt" is a paper or digital data containing purchase information issued by a store when a user purchases a product.

[1072] "Photographing means" refers to a device or application with a camera function that allows a user to photograph a receipt or other object.

[1073] "Transmission means" refers to a communication function for transmitting the captured receipt image to an external system such as a server.

[1074] A "server" is a central processing system that manages, analyzes, stores, and generates proposals on data.

[1075] "Optical character recognition technology" is a technology that extracts text data from images, and is called Optical Character Recognition (OCR) in English.

[1076] "Analysis means" refers to the function of converting the received receipt image into text data using optical character recognition technology and analyzing the content.

[1077] "Text data" is character information extracted by the analysis means.

[1078] "Product information" refers to purchase data such as the product name, price, quantity, and purchase date and time listed on the receipt.

[1079] "Storage means" refers to a function for storing analyzed product information in a database and associating it with a user's account.

[1080] The "proposal generating means" is a function that manages and proposes various products owned by the user based on the stored product information.

[1081] "Emotion analysis means" refers to the function of recognizing and analyzing the user's emotional state from their voice, facial expressions, etc., and adjusting the content of suggestions based on the results.

[1082] The "notification means" is a function for notifying the user of the generated proposal.

[1083] The "action processing means" refers to a function that receives an action in response to a user's suggestion and updates data based on that action.

[1084] "Content" refers to entertainment or information media that can be viewed or subscribed to, such as movies, dramas, music, and books.

[1085] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. This system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[1086] System Configuration

[1087] The system includes the following main components:

[1088] 1. Photography Method:

[1089] The user uses a dedicated app to take a photo of the receipt using the camera function of their smartphone, smart glasses, or head-mounted display, and the captured image of the receipt is saved on the device.

[1090] 2. Means of transmission:

[1091] After the user takes a photo of the receipt, the image is sent to the server via a dedicated app, using an internet connection.

[1092] 3. Analysis method:

[1093] The server converts the received receipt image into text data using optical character recognition (OCR) technology. The OCR engine used is "Tesseract OCR."

[1094] 4. Preservation means:

[1095] Product information is extracted from the analyzed text data, associated with the user's account, and stored in a database (MongoDB).

[1096] 5. Proposal generation means:

[1097] Based on the stored product information, the system manages the user's inventory and generates tailored suggestions based on the user's emotional state, including recipe suggestions, product purchases, and entertainment content suggestions.

[1098] 6. Emotion analysis means:

[1099] The system analyzes the user's voice input and facial expression data to recognize the user's emotional state. This analysis is performed using Microsoft Azure Cognitive Services.

[1100] 7. Means of notification:

[1101] The generated suggestions are notified to the user using Firebase Cloud Messaging (FCM). Users can receive notifications via their smartphones, smart glasses, or head-mounted displays.

[1102] 8. Action Processing Means:

[1103] Receive an action from the user to accept or decline the suggestion and update the data accordingly.

[1104] Specific examples

[1105] Example: Movie proposal

[1106] 1. Shooting and sending:

[1107] The user takes a photo of the movie theater receipt with their smartphone and sends the image to the server via a dedicated app.

[1108] 2. Image Analysis:

[1109] The server uses Tesseract OCR to analyze the receipt image, extract product information such as the movie title, viewing date and time, and ticket purchase amount, and stores this information in MongoDB.

[1110] 3. Emotion recognition:

[1111] The emotion engine analyzes the user's facial expressions and voice to recognize their current emotional state. For example, the analysis may determine that the user is "tired."

[1112] 4. Proposal generation:

[1113] The server suggests relaxing movies (e.g., comedy movies) to the user based on the stored product information and emotional state.

[1114] 5. Notice and Execution:

[1115] Firebase Cloud Messaging is used to notify the user of the suggestion, and the user watches a relaxing movie.

[1116] Prompt Sentence Examples

[1117] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[1118] This system configuration and process makes it possible to provide suggestions tailored to the user's current emotional state, enabling more personalized services.

[1119] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1120] Step 1:

[1121] Capture and send receipt

[1122] After receiving a receipt at the movie theater, the user launches a dedicated app on their smartphone or smart glasses and takes a photo of the receipt. The captured image is previewed on the app, and after the user confirms it, they press the send button to send the image to the server. The input here is the captured receipt image, and the output is the image data sent to the server.

[1123] Step 2:

[1124] Receipt image analysis

[1125] The server retrieves the received receipt image and uses the OCR engine, Tesseract OCR, to extract text data from the image. The specific operation of image analysis is to read the image file, detect the text area, and convert it into text data. The input here is the receipt image data, and the output is extracted product information (movie title, viewing date and time, purchase amount, etc.).

[1126] Step 3:

[1127] Saving product information

[1128] The server organizes the product information analyzed by OCR, associates it with the user's account, and stores it in a database (MongoDB). The input here is the analyzed text data, and the output is the completed save to the database. Specifically, the product information is stored, associated with the user's account as a key.

[1129] Step 4:

[1130] Emotion analysis

[1131] While the user is taking a photo of the receipt, a dedicated app records video and audio, capturing the user's facial expressions and voice. The emotion engine (Microsoft Azure Cognitive Services) analyzes this data to recognize and determine the user's emotional state. The input here is the user's voice and facial expression data, and the output is the recognized emotional state. Specific operations involve the use of facial expression analysis algorithms and voice analysis algorithms.

[1132] Step 5:

[1133] Proposal Generation

[1134] The server generates recommendations appropriate for the user based on the stored product information and emotional state. In this case, if the user is determined to be "tired," it will suggest relaxing movies (e.g., comedy movies). The input here is product information and emotional state, and the output is a list of suggested content. Specific operations include filtering and a recommendation engine based on the emotional state.

[1135] Step 6:

[1136] Proposal Notification

[1137] The server sends the generated suggestions to the user's device using Firebase Cloud Messaging (FCM). The user receives the notification on their smartphone, smart glasses, or head-mounted display. The input here is the suggestion content, and the output is the notification sent to the device. Specifically, the notification is sent in real time via the FCM API.

[1138] Step 7:

[1139] Handling User Actions

[1140] The user receives a notification and confirms the content suggestion. If the user chooses to watch the suggested movie, the server records the action and updates the database. The input is the user's action, and the output is the updated database. Specific operations include adding a viewing record and saving it as training data for the recommendation algorithm.

[1141] Prompt Sentence Examples

[1142] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[1143] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1144] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1145] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1146] [Third embodiment]

[1147] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1148] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1149] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1150] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1151] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1152] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1153] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1154] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1155] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1156] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1157] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1158] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1159] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. The system includes a terminal on which a dedicated app is installed and a server connected via a network.

[1160] System Configuration

[1161] The system includes the following main components:

[1162] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[1163] 2. Transmission method: The captured receipt image is sent to the server.

[1164] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[1165] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[1166] 5. Proposal generation means: Based on the stored product information, manages the user's possessions and generates various proposals.

[1167] 6. Notification means: The generated proposal is notified to the user's terminal.

[1168] 7. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[1169] Program processing and specific examples

[1170] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[1171] Capture and send receipt

[1172] User:

[1173] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[1174] Device:

[1175] The device temporarily saves the captured receipt image. The user checks the image and, if there are no problems, presses the "Start Analysis" button. The device then sends the receipt image to the server.

[1176] Receipt image analysis and product data extraction

[1177] server:

[1178] The server passes the received receipt image to the OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[1179] Proposal generation and distribution

[1180] server:

[1181] The system updates the management data for various products owned by the user based on the saved data. For example, for food, it generates recipe suggestions based on the ingredients owned and suggests purchasing the necessary ingredients. For clothing, it suggests flea market prices and new fashions. For household goods, it compares prices of detergents and beauty products and suggests the lowest prices.

[1182] Device:

[1183] Once a proposal is generated, the server notifies the user's device, which then displays the proposal to the user for further confirmation.

[1184] Executing a user action

[1185] User:

[1186] The user can check the suggestions and take action such as creating a recipe, purchasing missing ingredients, listing clothing, purchasing daily necessities, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy the basil they are running low on.

[1187] Device:

[1188] Depending on the action selected by the user, the device will display an appropriate screen, such as supporting the process of selling clothes at a flea market, or providing links to purchase ingredients that are in short supply.

[1189] Save and notify results

[1190] server:

[1191] Once the user completes their action, the server updates the data based on that behavior and records it to improve the accuracy of future suggestions.

[1192] Device:

[1193] The user is notified that the transaction is complete. For example, a notification saying "Basil purchase complete" is displayed on the device.

[1194] Specific examples

[1195] Example 1: Food suggestion implementation

[1196] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[1197] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[1198] 3. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[1199] 4. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[1200] 5. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[1201] In this way, a system is realized that allows users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[1202] The processing flow will be explained below.

[1203] Step 1:

[1204] User:

[1205] The user launches the app, takes a photo of the receipt using the camera, presses the capture button, and adjusts the image so that the entire receipt is visible.

[1206] Step 2:

[1207] Device:

[1208] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[1209] Step 3:

[1210] server:

[1211] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[1212] Step 4:

[1213] server:

[1214] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[1215] Step 5:

[1216] server:

[1217] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[1218] Step 6:

[1219] server:

[1220] The system updates inventory management data based on the saved product information. For example, for food, it generates recipe suggestions based on the ingredients you have. It also creates a list of ingredients you need or are lacking and suggests purchases.

[1221] Step 7:

[1222] server:

[1223] Based on the user's profile and past data, the system generates flea market price suggestions and new product recommendations. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[1224] Step 8:

[1225] server:

[1226] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products.

[1227] Step 9:

[1228] Device:

[1229] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[1230] Step 10:

[1231] User:

[1232] After checking the suggested content, the user selects an action such as creating a recipe, purchasing ingredients, listing clothing, purchasing a product, etc. For example, if the user accepts the recipe suggestion, they select "View Recipe."

[1233] Step 11:

[1234] Device:

[1235] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[1236] Step 12:

[1237] server:

[1238] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[1239] Step 13:

[1240] Device:

[1241] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[1242] Example 1

[1243] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1244] In conventional item management systems, it was difficult to efficiently manage information about items owned by users. Furthermore, the technology for making optimal suggestions to users was insufficient, so users had to search for and select information themselves, which resulted in issues such as reduced user convenience and complicated item management.

[1245] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1246] In this invention, the server includes a photographing means for a user to photograph a certificate, a transmitting means for transmitting the photographed certificate image to an information processing device, an analyzing means for converting the received certificate image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing the information in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, a notifying means for notifying the user of the generated proposals, and an action processing means for executing an action selected by the user based on the proposal received and updating the data. This allows the user to automatically manage product information and receive optimal proposals simply by photographing the certificate.

[1247] "Photographing means" refers to a camera function that allows a user to photograph a document (such as a receipt) and an application function that supports such photographing.

[1248] "Transmission means" refers to a communication function or protocol for transmitting the photographed certificate image to an information processing device (such as a server).

[1249] "Analysis means" refers to software or algorithms for converting received certificate images into text data using optical character recognition (OCR) technology.

[1250] "Storage means" refers to a database or storage function for storing product information extracted from the analyzed text data in association with the user's account.

[1251] "Proposal generation means" refers to an algorithm or generative AI model for managing various products owned by a user and generating optimal proposals based on stored product information.

[1252] "Notification means" refers to a notification function (e.g., push notification or email notification) or its implementation for notifying the user of the generated proposal.

[1253] "Action processing means" refers to a processing function for executing an action selected by the user based on the proposal received and for reflecting the result in the database.

[1254] "Voucher" refers to a document issued as a result of a transaction (e.g., receipt, invoice).

[1255] "Information processing device" refers to a computer system or server for receiving, analyzing, storing data, and generating suggestions.

[1256] "Optical character recognition technology (OCR)" refers to technology for extracting character information from an image and converting it into text data.

[1257] "Generative AI model" refers to an artificial intelligence model (e.g., GPT-3, GPT-4) that generates optimal suggestions based on user data.

[1258] MODE FOR CARRYING OUT THE INVENTION

[1259] System Configuration

[1260] This invention is a system that allows users to photograph documents (receipts, etc.) and efficiently manage and display the information. The system includes a terminal with a dedicated app installed and an information processing device (server) connected via a network.

[1261] Hardware and Software

[1262] Device:

[1263] The device is a mobile device such as a smartphone or tablet that has a camera and internet connection, and a dedicated app installed.

[1264] server:

[1265] A server is a computer system equipped with a high-performance processor and a large amount of memory. The following software is required for a server:

[1266] OCR engine: Optical character recognition software such as ABBYY FineReader or Tesseract.

[1267] Database: A relational database management system such as MySQL or PostgreSQL.

[1268] Generative AI models: Use artificial intelligence models such as GPT-3 and GPT-4.

[1269] Explaining program processing in natural language

[1270] The user takes a photo of the certificate for an item purchased in daily life using a dedicated app. During this process, the image of the certificate taken by the user is temporarily saved in the device.

[1271] Transmission method:

[1272] When the user presses the "Start Analysis" button, the device sends the certificate image to the server via an internet connection.

[1273] Analysis method:

[1274] The server passes the received certificate image to an OCR engine, extracts the text information from the image, and converts the image into text data using ABBYY FineReader or Tesseract.

[1275] Storage method:

[1276] The server extracts product names, prices, quantities, purchase dates, etc. from the analyzed text data and stores them in a database. At this time, product names are standardized to accommodate differences in notation at different stores.

[1277] Suggestion generation means:

[1278] The server updates the user's inventory based on the stored product data. For food, it generates recipe suggestions based on the ingredients the user has in their possession. It uses a generative AI model to suggest optimal recipes based on the user's preferences and past selection history.

[1279] For example, if a user purchases tomatoes, pasta, and cheese at a supermarket, the server will suggest a recipe for "tomato pasta" based on this information and suggest purchasing any basil that is in short supply.

[1280] Means of notification:

[1281] Once a suggestion is generated, the server notifies the device of its contents, and upon receiving the notification, the device displays the suggestion to the user.

[1282] Action processing method:

[1283] After the user checks the suggestions and selects the necessary action, the device will provide a screen and link for carrying out that action, such as providing a link to purchase missing ingredients so the user can purchase them from an online shop.

[1284] Save the results:

[1285] The server records the results of the user's actions in a database and improves the accuracy of suggestions from the next time onwards, thereby making the user's life more efficient.

[1286] Specific examples

[1287] Example 1: Food suggestion implementation

[1288] A user takes a photo of a receipt from a supermarket purchase, and the device sends the image to the server.

[1289] The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[1290] The server uses a generative AI model to suggest a recipe for "tomato pasta" and suggests purchasing the missing basil.

[1291] The terminal notifies the user of the proposed content, and the user decides to purchase the basil.

[1292] The device assists with the purchase process, and the server records the updated data, allowing users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[1293] Prompt Sentence Examples

[1294] A specific example of a prompt sentence that a user might enter into a generative AI model is as follows:

[1295] Example prompt: "I took a photo of the receipt for the groceries I bought at the supermarket. Can you suggest a meal based on the ingredients I have on hand?"

[1296] By entering such prompts, the generative AI model will suggest appropriate recipes and notify you of any missing ingredients.

[1297] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1298] Step 1: User takes a photo of the certificate

[1299] User:

[1300] The user launches the dedicated app and takes a photo of the receipt using the camera on their smartphone or tablet. Specifically, they make sure the receipt fits on the screen and press the capture button.

[1301] input:

[1302] Physical voucher.

[1303] output:

[1304] A voucher image in digital image format.

[1305] Specific behavior:

[1306] The smartphone camera takes a photo of the certificate and saves the image in a dedicated app.

[1307] Step 2: Send the certificate image to the server

[1308] Device:

[1309] The terminal temporarily stores the captured certificate image. When the user checks the image and presses the "Start Analysis" button, the terminal sends the certificate image to the server.

[1310] input:

[1311] A voucher image in digital image format.

[1312] output:

[1313] The voucher image sent to the server.

[1314] Specific behavior:

[1315] Using the sending function within the dedicated app, the certificate image is uploaded to the server via the Internet.

[1316] Step 3: Analysis of the voucher image

[1317] server:

[1318] The server inputs the received certificate image into an OCR engine and extracts text information from the image.

[1319] input:

[1320] The voucher image sent to the server.

[1321] output:

[1322] Extracted text data (product name, price, quantity, purchase date, etc.).

[1323] Specific behavior:

[1324] An OCR engine (such as ABBYY FineReader or Tesseract) is used to convert the text information in the certificate image into text data.

[1325] Step 4: Store and standardize text data

[1326] server:

[1327] The server extracts product information from the extracted text data, standardizes the product name, associates it with the user's account, and stores it in a database.

[1328] input:

[1329] The extracted text data.

[1330] output:

[1331] Standardized product information stored in a database.

[1332] Specific behavior:

[1333] Product information (product name, price, quantity, purchase date, etc.) is inserted into the database using an SQL query, and the product is recognized as the same product according to product name standardization rules.

[1334] Step 5: Generate proposals

[1335] server:

[1336] The server uses the stored product information to update the user's inventory using a generative AI model and generate optimal suggestions, such as recipe suggestions based on the ingredients the user owns.

[1337] input:

[1338] Standardized product information stored in a database.

[1339] output:

[1340] Suggestions for users.

[1341] Specific behavior:

[1342] Using generative AI models (such as GPT-3 and GPT-4), it analyzes the user's inventory and generates optimal suggestions based on the user's preferences and past selection history.

[1343] Step 6: Proposal Notification

[1344] server:

[1345] The generated proposal is sent to the device for notification.

[1346] input:

[1347] The generated proposal.

[1348] output:

[1349] A notification message that is displayed on the user's terminal.

[1350] Specific behavior:

[1351] The generated suggestions are sent to the user's device using push notifications or email notifications.

[1352] Step 7: User Actions

[1353] User:

[1354] The user reviews the suggestions and selects and executes the necessary action, for example, purchasing ingredients for the suggested recipe.

[1355] input:

[1356] User selection based on suggestions.

[1357] output:

[1358] The result of the action taken (e.g., checkout).

[1359] Specific behavior:

[1360] From within the dedicated app, users can access links and follow procedures to purchase the necessary ingredients from online shops.

[1361] Step 8: Save and notify results

[1362] server:

[1363] The results of the user's actions are recorded in a database to improve the accuracy of the next suggestion, and the user is notified when the action is complete.

[1364] input:

[1365] The result of the action taken.

[1366] output:

[1367] Action results and user notification messages stored in the database.

[1368] Specific behavior:

[1369] Update the database using SQL queries to save the action results, and use the notification function to send an action completion message to the user's device.

[1370] (Application example 1)

[1371] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1372] In modern society, it remains difficult for users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations based on that information. In particular, efficiently managing ingredients and purchasing the ingredients they need requires a lot of effort, placing a heavy burden on users. For this reason, there is a need for a system that allows users to efficiently manage their owned items, receive optimal recommendations to optimize their lives, and quickly order ingredients they need through food delivery services.

[1373] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1374] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmitting means for transmitting the photographed receipt image to the server, an analyzing means for converting the received receipt image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, a notifying means for notifying the user of the generated proposals, an action processing means for receiving user actions and updating the data, and an order processing means for allowing the user to order missing ingredients via a food delivery service. This enables efficient management of the user's purchased items, optimal proposals, and rapid ordering of missing ingredients.

[1375] "Photographing means" refers to a device or method that allows a user to photograph a receipt.

[1376] "Transmission means" refers to a device or method for transmitting a captured receipt image to a server.

[1377] The "analysis means" is a device or method for converting a received receipt image into text data using optical character recognition technology.

[1378] The "storage means" refers to a device or method for extracting product information from the analyzed text data and storing it in association with the user's account.

[1379] The "proposal generating means" is a device or method for managing and proposing various products owned by a user based on the stored product information.

[1380] The "notification means" is a device or method for notifying the user of the generated proposal.

[1381] An "action processing means" is a device or method for receiving a user's action and updating data.

[1382] "Order processing means" refers to a device or method that allows a user to order missing ingredients through a food delivery service.

[1383] "Optical character recognition technology" is a technology for extracting text data from images.

[1384] This invention provides a system for enabling a user to efficiently manage the goods purchased or items owned in daily life and receive optimal suggestions. This system includes a photographing means, a transmitting means, an analyzing means, a storing means, a suggestion generating means, a notifying means, an action processing means, and an order processing means.

[1385] System Program

[1386] The system is programmed as follows:

[1387] First, the user takes a photo of the receipt using a device such as a smartphone or tablet. The image capturing unit captures this receipt image and temporarily stores it on the device. The user checks the captured image, and if there are no particular problems, they press the "Start Analysis" button, which causes the receipt image to be sent to the server by the transmission unit.

[1388] The server uses optical character recognition (OCR) technology to analyze the received receipt image and extract text data from the image. This analysis is performed using software such as PIL (Python Imaging Library) and Pytesseract. Product information is extracted from the analyzed text data and stored in a database in association with the user's account using a storage method. This stored data includes the product name, price, quantity, purchase date, etc.

[1389] The proposal generation means updates the management information for various products owned by the user based on the saved data and generates optimal proposals. For example, for food, it suggests recipes based on the ingredients possessed. It also suggests purchasing ingredients that are in short supply. A generative AI model is used to make proposals based on the user's purchase history and preferences.

[1390] The notification means notifies the user's device of the suggested recipe and information about missing ingredients. The user can receive the notification and check the suggested content. For example, a notification saying "Basil is needed to make tomato pasta" may be displayed.

[1391] When a user takes some action, for example, when ordering missing ingredients, the order processing means comes into play. The user can order the ingredients they need through an order link linked to a food delivery service. Once the order is confirmed, the data is updated in the server's database.

[1392] Hardware and software used

[1393] The hardware used includes smartphones and tablets, which act as the image capture and transmission devices. The server handles optical character recognition analysis, data storage, suggestion generation, notifications, and order processing. The software used includes PIL, Pytesseract, and a REST API.

[1394] Specific examples

[1395] A specific example is when a user buys "tomatoes, pasta, and cheese" at a supermarket and then takes a photo of the receipt and analyzes it. After analysis, "tomato pasta" is suggested as a recipe, but the necessary "basil" is missing, so a link to purchase basil via food delivery is provided. Through this link, the user can quickly order the missing ingredients.

[1396] Prompt Sentence Examples

[1397] "Please take a photo of the receipt for the items purchased at the supermarket and analyze it using OCR. Based on the analyzed content, please suggest the most suitable recipe based on the stored ingredient information, and allow the user to order any missing ingredients through food delivery. For example, if a user purchases 'tomatoes, pasta, and cheese,' the system will suggest a recipe for 'tomato pasta,' and allow the user to order the necessary 'basil.'"

[1398] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1399] Step 1:

[1400] The user takes a photo of the receipt they received at the supermarket using their smartphone. The input is the receipt image, and the output is image data temporarily saved on the smartphone. This operation allows the user to digitally record the product information they purchased.

[1401] Step 2:

[1402] The terminal checks the captured receipt image, and when the user presses the "Start analysis" button, the terminal uses a transmission means to send the receipt image to the server. The input is the temporarily saved receipt image data, and the output is the image data sent to the server via the Internet.

[1403] Step 3:

[1404] The server receives the sent receipt image and analyzes it using optical character recognition (OCR) technology. The input is the receipt image data, and the output is the extracted text data. The server uses PIL and Pytesseract to extract text information such as product name, price, quantity, and purchase date from the image data.

[1405] Step 4:

[1406] The server extracts product information from the parsed text data and stores it in a database. The input is the text data obtained by OCR analysis, and the output is product information stored in association with the user's account. This data includes the product name, price, quantity, purchase date, etc.

[1407] Step 5:

[1408] Based on the stored product information, the server uses a generative AI model to generate suggestions. The input is the product information stored in the database, and the output is suggestions based on the user's preferences and purchase history. For example, it can suggest recipes based on ingredients the user has on hand and suggest purchasing ingredients they are running low on.

[1409] Step 6:

[1410] The server notifies the user's smartphone of the proposed content using a notification means. The input is the generated proposal content, and the output is a proposal notification displayed on the user's device. For example, a notification saying "Basil is needed to make tomato pasta" is displayed.

[1411] Step 7:

[1412] The user checks the notification and, if there are any missing ingredients, places an order through a food delivery service via their smartphone. The input is information about the missing ingredients, and the output is an order link generated by the order processing means and the food order via that link.

[1413] Step 8:

[1414] The server receives the user's completed order and updates the database. The input is the user's order information obtained through the order processing means, and the output is the updated property information and data to be reflected in the next proposal.

[1415] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1416] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[1417] System Configuration

[1418] The system includes the following main components:

[1419] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[1420] 2. Transmission method: The captured receipt image is sent to the server.

[1421] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[1422] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[1423] 5. Proposal generation means: Based on the stored product information, manages the user's belongings and generates various proposals.

[1424] 6. Emotion engine: Recognizes and analyzes user emotions and adjusts suggestions accordingly.

[1425] 7. Notification means: The generated proposal is notified to the user's terminal.

[1426] 8. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[1427] Program processing and specific examples

[1428] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[1429] Capture and send receipt

[1430] User:

[1431] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[1432] Device:

[1433] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[1434] Receipt image analysis and product data extraction

[1435] server:

[1436] The server passes the received receipt image to an OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[1437] Suggestion generation and emotion regulation

[1438] server:

[1439] Based on the saved data, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed or in short supply and suggests purchases.

[1440] Emotion Engine:

[1441] The emotion engine analyzes the user's voice input and facial expression data and compares it with past emotional data. For example, it recognizes emotions from facial expressions and voice when the user takes a photo of a receipt with the camera and adjusts the suggestions based on that emotional state.

[1442] Proposal generation and distribution

[1443] server:

[1444] Based on the user's profile, past data, and the results of analysis by the emotion engine, the system generates suggestions for flea market prices and new fashions. It also compares prices for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[1445] Distributing proposals and taking action

[1446] server:

[1447] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[1448] Device:

[1449] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[1450] Executing a user action

[1451] User:

[1452] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[1453] Device:

[1454] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[1455] Save and notify results

[1456] server:

[1457] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[1458] Device:

[1459] The user is notified that the process is complete. For example, a notification saying "Basil purchase complete" is displayed on the device. This allows the user to proceed to the next step.

[1460] Specific examples

[1461] Example 1: Food suggestion and emotion regulation implementation

[1462] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[1463] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[1464] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. If the user is tired, it will prioritize suggesting easy-to-make recipes.

[1465] 4. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[1466] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[1467] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[1468] In this way, more personalized services can be provided by making suggestions that take the user's emotions into consideration.

[1469] The processing flow will be explained below.

[1470] Step 1:

[1471] User:

[1472] The user launches the app, uses the camera to take a photo of the receipt, adjusts the image so that the entire receipt fits in the frame, and presses the capture button.

[1473] Step 2:

[1474] Device:

[1475] The device displays a preview of the captured receipt image to the user. When the user confirms the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[1476] Step 3:

[1477] server:

[1478] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[1479] Step 4:

[1480] server:

[1481] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[1482] Step 5:

[1483] server:

[1484] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[1485] Step 6:

[1486] server:

[1487] Based on the saved product information, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed and those in short supply and suggests purchases.

[1488] Step 7:

[1489] Emotion Engine:

[1490] The emotion engine analyzes voice input and facial expressions while the user is using the app, and recognizes emotions in real time. The recognized emotion data is sent to the server.

[1491] Step 8:

[1492] server:

[1493] The server receives emotional data from the emotion engine and adjusts the suggestions accordingly. For example, if the user is tired, it will prioritize easy-to-make recipes.

[1494] Step 9:

[1495] server:

[1496] Based on the user's profile, past data, and the results of the emotion engine analysis, the system generates suggestions for flea market prices and new fashions. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[1497] Step 10:

[1498] server:

[1499] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[1500] Step 11:

[1501] Device:

[1502] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[1503] Step 12:

[1504] User:

[1505] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[1506] Step 13:

[1507] Device:

[1508] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[1509] Step 14:

[1510] server:

[1511] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[1512] Step 15:

[1513] Device:

[1514] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[1515] Example 2

[1516] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1517] Conventional systems have had difficulty effectively managing the products purchased and items owned by users and providing appropriate suggestions based on the user's situation. They also lacked the ability to provide personalized suggestions that take into account the user's emotional state. As a result, the suggestions users received were often not sufficiently useful.

[1518] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1519] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmitting means for transmitting the photographed receipt image to the server, an analyzing means for converting the received receipt image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, an emotion recognition means for analyzing the user's voice and facial expression and adjusting the content of the proposals, a notifying means for notifying the user of the generated proposals, and an action processing means for receiving user actions and updating data. This enables the user to efficiently manage the products purchased and items owned in their daily lives and receive optimal proposals according to the user's emotional state.

[1520] "Photographing means" refers to a device or function that allows a user to photograph a receipt.

[1521] "Transmission means" refers to a device or function for transmitting a captured receipt image to a server.

[1522] The term "analysis means" refers to a device or function that converts the received receipt image into text data using optical character recognition technology.

[1523] "Storage means" refers to a device or function that extracts product information from the analyzed text data and stores the information in association with the user's account.

[1524] The "proposal generating means" refers to a device or function that manages and proposes various products owned by the user based on the stored product information.

[1525] "Emotion recognition means" refers to a device or function that analyzes the user's voice and facial expressions and adjusts the content of suggestions.

[1526] "Notification means" refers to a device or function for notifying the user of the generated suggestions.

[1527] "Action processing means" refers to a device or function that receives a user action and updates data.

[1528] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal on which a dedicated application is installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[1529] System Configuration

[1530] The system includes the following main components:

[1531] Filming method

[1532] Users take a photo of the receipt using a dedicated application, which utilizes the camera function of their smartphone or tablet.

[1533] Transmission method

[1534] The device then sends the captured receipt image to a server via the internet, typically using Wi-Fi or mobile data.

[1535] Analysis means

[1536] The server passes the received receipt image to an optical character recognition (OCR) engine to extract text data from the image, using commonly available commercial or open source software (e.g., Tesseract OCR).

[1537] Preservation means

[1538] The server extracts product information from the analyzed text data, associates it with the user's account, and stores it in a database. The database management system can be MySQL or PostgreSQL.

[1539] Proposal generation means

[1540] The server manages the various products owned by the user based on the stored product information. Specifically, it suggests recipes based on the ingredients owned and suggests purchasing ingredients that are in short supply. It also manages the clothing owned, suggests flea market prices after a certain period of time, and supports the listing of items.

[1541] emotion recognition means

[1542] The emotion engine analyzes the user's voice input and facial expression data to tailor its suggestions, using machine learning models (e.g., generative AI models built using TensorFlow or PyTorch) to analyze the voice and facial expressions.

[1543] Notification means

[1544] The server notifies the user of the generated suggestions via push notification, SMS, email, or other methods.

[1545] Action Processing Means

[1546] It receives user actions (e.g., purchasing a product or putting it up for sale at a flea market), sends related data to the server, and updates the database. This means includes a communication function that receives operational inputs from the terminal and sends that information to the server.

[1547] Specific examples

[1548] Example 1: Food suggestion and emotion regulation implementation

[1549] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[1550] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[1551] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is tired, it will prioritize and suggest easy-to-make recipes.

[1552] 4. Suggestion Generation: The server suggests a recipe for "Tomato Pasta" and suggests purchasing the missing basil.

[1553] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[1554] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[1555] Example prompts for generative AI models

[1556] "Please explain the entire process of using this system, from taking a photo of a receipt for groceries purchased at a supermarket to receiving suggestions."

[1557] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1558] Step 1:

[1559] Take a photo of the receipt and confirm the input

[1560] User:

[1561] The user launches the dedicated app and uses the camera function to take a photo of the receipt.

[1562] Input: Receipt image

[1563] Device:

[1564] The device displays the captured receipt image and allows the user to check it on the preview screen. After checking it, the user taps the "Start analysis" button.

[1565] Output: Image of receipt approved by user

[1566] Step 2:

[1567] Sending images

[1568] Device:

[1569] The terminal receives a user's operation and sends the receipt image to the server, using an internet connection for image transmission.

[1570] Input: User approved receipt image

[1571] Output: Receipt image sent to the server

[1572] Step 3:

[1573] Image reception and OCR analysis

[1574] server:

[1575] The server receives the receipt image sent from the terminal and passes it to the OCR engine, which extracts the text data from the image.

[1576] Input: Receipt image

[1577] Output: Text data (product name, price, quantity, purchase date, etc.)

[1578] Step 4:

[1579] Extracting and storing product data

[1580] server:

[1581] The server analyzes the extracted text data, extracts product information, converts the data into fields such as product name, price, quantity, and purchase date, associates it with the user's account, and stores it in a database.

[1582] Input: Text data

[1583] Output: Saved product information

[1584] Step 5:

[1585] Proposal Generation

[1586] server:

[1587] The server updates the management data for the various products owned by the user based on the saved product information. It then suggests recipes based on the ingredients owned and suggests purchasing any ingredients that are lacking. For owned clothing, it also suggests a flea market price after a certain period of time has passed and supports the user in listing the items.

[1588] Input: Saved product information

[1589] Output: Suggestions (recipe suggestions, suggestions for purchasing missing ingredients, flea market suggestions, etc.)

[1590] Step 6:

[1591] Emotion recognition and suggestion adjustment

[1592] Emotion Engine:

[1593] The emotion engine analyzes the user's voice input and facial expression data to determine their current emotional state, and adjusts the suggestions accordingly. For example, if the user is tired, it will suggest easy-to-make recipes.

[1594] Input: User voice input and facial expression data

[1595] Output: Adjusted proposal

[1596] Step 7:

[1597] Proposal Notification

[1598] server:

[1599] The server sends the adjusted proposal to the user's terminal.

[1600] Input: Adjusted proposal

[1601] Output: Suggestions sent to the device

[1602] Device:

[1603] The device will notify the user of the suggestions it receives from the server via a pop-up or notification bar, and will include a link for the user to view more details.

[1604] Input: Adjusted proposal

[1605] Output: Proposal notification to user

[1606] Step 8:

[1607] Executing a user action

[1608] User:

[1609] The user checks the proposed information and selects an action to take, such as accepting the recipe suggestion and purchasing the missing ingredients.

[1610] Input: Proposal

[1611] Output: User action selection

[1612] Device:

[1613] The device responds to the user's selection and supports specific actions, such as providing information about the nearest store or a link to purchase from an online store.

[1614] Input: User action selection

[1615] Output: Action support (store information, purchase link, etc.)

[1616] Step 9:

[1617] Save and notify results

[1618] server:

[1619] Once a user completes an action, the server updates the database based on that action. For example, if a user purchases basil, that purchase is recorded.

[1620] Input: User action completion information

[1621] Output: Updated database

[1622] Device:

[1623] Notify the user that the transaction is complete. For example, a notification might say, "Your basil purchase is complete."

[1624] Input: Completion notification information from the server

[1625] Output: Notification of completion to the user

[1626] (Application example 2)

[1627] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1628] In recent years, there has been a demand for systems that can efficiently manage users' purchasing behavior and possessions in their daily lives and provide optimal suggestions to users. However, current systems lack the ability to provide suggestions that take into account the user's emotional state, and the suggestions provided are often inappropriate for the user's current situation or emotional state. This has led to problems such as users feeling indifferent or dissatisfied with the suggestions provided.

[1629] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a photographing means for the user to photograph a receipt; a transmitting means for transmitting the photographed receipt image to the server; an analyzing means for converting the received receipt image into text data using optical character recognition technology; a storing means for extracting product information from the analyzed text data and storing it in association with the user's account; a proposal generating means for managing and proposing various products owned by the user based on the stored product information; an emotion analyzing means for recognizing the user's emotional state and adjusting the content of the proposals; a notifying means for notifying the user of the generated proposals; and an action processing means for receiving user actions and updating data. This makes it possible to provide proposals tailored to the user's current emotional state, thereby realizing a more personalized service.

[1630] "User" refers to an individual or corporation that uses this system.

[1631] A "receipt" is a paper or digital data containing purchase information issued by a store when a user purchases a product.

[1632] "Photographing means" refers to a device or application with a camera function that allows a user to photograph a receipt or other object.

[1633] "Transmission means" refers to a communication function for transmitting the captured receipt image to an external system such as a server.

[1634] A "server" is a central processing system that manages, analyzes, stores, and generates proposals on data.

[1635] "Optical character recognition technology" is a technology that extracts text data from images, and is called Optical Character Recognition (OCR) in English.

[1636] "Analysis means" refers to the function of converting the received receipt image into text data using optical character recognition technology and analyzing the content.

[1637] "Text data" is character information extracted by the analysis means.

[1638] "Product information" refers to purchase data such as the product name, price, quantity, and purchase date and time listed on the receipt.

[1639] "Storage means" refers to a function for storing analyzed product information in a database and associating it with a user's account.

[1640] The "proposal generating means" is a function that manages and proposes various products owned by the user based on the stored product information.

[1641] "Emotion analysis means" refers to the function of recognizing and analyzing the user's emotional state from their voice, facial expressions, etc., and adjusting the content of suggestions based on the results.

[1642] The "notification means" is a function for notifying the user of the generated proposal.

[1643] The "action processing means" refers to a function that receives an action in response to a user's suggestion and updates data based on that action.

[1644] "Content" refers to entertainment or information media that can be viewed or subscribed to, such as movies, dramas, music, and books.

[1645] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. This system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[1646] System Configuration

[1647] The system includes the following main components:

[1648] 1. Photography Method:

[1649] The user uses a dedicated app to take a photo of the receipt using the camera function of their smartphone, smart glasses, or head-mounted display, and the captured image of the receipt is saved on the device.

[1650] 2. Means of transmission:

[1651] After the user takes a photo of the receipt, the image is sent to the server via a dedicated app, using an internet connection.

[1652] 3. Analysis method:

[1653] The server converts the received receipt image into text data using optical character recognition (OCR) technology. The OCR engine used is "Tesseract OCR."

[1654] 4. Preservation means:

[1655] Product information is extracted from the analyzed text data, associated with the user's account, and stored in a database (MongoDB).

[1656] 5. Proposal generation means:

[1657] Based on the stored product information, the system manages the user's inventory and generates tailored suggestions based on the user's emotional state, including recipe suggestions, product purchases, and entertainment content suggestions.

[1658] 6. Emotion analysis means:

[1659] The system analyzes the user's voice input and facial expression data to recognize the user's emotional state. This analysis is performed using Microsoft Azure Cognitive Services.

[1660] 7. Means of notification:

[1661] The generated suggestions are notified to the user using Firebase Cloud Messaging (FCM). Users can receive notifications via their smartphones, smart glasses, or head-mounted displays.

[1662] 8. Action Processing Means:

[1663] Receive an action from the user to accept or decline the suggestion and update the data accordingly.

[1664] Specific examples

[1665] Example: Movie proposal

[1666] 1. Shooting and sending:

[1667] The user takes a photo of the movie theater receipt with their smartphone and sends the image to the server via a dedicated app.

[1668] 2. Image Analysis:

[1669] The server uses Tesseract OCR to analyze the receipt image, extract product information such as the movie title, viewing date and time, and ticket purchase amount, and stores this information in MongoDB.

[1670] 3. Emotion recognition:

[1671] The emotion engine analyzes the user's facial expressions and voice to recognize their current emotional state. For example, the analysis may determine that the user is "tired."

[1672] 4. Proposal generation:

[1673] The server suggests relaxing movies (e.g., comedy movies) to the user based on the stored product information and emotional state.

[1674] 5. Notice and Execution:

[1675] Firebase Cloud Messaging is used to notify the user of the suggestion, and the user watches a relaxing movie.

[1676] Prompt Sentence Examples

[1677] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[1678] This system configuration and process makes it possible to provide suggestions tailored to the user's current emotional state, enabling more personalized services.

[1679] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1680] Step 1:

[1681] Capture and send receipt

[1682] After receiving a receipt at the movie theater, the user launches a dedicated app on their smartphone or smart glasses and takes a photo of the receipt. The captured image is previewed on the app, and after the user confirms it, they press the send button to send the image to the server. The input here is the captured receipt image, and the output is the image data sent to the server.

[1683] Step 2:

[1684] Receipt image analysis

[1685] The server retrieves the received receipt image and uses the OCR engine, Tesseract OCR, to extract text data from the image. The specific operation of image analysis is to read the image file, detect the text area, and convert it into text data. The input here is the receipt image data, and the output is extracted product information (movie title, viewing date and time, purchase amount, etc.).

[1686] Step 3:

[1687] Saving product information

[1688] The server organizes the product information analyzed by OCR, associates it with the user's account, and stores it in a database (MongoDB). The input here is the analyzed text data, and the output is the completed save to the database. Specifically, the product information is stored, associated with the user's account as a key.

[1689] Step 4:

[1690] Emotion analysis

[1691] While the user is taking a photo of the receipt, a dedicated app records video and audio, capturing the user's facial expressions and voice. The emotion engine (Microsoft Azure Cognitive Services) analyzes this data to recognize and determine the user's emotional state. The input here is the user's voice and facial expression data, and the output is the recognized emotional state. Specific operations involve the use of facial expression analysis algorithms and voice analysis algorithms.

[1692] Step 5:

[1693] Proposal Generation

[1694] The server generates recommendations appropriate for the user based on the stored product information and emotional state. In this case, if the user is determined to be "tired," it will suggest relaxing movies (e.g., comedy movies). The input here is product information and emotional state, and the output is a list of suggested content. Specific operations include filtering and a recommendation engine based on the emotional state.

[1695] Step 6:

[1696] Proposal Notification

[1697] The server sends the generated suggestions to the user's device using Firebase Cloud Messaging (FCM). The user receives the notification on their smartphone, smart glasses, or head-mounted display. The input here is the suggestion content, and the output is the notification sent to the device. Specifically, the notification is sent in real time via the FCM API.

[1698] Step 7:

[1699] Handling User Actions

[1700] The user receives a notification and confirms the content suggestion. If the user chooses to watch the suggested movie, the server records the action and updates the database. The input is the user's action, and the output is the updated database. Specific operations include adding a viewing record and saving it as training data for the recommendation algorithm.

[1701] Prompt Sentence Examples

[1702] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[1703] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1704] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1705] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1706] [Fourth embodiment]

[1707] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1708] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1709] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1710] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1711] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1712] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1713] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1714] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1715] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1716] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1717] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1718] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1719] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1720] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. The system includes a terminal on which a dedicated app is installed and a server connected via a network.

[1721] System Configuration

[1722] The system includes the following main components:

[1723] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[1724] 2. Transmission method: The captured receipt image is sent to the server.

[1725] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[1726] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[1727] 5. Proposal generation means: Based on the stored product information, manages the user's possessions and generates various proposals.

[1728] 6. Notification means: The generated proposal is notified to the user's terminal.

[1729] 7. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[1730] Program processing and specific examples

[1731] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[1732] Capture and send receipt

[1733] User:

[1734] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[1735] Device:

[1736] The device temporarily saves the captured receipt image. The user checks the image and, if there are no problems, presses the "Start Analysis" button. The device then sends the receipt image to the server.

[1737] Receipt image analysis and product data extraction

[1738] server:

[1739] The server passes the received receipt image to the OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[1740] Proposal generation and distribution

[1741] server:

[1742] The system updates the management data for various products owned by the user based on the saved data. For example, for food, it generates recipe suggestions based on the ingredients owned and suggests purchasing the necessary ingredients. For clothing, it suggests flea market prices and new fashions. For household goods, it compares prices of detergents and beauty products and suggests the lowest prices.

[1743] Device:

[1744] Once a proposal is generated, the server notifies the user's device, which then displays the proposal to the user for further confirmation.

[1745] Executing a user action

[1746] User:

[1747] The user can check the suggestions and take action such as creating a recipe, purchasing missing ingredients, listing clothing, purchasing daily necessities, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy the basil they are running low on.

[1748] Device:

[1749] Depending on the action selected by the user, the device will display an appropriate screen, such as supporting the process of selling clothes at a flea market, or providing links to purchase ingredients that are in short supply.

[1750] Save and notify results

[1751] server:

[1752] Once the user completes their action, the server updates the data based on that behavior and records it to improve the accuracy of future suggestions.

[1753] Device:

[1754] The user is notified that the transaction is complete. For example, a notification saying "Basil purchase complete" is displayed on the device.

[1755] Specific examples

[1756] Example 1: Food suggestion implementation

[1757] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[1758] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[1759] 3. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[1760] 4. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[1761] 5. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[1762] In this way, a system is realized that allows users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[1763] The processing flow will be explained below.

[1764] Step 1:

[1765] User:

[1766] The user launches the app, takes a photo of the receipt using the camera, presses the capture button, and adjusts the image so that the entire receipt is visible.

[1767] Step 2:

[1768] Device:

[1769] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[1770] Step 3:

[1771] server:

[1772] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[1773] Step 4:

[1774] server:

[1775] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[1776] Step 5:

[1777] server:

[1778] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[1779] Step 6:

[1780] server:

[1781] The system updates inventory management data based on the saved product information. For example, for food, it generates recipe suggestions based on the ingredients you have. It also creates a list of ingredients you need or are lacking and suggests purchases.

[1782] Step 7:

[1783] server:

[1784] Based on the user's profile and past data, the system generates flea market price suggestions and new product recommendations. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[1785] Step 8:

[1786] server:

[1787] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products.

[1788] Step 9:

[1789] Device:

[1790] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[1791] Step 10:

[1792] User:

[1793] After checking the suggested content, the user selects an action such as creating a recipe, purchasing ingredients, listing clothing, purchasing a product, etc. For example, if the user accepts the recipe suggestion, they select "View Recipe."

[1794] Step 11:

[1795] Device:

[1796] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[1797] Step 12:

[1798] server:

[1799] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[1800] Step 13:

[1801] Device:

[1802] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[1803] Example 1

[1804] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1805] In conventional item management systems, it was difficult to efficiently manage information about items owned by users. Furthermore, the technology for making optimal suggestions to users was insufficient, so users had to search for and select information themselves, which resulted in issues such as reduced user convenience and complicated item management.

[1806] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1807] In this invention, the server includes a photographing means for a user to photograph a certificate, a transmitting means for transmitting the photographed certificate image to an information processing device, an analyzing means for converting the received certificate image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing the information in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, a notifying means for notifying the user of the generated proposals, and an action processing means for executing an action selected by the user based on the proposal received and updating the data. This allows the user to automatically manage product information and receive optimal proposals simply by photographing the certificate.

[1808] "Photographing means" refers to a camera function that allows a user to photograph a document (such as a receipt) and an application function that supports such photographing.

[1809] "Transmission means" refers to a communication function or protocol for transmitting the photographed certificate image to an information processing device (such as a server).

[1810] "Analysis means" refers to software or algorithms for converting received certificate images into text data using optical character recognition (OCR) technology.

[1811] "Storage means" refers to a database or storage function for storing product information extracted from the analyzed text data in association with the user's account.

[1812] "Proposal generation means" refers to an algorithm or generative AI model for managing various products owned by a user and generating optimal proposals based on stored product information.

[1813] "Notification means" refers to a notification function (e.g., push notification or email notification) or its implementation for notifying the user of the generated proposal.

[1814] "Action processing means" refers to a processing function for executing an action selected by the user based on the proposal received and for reflecting the result in the database.

[1815] "Voucher" refers to a document issued as a result of a transaction (e.g., receipt, invoice).

[1816] "Information processing device" refers to a computer system or server for receiving, analyzing, storing data, and generating suggestions.

[1817] "Optical character recognition technology (OCR)" refers to technology for extracting character information from an image and converting it into text data.

[1818] "Generative AI model" refers to an artificial intelligence model (e.g., GPT-3, GPT-4) that generates optimal suggestions based on user data.

[1819] MODE FOR CARRYING OUT THE INVENTION

[1820] System Configuration

[1821] This invention is a system that allows users to photograph documents (receipts, etc.) and efficiently manage and display the information. The system includes a terminal with a dedicated app installed and an information processing device (server) connected via a network.

[1822] Hardware and Software

[1823] Device:

[1824] The device is a mobile device such as a smartphone or tablet that has a camera and internet connection, and a dedicated app installed.

[1825] server:

[1826] A server is a computer system equipped with a high-performance processor and a large amount of memory. The following software is required for a server:

[1827] OCR engine: Optical character recognition software such as ABBYY FineReader or Tesseract.

[1828] Database: A relational database management system such as MySQL or PostgreSQL.

[1829] Generative AI models: Use artificial intelligence models such as GPT-3 and GPT-4.

[1830] Explaining program processing in natural language

[1831] The user takes a photo of the certificate for an item purchased in daily life using a dedicated app. During this process, the image of the certificate taken by the user is temporarily saved in the device.

[1832] Transmission method:

[1833] When the user presses the "Start Analysis" button, the device sends the certificate image to the server via an internet connection.

[1834] Analysis method:

[1835] The server passes the received certificate image to an OCR engine, extracts the text information from the image, and converts the image into text data using ABBYY FineReader or Tesseract.

[1836] Storage method:

[1837] The server extracts product names, prices, quantities, purchase dates, etc. from the analyzed text data and stores them in a database. At this time, product names are standardized to accommodate differences in notation at different stores.

[1838] Suggestion generation means:

[1839] The server updates the user's inventory based on the stored product data. For food, it generates recipe suggestions based on the ingredients the user has in their possession. It uses a generative AI model to suggest optimal recipes based on the user's preferences and past selection history.

[1840] For example, if a user purchases tomatoes, pasta, and cheese at a supermarket, the server will suggest a recipe for "tomato pasta" based on this information and suggest purchasing any basil that is in short supply.

[1841] Means of notification:

[1842] Once a suggestion is generated, the server notifies the device of its contents, and upon receiving the notification, the device displays the suggestion to the user.

[1843] Action processing method:

[1844] After the user checks the suggestions and selects the necessary action, the device will provide a screen and link for carrying out that action, such as providing a link to purchase missing ingredients so the user can purchase them from an online shop.

[1845] Save the results:

[1846] The server records the results of the user's actions in a database and improves the accuracy of suggestions from the next time onwards, thereby making the user's life more efficient.

[1847] Specific examples

[1848] Example 1: Food suggestion implementation

[1849] A user takes a photo of a receipt from a supermarket purchase, and the device sends the image to the server.

[1850] The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[1851] The server uses a generative AI model to suggest a recipe for "tomato pasta" and suggests purchasing the missing basil.

[1852] The terminal notifies the user of the proposed content, and the user decides to purchase the basil.

[1853] The device assists with the purchase process, and the server records the updated data, allowing users to efficiently manage their belongings and receive necessary suggestions to optimize their lives.

[1854] Prompt Sentence Examples

[1855] A specific example of a prompt sentence that a user might enter into a generative AI model is as follows:

[1856] Example prompt: "I took a photo of the receipt for the groceries I bought at the supermarket. Can you suggest a meal based on the ingredients I have on hand?"

[1857] By entering such prompts, the generative AI model will suggest appropriate recipes and notify you of any missing ingredients.

[1858] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1859] Step 1: User takes a photo of the certificate

[1860] User:

[1861] The user launches the dedicated app and takes a photo of the receipt using the camera on their smartphone or tablet. Specifically, they make sure the receipt fits on the screen and press the capture button.

[1862] input:

[1863] Physical voucher.

[1864] output:

[1865] A voucher image in digital image format.

[1866] Specific behavior:

[1867] The smartphone camera takes a photo of the certificate and saves the image in a dedicated app.

[1868] Step 2: Send the certificate image to the server

[1869] Device:

[1870] The terminal temporarily stores the captured certificate image. When the user checks the image and presses the "Start Analysis" button, the terminal sends the certificate image to the server.

[1871] input:

[1872] A voucher image in digital image format.

[1873] output:

[1874] The voucher image sent to the server.

[1875] Specific behavior:

[1876] Using the sending function within the dedicated app, the certificate image is uploaded to the server via the Internet.

[1877] Step 3: Analysis of the voucher image

[1878] server:

[1879] The server inputs the received certificate image into an OCR engine and extracts text information from the image.

[1880] input:

[1881] The voucher image sent to the server.

[1882] output:

[1883] Extracted text data (product name, price, quantity, purchase date, etc.).

[1884] Specific behavior:

[1885] An OCR engine (such as ABBYY FineReader or Tesseract) is used to convert the text information in the certificate image into text data.

[1886] Step 4: Store and standardize text data

[1887] server:

[1888] The server extracts product information from the extracted text data, standardizes the product name, associates it with the user's account, and stores it in a database.

[1889] input:

[1890] The extracted text data.

[1891] output:

[1892] Standardized product information stored in a database.

[1893] Specific behavior:

[1894] Product information (product name, price, quantity, purchase date, etc.) is inserted into the database using an SQL query, and the product is recognized as the same product according to product name standardization rules.

[1895] Step 5: Generate proposals

[1896] server:

[1897] The server uses the stored product information to update the user's inventory using a generative AI model and generate optimal suggestions, such as recipe suggestions based on the ingredients the user owns.

[1898] input:

[1899] Standardized product information stored in a database.

[1900] output:

[1901] Suggestions for users.

[1902] Specific behavior:

[1903] Using generative AI models (such as GPT-3 and GPT-4), it analyzes the user's inventory and generates optimal suggestions based on the user's preferences and past selection history.

[1904] Step 6: Proposal Notification

[1905] server:

[1906] The generated proposal is sent to the device for notification.

[1907] input:

[1908] The generated proposal.

[1909] output:

[1910] A notification message that is displayed on the user's terminal.

[1911] Specific behavior:

[1912] The generated suggestions are sent to the user's device using push notifications or email notifications.

[1913] Step 7: User Actions

[1914] User:

[1915] The user reviews the suggestions and selects and executes the necessary action, for example, purchasing ingredients for the suggested recipe.

[1916] input:

[1917] User selection based on suggestions.

[1918] output:

[1919] The result of the action taken (e.g., checkout).

[1920] Specific behavior:

[1921] From within the dedicated app, users can access links and follow procedures to purchase the necessary ingredients from online shops.

[1922] Step 8: Save and notify results

[1923] server:

[1924] The results of the user's actions are recorded in a database to improve the accuracy of the next suggestion, and the user is notified when the action is complete.

[1925] input:

[1926] The result of the action taken.

[1927] output:

[1928] Action results and user notification messages stored in the database.

[1929] Specific behavior:

[1930] Update the database using SQL queries to save the action results, and use the notification function to send an action completion message to the user's device.

[1931] (Application example 1)

[1932] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1933] In modern society, it remains difficult for users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations based on that information. In particular, efficiently managing ingredients and purchasing the ingredients they need requires a lot of effort, placing a heavy burden on users. For this reason, there is a need for a system that allows users to efficiently manage their owned items, receive optimal recommendations to optimize their lives, and quickly order ingredients they need through food delivery services.

[1934] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1935] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmitting means for transmitting the photographed receipt image to the server, an analyzing means for converting the received receipt image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, a notifying means for notifying the user of the generated proposals, an action processing means for receiving user actions and updating the data, and an order processing means for allowing the user to order missing ingredients via a food delivery service. This enables efficient management of the user's purchased items, optimal proposals, and rapid ordering of missing ingredients.

[1936] "Photographing means" refers to a device or method that allows a user to photograph a receipt.

[1937] "Transmission means" refers to a device or method for transmitting a captured receipt image to a server.

[1938] The "analysis means" is a device or method for converting a received receipt image into text data using optical character recognition technology.

[1939] The "storage means" refers to a device or method for extracting product information from the analyzed text data and storing it in association with the user's account.

[1940] The "proposal generating means" is a device or method for managing and proposing various products owned by a user based on the stored product information.

[1941] The "notification means" is a device or method for notifying the user of the generated proposal.

[1942] An "action processing means" is a device or method for receiving a user's action and updating data.

[1943] "Order processing means" refers to a device or method that allows a user to order missing ingredients through a food delivery service.

[1944] "Optical character recognition technology" is a technology for extracting text data from images.

[1945] This invention provides a system for enabling a user to efficiently manage the goods purchased or items owned in daily life and receive optimal suggestions. This system includes a photographing means, a transmitting means, an analyzing means, a storing means, a suggestion generating means, a notifying means, an action processing means, and an order processing means.

[1946] System Program

[1947] The system is programmed as follows:

[1948] First, the user takes a photo of the receipt using a device such as a smartphone or tablet. The image capturing unit captures this receipt image and temporarily stores it on the device. The user checks the captured image, and if there are no particular problems, they press the "Start Analysis" button, which causes the receipt image to be sent to the server by the transmission unit.

[1949] The server uses optical character recognition (OCR) technology to analyze the received receipt image and extract text data from the image. This analysis is performed using software such as PIL (Python Imaging Library) and Pytesseract. Product information is extracted from the analyzed text data and stored in a database in association with the user's account using a storage method. This stored data includes the product name, price, quantity, purchase date, etc.

[1950] The proposal generation means updates the management information for various products owned by the user based on the saved data and generates optimal proposals. For example, for food, it suggests recipes based on the ingredients possessed. It also suggests purchasing ingredients that are in short supply. A generative AI model is used to make proposals based on the user's purchase history and preferences.

[1951] The notification means notifies the user's device of the suggested recipe and information about missing ingredients. The user can receive the notification and check the suggested content. For example, a notification saying "Basil is needed to make tomato pasta" may be displayed.

[1952] When a user takes some action, for example, when ordering missing ingredients, the order processing means comes into play. The user can order the ingredients they need through an order link linked to a food delivery service. Once the order is confirmed, the data is updated in the server's database.

[1953] Hardware and software used

[1954] The hardware used includes smartphones and tablets, which act as the image capture and transmission devices. The server handles optical character recognition analysis, data storage, suggestion generation, notifications, and order processing. The software used includes PIL, Pytesseract, and a REST API.

[1955] Specific examples

[1956] A specific example is when a user buys "tomatoes, pasta, and cheese" at a supermarket and then takes a photo of the receipt and analyzes it. After analysis, "tomato pasta" is suggested as a recipe, but the necessary "basil" is missing, so a link to purchase basil via food delivery is provided. Through this link, the user can quickly order the missing ingredients.

[1957] Prompt Sentence Examples

[1958] "Please take a photo of the receipt for the items purchased at the supermarket and analyze it using OCR. Based on the analyzed content, please suggest the most suitable recipe based on the stored ingredient information, and allow the user to order any missing ingredients through food delivery. For example, if a user purchases 'tomatoes, pasta, and cheese,' the system will suggest a recipe for 'tomato pasta,' and allow the user to order the necessary 'basil.'"

[1959] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1960] Step 1:

[1961] The user takes a photo of the receipt they received at the supermarket using their smartphone. The input is the receipt image, and the output is image data temporarily saved on the smartphone. This operation allows the user to digitally record the product information they purchased.

[1962] Step 2:

[1963] The terminal checks the captured receipt image, and when the user presses the "Start analysis" button, the terminal uses a transmission means to send the receipt image to the server. The input is the temporarily saved receipt image data, and the output is the image data sent to the server via the Internet.

[1964] Step 3:

[1965] The server receives the sent receipt image and analyzes it using optical character recognition (OCR) technology. The input is the receipt image data, and the output is the extracted text data. The server uses PIL and Pytesseract to extract text information such as product name, price, quantity, and purchase date from the image data.

[1966] Step 4:

[1967] The server extracts product information from the parsed text data and stores it in a database. The input is the text data obtained by OCR analysis, and the output is product information stored in association with the user's account. This data includes the product name, price, quantity, purchase date, etc.

[1968] Step 5:

[1969] Based on the stored product information, the server uses a generative AI model to generate suggestions. The input is the product information stored in the database, and the output is suggestions based on the user's preferences and purchase history. For example, it can suggest recipes based on ingredients the user has on hand and suggest purchasing ingredients they are running low on.

[1970] Step 6:

[1971] The server notifies the user's smartphone of the proposed content using a notification means. The input is the generated proposal content, and the output is a proposal notification displayed on the user's device. For example, a notification saying "Basil is needed to make tomato pasta" is displayed.

[1972] Step 7:

[1973] The user checks the notification and, if there are any missing ingredients, places an order through a food delivery service via their smartphone. The input is information about the missing ingredients, and the output is an order link generated by the order processing means and the food order via that link.

[1974] Step 8:

[1975] The server receives the user's completed order and updates the database. The input is the user's order information obtained through the order processing means, and the output is the updated property information and data to be reflected in the next proposal.

[1976] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1977] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[1978] System Configuration

[1979] The system includes the following main components:

[1980] 1. Photo capture method: The user takes a photo of the receipt using a dedicated app.

[1981] 2. Transmission method: The captured receipt image is sent to the server.

[1982] 3. Analysis method: The server analyzes the received receipt image using optical character recognition (OCR) technology and converts it into text data.

[1983] 4. Storage method: Product information is extracted from the analyzed text data and stored in a database.

[1984] 5. Proposal generation means: Based on the stored product information, manages the user's belongings and generates various proposals.

[1985] 6. Emotion engine: Recognizes and analyzes user emotions and adjusts suggestions accordingly.

[1986] 7. Notification means: The generated proposal is notified to the user's terminal.

[1987] 8. Action processing means: Supports the process when the user takes action based on the suggestions and updates the data.

[1988] Program processing and specific examples

[1989] Below, the processing of the system's programs will be explained in natural language, with concrete examples.

[1990] Capture and send receipt

[1991] User:

[1992] The user purchases ingredients at a supermarket, receives a receipt at the cash register, then launches the app and uses the camera to take a photo of the receipt.

[1993] Device:

[1994] The device checks the captured receipt image and displays a preview to the user. When the user checks the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[1995] Receipt image analysis and product data extraction

[1996] server:

[1997] The server passes the received receipt image to an OCR engine, which extracts text data from the image. Information such as product name, price, quantity, and purchase date is then identified from the text data and stored in a database. At this time, product names are standardized to accommodate differences in how they are written at different stores.

[1998] Suggestion generation and emotion regulation

[1999] server:

[2000] Based on the saved data, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed or in short supply and suggests purchases.

[2001] Emotion Engine:

[2002] The emotion engine analyzes the user's voice input and facial expression data and compares it with past emotional data. For example, it recognizes emotions from facial expressions and voice when the user takes a photo of a receipt with the camera and adjusts the suggestions based on that emotional state.

[2003] Proposal generation and distribution

[2004] server:

[2005] Based on the user's profile, past data, and the results of analysis by the emotion engine, the system generates suggestions for flea market prices and new fashions. It also compares prices for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[2006] Distributing proposals and taking action

[2007] server:

[2008] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[2009] Device:

[2010] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[2011] Executing a user action

[2012] User:

[2013] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[2014] Device:

[2015] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[2016] Save and notify results

[2017] server:

[2018] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[2019] Device:

[2020] The user is notified that the process is complete. For example, a notification saying "Basil purchase complete" is displayed on the device. This allows the user to proceed to the next step.

[2021] Specific examples

[2022] Example 1: Food suggestion and emotion regulation implementation

[2023] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[2024] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[2025] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. If the user is tired, it will prioritize suggesting easy-to-make recipes.

[2026] 4. Suggestion generation: The server suggests a recipe for "tomato pasta" and suggests purchasing the missing basil.

[2027] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[2028] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[2029] In this way, more personalized services can be provided by making suggestions that take the user's emotions into consideration.

[2030] The processing flow will be explained below.

[2031] Step 1:

[2032] User:

[2033] The user launches the app, uses the camera to take a photo of the receipt, adjusts the image so that the entire receipt fits in the frame, and presses the capture button.

[2034] Step 2:

[2035] Device:

[2036] The device displays a preview of the captured receipt image to the user. When the user confirms the preview and presses the "Start analysis" button, the device sends the receipt image to the server.

[2037] Step 3:

[2038] server:

[2039] The server passes the received receipt image to the OCR engine, which analyzes the character data in the image and converts the character information on the receipt into text data.

[2040] Step 4:

[2041] server:

[2042] The server extracts product names, prices, quantities, purchase dates, etc. from the text data obtained through OCR processing. Product names are standardized to eliminate differences in how they are written at different stores. The extracted information is organized as product data.

[2043] Step 5:

[2044] server:

[2045] The server stores the organized product data in a database in association with the user's account, making this stored information available for future reference by the user.

[2046] Step 6:

[2047] server:

[2048] Based on the saved product information, the system updates the management data for various products owned by the user. For example, for food, it generates recipe suggestions based on the ingredients in the user's possession. It also creates a list of ingredients needed and those in short supply and suggests purchases.

[2049] Step 7:

[2050] Emotion Engine:

[2051] The emotion engine analyzes voice input and facial expressions while the user is using the app, and recognizes emotions in real time. The recognized emotion data is sent to the server.

[2052] Step 8:

[2053] server:

[2054] The server receives emotional data from the emotion engine and adjusts the suggestions accordingly. For example, if the user is tired, it will prioritize easy-to-make recipes.

[2055] Step 9:

[2056] server:

[2057] Based on the user's profile, past data, and the results of the emotion engine analysis, the system generates suggestions for flea market prices and new fashions. It also performs price comparisons for everyday items and suggests the lowest prices at nearby stores and online stores. These suggestions are compiled as suggestion data.

[2058] Step 10:

[2059] server:

[2060] The generated suggestion data is sent to the user's device. The suggestion data includes recipe suggestions, links to purchase missing ingredients, recommendations for items to sell at flea markets, and price comparison data for products. It also includes notifications and reminders that correspond to the user's current emotional state based on the analysis results of the emotion engine.

[2061] Step 11:

[2062] Device:

[2063] The device analyzes the received suggestion data and notifies the user via a pop-up or notification bar, and includes a link where the user can view more information.

[2064] Step 12:

[2065] User:

[2066] After checking the suggestions, users can select actions such as creating a recipe, purchasing ingredients, listing clothing, purchasing products, etc. For example, if a user receives a recipe suggestion for "tomato pasta," they can go buy basil, which they are running low on.

[2067] Step 13:

[2068] Device:

[2069] Depending on the user's selection, the device will support specific actions. For example, if the user wants to purchase ingredients, the device will provide links to the nearest store or online store. It also supports the process of selling the item at a flea market.

[2070] Step 14:

[2071] server:

[2072] Once a user completes an action, the server updates the data based on that action. For example, if a user purchases basil, the purchase is added to the database.

[2073] Step 15:

[2074] Device:

[2075] Inform the user that the action was successfully completed, for example, by displaying a message such as "Basil purchase complete," allowing the user to proceed to the next step.

[2076] Example 2

[2077] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2078] Conventional systems have had difficulty effectively managing the products purchased and items owned by users and providing appropriate suggestions based on the user's situation. They also lacked the ability to provide personalized suggestions that take into account the user's emotional state. As a result, the suggestions users received were often not sufficiently useful.

[2079] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2080] In this invention, the server includes a photographing means for the user to photograph a receipt, a transmitting means for transmitting the photographed receipt image to the server, an analyzing means for converting the received receipt image into text data using optical character recognition technology, a storing means for extracting product information from the analyzed text data and storing it in association with the user's account, a proposal generating means for managing and proposing various products owned by the user based on the stored product information, an emotion recognition means for analyzing the user's voice and facial expression and adjusting the content of the proposals, a notifying means for notifying the user of the generated proposals, and an action processing means for receiving user actions and updating data. This enables the user to efficiently manage the products purchased and items owned in their daily lives and receive optimal proposals according to the user's emotional state.

[2081] "Photographing means" refers to a device or function that allows a user to photograph a receipt.

[2082] "Transmission means" refers to a device or function for transmitting a captured receipt image to a server.

[2083] The term "analysis means" refers to a device or function that converts the received receipt image into text data using optical character recognition technology.

[2084] "Storage means" refers to a device or function that extracts product information from the analyzed text data and stores the information in association with the user's account.

[2085] The "proposal generating means" refers to a device or function that manages and proposes various products owned by the user based on the stored product information.

[2086] "Emotion recognition means" refers to a device or function that analyzes the user's voice and facial expressions and adjusts the content of suggestions.

[2087] "Notification means" refers to a device or function for notifying the user of the generated suggestions.

[2088] "Action processing means" refers to a device or function that receives a user action and updates data.

[2089] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal recommendations. The system includes a terminal on which a dedicated application is installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[2090] System Configuration

[2091] The system includes the following main components:

[2092] Filming method

[2093] Users take a photo of the receipt using a dedicated application, which utilizes the camera function of their smartphone or tablet.

[2094] Transmission method

[2095] The device then sends the captured receipt image to a server via the internet, typically using Wi-Fi or mobile data.

[2096] Analysis means

[2097] The server passes the received receipt image to an optical character recognition (OCR) engine to extract text data from the image, using commonly available commercial or open source software (e.g., Tesseract OCR).

[2098] Preservation means

[2099] The server extracts product information from the analyzed text data, associates it with the user's account, and stores it in a database. The database management system can be MySQL or PostgreSQL.

[2100] Proposal generation means

[2101] The server manages the various products owned by the user based on the stored product information. Specifically, it suggests recipes based on the ingredients owned and suggests purchasing ingredients that are in short supply. It also manages the clothing owned, suggests flea market prices after a certain period of time, and supports the listing of items.

[2102] emotion recognition means

[2103] The emotion engine analyzes the user's voice input and facial expression data to tailor its suggestions, using machine learning models (e.g., generative AI models built using TensorFlow or PyTorch) to analyze the voice and facial expressions.

[2104] Notification means

[2105] The server notifies the user of the generated suggestions via push notification, SMS, email, or other methods.

[2106] Action Processing Means

[2107] It receives user actions (e.g., purchasing a product or putting it up for sale at a flea market), sends related data to the server, and updates the database. This means includes a communication function that receives operational inputs from the terminal and sends that information to the server.

[2108] Specific examples

[2109] Example 1: Food suggestion and emotion regulation implementation

[2110] 1. Photograph and send: The user photographs a receipt from a supermarket purchase, and the device sends the image to the server.

[2111] 2. Image analysis: The server analyzes the receipt using OCR and extracts information about tomatoes, cheese, and pasta.

[2112] 3. Emotion Recognition: The emotion engine analyzes the user's emotional state from their facial expressions and voice. For example, if the user is tired, it will prioritize and suggest easy-to-make recipes.

[2113] 4. Suggestion Generation: The server suggests a recipe for "Tomato Pasta" and suggests purchasing the missing basil.

[2114] 5. Notification and execution: The device notifies the user of the proposal, and the user decides to purchase basil.

[2115] 6. Purchase procedure and completion notification: The terminal supports the purchase procedure, and the server records the updated data.

[2116] Example prompts for generative AI models

[2117] "Please explain the entire process of using this system, from taking a photo of a receipt for groceries purchased at a supermarket to receiving suggestions."

[2118] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2119] Step 1:

[2120] Take a photo of the receipt and confirm the input

[2121] User:

[2122] The user launches the dedicated app and uses the camera function to take a photo of the receipt.

[2123] Input: Receipt image

[2124] Device:

[2125] The device displays the captured receipt image and allows the user to check it on the preview screen. After checking it, the user taps the "Start analysis" button.

[2126] Output: Image of receipt approved by user

[2127] Step 2:

[2128] Sending images

[2129] Device:

[2130] The terminal receives a user's operation and sends the receipt image to the server, using an internet connection for image transmission.

[2131] Input: User approved receipt image

[2132] Output: Receipt image sent to the server

[2133] Step 3:

[2134] Image reception and OCR analysis

[2135] server:

[2136] The server receives the receipt image sent from the terminal and passes it to the OCR engine, which extracts the text data from the image.

[2137] Input: Receipt image

[2138] Output: Text data (product name, price, quantity, purchase date, etc.)

[2139] Step 4:

[2140] Extracting and storing product data

[2141] server:

[2142] The server analyzes the extracted text data, extracts product information, converts the data into fields such as product name, price, quantity, and purchase date, associates it with the user's account, and stores it in a database.

[2143] Input: Text data

[2144] Output: Saved product information

[2145] Step 5:

[2146] Proposal Generation

[2147] server:

[2148] The server updates the management data for the various products owned by the user based on the saved product information. It then suggests recipes based on the ingredients owned and suggests purchasing any ingredients that are lacking. For owned clothing, it also suggests a flea market price after a certain period of time has passed and supports the user in listing the items.

[2149] Input: Saved product information

[2150] Output: Suggestions (recipe suggestions, suggestions for purchasing missing ingredients, flea market suggestions, etc.)

[2151] Step 6:

[2152] Emotion recognition and suggestion adjustment

[2153] Emotion Engine:

[2154] The emotion engine analyzes the user's voice input and facial expression data to determine their current emotional state, and adjusts the suggestions accordingly. For example, if the user is tired, it will suggest easy-to-make recipes.

[2155] Input: User voice input and facial expression data

[2156] Output: Adjusted proposal

[2157] Step 7:

[2158] Proposal Notification

[2159] server:

[2160] The server sends the adjusted proposal to the user's terminal.

[2161] Input: Adjusted proposal

[2162] Output: Suggestions sent to the device

[2163] Device:

[2164] The device will notify the user of the suggestions it receives from the server via a pop-up or notification bar, and will include a link for the user to view more details.

[2165] Input: Adjusted proposal

[2166] Output: Proposal notification to user

[2167] Step 8:

[2168] Executing a user action

[2169] User:

[2170] The user checks the proposed information and selects an action to take, such as accepting the recipe suggestion and purchasing the missing ingredients.

[2171] Input: Proposal

[2172] Output: User action selection

[2173] Device:

[2174] The device responds to the user's selection and supports specific actions, such as providing information about the nearest store or a link to purchase from an online store.

[2175] Input: User action selection

[2176] Output: Action support (store information, purchase link, etc.)

[2177] Step 9:

[2178] Save and notify results

[2179] server:

[2180] Once a user completes an action, the server updates the database based on that action. For example, if a user purchases basil, that purchase is recorded.

[2181] Input: User action completion information

[2182] Output: Updated database

[2183] Device:

[2184] Notify the user that the transaction is complete. For example, a notification might say, "Your basil purchase is complete."

[2185] Input: Completion notification information from the server

[2186] Output: Notification of completion to the user

[2187] (Application example 2)

[2188] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2189] In recent years, there has been a demand for systems that can efficiently manage users' purchasing behavior and possessions in their daily lives and provide optimal suggestions to users. However, current systems lack the ability to provide suggestions that take into account the user's emotional state, and the suggestions provided are often inappropriate for the user's current situation or emotional state. This has led to problems such as users feeling indifferent or dissatisfied with the suggestions provided.

[2190] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: a photographing means for the user to photograph a receipt; a transmitting means for transmitting the photographed receipt image to the server; an analyzing means for converting the received receipt image into text data using optical character recognition technology; a storing means for extracting product information from the analyzed text data and storing it in association with the user's account; a proposal generating means for managing and proposing various products owned by the user based on the stored product information; an emotion analyzing means for recognizing the user's emotional state and adjusting the content of the proposals; a notifying means for notifying the user of the generated proposals; and an action processing means for receiving user actions and updating data. This makes it possible to provide proposals tailored to the user's current emotional state, thereby realizing a more personalized service.

[2191] "User" refers to an individual or corporation that uses this system.

[2192] A "receipt" is a paper or digital data containing purchase information issued by a store when a user purchases a product.

[2193] "Photographing means" refers to a device or application with a camera function that allows a user to photograph a receipt or other object.

[2194] "Transmission means" refers to a communication function for transmitting the captured receipt image to an external system such as a server.

[2195] A "server" is a central processing system that manages, analyzes, stores, and generates proposals on data.

[2196] "Optical character recognition technology" is a technology that extracts text data from images, and is called Optical Character Recognition (OCR) in English.

[2197] "Analysis means" refers to the function of converting the received receipt image into text data using optical character recognition technology and analyzing the content.

[2198] "Text data" is character information extracted by the analysis means.

[2199] "Product information" refers to purchase data such as the product name, price, quantity, and purchase date and time listed on the receipt.

[2200] "Storage means" refers to a function for storing analyzed product information in a database and associating it with a user's account.

[2201] The "proposal generating means" is a function that manages and proposes various products owned by the user based on the stored product information.

[2202] "Emotion analysis means" refers to the function of recognizing and analyzing the user's emotional state from their voice, facial expressions, etc., and adjusting the content of suggestions based on the results.

[2203] The "notification means" is a function for notifying the user of the generated proposal.

[2204] The "action processing means" refers to a function that receives an action in response to a user's suggestion and updates data based on that action.

[2205] "Content" refers to entertainment or information media that can be viewed or subscribed to, such as movies, dramas, music, and books.

[2206] This invention is a system that allows users to efficiently manage the items they purchase and own in their daily lives and receive optimal suggestions. This system includes a terminal with a dedicated app installed, a server connected via a network, and an emotion engine that recognizes and analyzes the user's emotions.

[2207] System Configuration

[2208] The system includes the following main components:

[2209] 1. Photography Method:

[2210] The user uses a dedicated app to take a photo of the receipt using the camera function of their smartphone, smart glasses, or head-mounted display, and the captured image of the receipt is saved on the device.

[2211] 2. Means of transmission:

[2212] After the user takes a photo of the receipt, the image is sent to the server via a dedicated app, using an internet connection.

[2213] 3. Analysis method:

[2214] The server converts the received receipt image into text data using optical character recognition (OCR) technology. The OCR engine used is "Tesseract OCR."

[2215] 4. Preservation means:

[2216] Product information is extracted from the analyzed text data, associated with the user's account, and stored in a database (MongoDB).

[2217] 5. Proposal generation means:

[2218] Based on the stored product information, the system manages the user's inventory and generates tailored suggestions based on the user's emotional state, including recipe suggestions, product purchases, and entertainment content suggestions.

[2219] 6. Emotion analysis means:

[2220] The system analyzes the user's voice input and facial expression data to recognize the user's emotional state. This analysis is performed using Microsoft Azure Cognitive Services.

[2221] 7. Means of notification:

[2222] The generated suggestions are notified to the user using Firebase Cloud Messaging (FCM). Users can receive notifications via their smartphones, smart glasses, or head-mounted displays.

[2223] 8. Action Processing Means:

[2224] Receive an action from the user to accept or decline the suggestion and update the data accordingly.

[2225] Specific examples

[2226] Example: Movie proposal

[2227] 1. Shooting and sending:

[2228] The user takes a photo of the movie theater receipt with their smartphone and sends the image to the server via a dedicated app.

[2229] 2. Image Analysis:

[2230] The server uses Tesseract OCR to analyze the receipt image, extract product information such as the movie title, viewing date and time, and ticket purchase amount, and stores this information in MongoDB.

[2231] 3. Emotion recognition:

[2232] The emotion engine analyzes the user's facial expressions and voice to recognize their current emotional state. For example, the analysis may determine that the user is "tired."

[2233] 4. Proposal generation:

[2234] The server suggests relaxing movies (e.g., comedy movies) to the user based on the stored product information and emotional state.

[2235] 5. Notice and Execution:

[2236] Firebase Cloud Messaging is used to notify the user of the suggestion, and the user watches a relaxing movie.

[2237] Prompt Sentence Examples

[2238] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[2239] This system configuration and process makes it possible to provide suggestions tailored to the user's current emotional state, enabling more personalized services.

[2240] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2241] Step 1:

[2242] Capture and send receipt

[2243] After receiving a receipt at the movie theater, the user launches a dedicated app on their smartphone or smart glasses and takes a photo of the receipt. The captured image is previewed on the app, and after the user confirms it, they press the send button to send the image to the server. The input here is the captured receipt image, and the output is the image data sent to the server.

[2244] Step 2:

[2245] Receipt image analysis

[2246] The server retrieves the received receipt image and uses the OCR engine, Tesseract OCR, to extract text data from the image. The specific operation of image analysis is to read the image file, detect the text area, and convert it into text data. The input here is the receipt image data, and the output is extracted product information (movie title, viewing date and time, purchase amount, etc.).

[2247] Step 3:

[2248] Saving product information

[2249] The server organizes the product information analyzed by OCR, associates it with the user's account, and stores it in a database (MongoDB). The input here is the analyzed text data, and the output is the completed save to the database. Specifically, the product information is stored, associated with the user's account as a key.

[2250] Step 4:

[2251] Emotion analysis

[2252] While the user is taking a photo of the receipt, a dedicated app records video and audio, capturing the user's facial expressions and voice. The emotion engine (Microsoft Azure Cognitive Services) analyzes this data to recognize and determine the user's emotional state. The input here is the user's voice and facial expression data, and the output is the recognized emotional state. Specific operations involve the use of facial expression analysis algorithms and voice analysis algorithms.

[2253] Step 5:

[2254] Proposal Generation

[2255] The server generates recommendations appropriate for the user based on the stored product information and emotional state. In this case, if the user is determined to be "tired," it will suggest relaxing movies (e.g., comedy movies). The input here is product information and emotional state, and the output is a list of suggested content. Specific operations include filtering and a recommendation engine based on the emotional state.

[2256] Step 6:

[2257] Proposal Notification

[2258] The server sends the generated suggestions to the user's device using Firebase Cloud Messaging (FCM). The user receives the notification on their smartphone, smart glasses, or head-mounted display. The input here is the suggestion content, and the output is the notification sent to the device. Specifically, the notification is sent in real time via the FCM API.

[2259] Step 7:

[2260] Handling User Actions

[2261] The user receives a notification and confirms the content suggestion. If the user chooses to watch the suggested movie, the server records the action and updates the database. The input is the user's action, and the output is the updated database. Specific operations include adding a viewing record and saving it as training data for the recommendation algorithm.

[2262] Prompt Sentence Examples

[2263] The user's recent emotional state has been determined to be "Tired," and a movie theater receipt for "Action Movie" has been saved. Suggest the following relaxing comedy movie as the next piece of content to be suggested:

[2264] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2265] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2266] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2267] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2268] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2269] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2270] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2271] Human emotions are based on various balances, such as posture and blood sugar levels....

Claims

1. A photographing means for a user to photograph a receipt; a transmission means for transmitting the captured receipt image to a server; an analysis means for converting the received receipt image into text data using optical character recognition technology; a storage means for extracting product information from the analyzed text data and storing the information in association with the user's account; a proposal generation means for managing and proposing various products owned by the user based on the stored product information; a notification means for notifying the user of the generated proposal; action processing means for receiving user actions and updating data; A system including:

2. The system according to claim 1 , wherein the suggestion generating means includes means for suggesting a recipe from ingredients possessed and suggesting the purchase of ingredients that are in short supply.

3. The system according to claim 1 , wherein the suggestion generating means includes means for managing owned clothing, suggesting a flea market price after a certain period of time has passed, and supporting the listing of the clothing.

Images