system

Abstract

We provide the system. [Solution] A means for recognizing information about an item using an information acquisition device, A means of transmitting item information to a server via a network, A means of generating advice based on item information using a server, combined with the user's financial data, A means of visually displaying advice generated using augmented reality technology, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In daily life, it is difficult for consumers to make appropriate purchase decisions regarding the items they purchase, and there is a problem that it is impossible to obtain appropriate advice in real time according to individual financial situations. Also, since there is a lack of means to immediately evaluate products in commercial facilities and understand the impact on one's own financial goals, it is also a problem that undesirable consumption habits are encouraged.

Means for Solving the Problems

[0005] In this invention, an information acquisition device is used to recognize information about an item, and this information is transmitted to a server via a network to obtain detailed information about the item. Furthermore, the server uses AI generation based on the user's purchase history and budget information to create financial advice. By applying augmented reality technology, the generated advice is visually displayed on the user's smartphone, enabling immediate support for purchasing decisions. In this way, users can make purchasing choices in real time that are tailored to their financial situation, contributing to the achievement of their financial goals.

[0006] An "information acquisition device" is a device used to recognize information about an item, and includes devices such as cameras and barcode readers.

[0007] A "network" is a means of communication for sending and receiving information, and refers to the internet and other systems that transfer digital data.

[0008] A "server" is a computer system that provides specific programs or services, and is used for centralized processing and storage of information.

[0009] A "user" refers to an individual who uses the system to evaluate products and receive financial advice.

[0010] "Financial data" refers to data used to show an individual's financial situation, such as a user's purchase history, budget information, income, and expenses.

[0011] "Advice" refers to suggestions or guidelines provided to help users make decisions when purchasing a product.

[0012] Augmented reality technology is a technology that overlays digital information onto the real world, providing users with visual information by overlaying it onto camera footage. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that visualizes in real time how a product purchased by a user in a commercial facility will affect their economic situation and financial goals. This system uses a terminal equipped with a camera as an information acquisition device to identify products that the user is interested in.

[0035] First, when a user scans a product using their device's camera, the device sends product information to a server. The server retrieves detailed information related to the product from its database and combines it with the user's financial data. Using the financial data the user has authorized—for example, purchase history and a set monthly budget—the server generates personalized advice using AI that aligns with the user's current lifestyle and long-term goals.

[0036] The generated advice is sent to the device and visually presented to the user using augmented reality technology. This allows the user to intuitively understand the advice displayed next to the item. This information includes the impact of the purchase on the monthly budget and considerations for future spending plans.

[0037] As a concrete example, consider a user who is looking to buy a new home appliance and scans the product in a store. In this case, the server checks the price of the product, compares it with the user's past purchase history of similar products, and analyzes their spending patterns based on their spending categories. As a result, the server provides the user with guidelines such as "This may exceed this month's budget" or "We recommend purchasing it next month or later."

[0038] In this way, the system aims to help users make more informed purchasing decisions and improve their quality of life. Through the system, users can take appropriate purchasing actions in real time that are tailored to their own financial situation.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user scans the items they are considering purchasing in the store using the camera on their device. The device then obtains an image or barcode of the items.

[0042] Step 2:

[0043] The terminal transmits the acquired product information to the server via the network. This includes data necessary for identifying the product (such as image information and barcode data).

[0044] Step 3:

[0045] The server uses the received product information to search the database and retrieve detailed product information. This allows the server to collect information such as the product's price and detailed specifications.

[0046] Step 4:

[0047] The server retrieves purchase history and budget information that the user has previously authorized. This prepares the server to utilize detailed data such as the user's purchasing patterns and monthly spending trends.

[0048] Step 5:

[0049] The server uses AI generation to analyze the impact of product purchases on the user's financial situation and generate personalized advice. For example, it can generate suggestions regarding the possibility of exceeding the budget or the timing of purchases.

[0050] Step 6:

[0051] The server sends the generated advice to the terminal.

[0052] Step 7:

[0053] The device uses the received advice information and displays the advice on the camera image using augmented reality technology. Users can visually confirm advice related to the product.

[0054] Step 8:

[0055] Users make purchasing decisions based on the advice provided. They choose to either postpone the purchase or proceed with the purchase based on the advice.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] A problem exists when consumers make purchasing decisions that are in line with their own economic situation and financial goals. Specifically, there are insufficient means to immediately understand how the cost of a desired purchase will affect their budget. As a result, consumers often cannot avoid overspending, which frequently hinders the achievement of long-term goals.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for identifying products using an information acquisition device, means for transmitting product information to a data processing device via wireless communication technology, means for creating guidelines based on the product information and integrated with the user's economic data using the data processing device, and means for visually displaying the created guidelines using augmented reality technology. This enables consumers to understand their financial situation in real time and make appropriate purchasing decisions.

[0061] An "information acquisition device" is a device used to identify products and has the function of acquiring information by reading product barcodes or QR codes (registered trademarks).

[0062] "Wireless communication technology" refers to technologies used to transmit product information, such as Wi-Fi and Bluetooth, which send and receive data without using cables.

[0063] A "data processing device" is a device that performs necessary processing based on received information, such as a server or computer, and is used to analyze information and create guidelines.

[0064] "User financial data" is a general term for financial information related to individual users, and includes data necessary for economic decision-making, such as purchase history, budget information, and savings targets.

[0065] "Guidelines" are advice and guidance created to support users' purchasing activities, and they propose specific actions that are appropriate to their economic situation and goals.

[0066] Augmented reality technology is a technology that overlays digital information onto the physical real environment, merging reality and virtuality through the presentation of visual information.

[0067] This invention is a system that provides users with real-time information based on their economic situation and financial goals when they purchase goods at a commercial facility.

[0068] First, the user scans the product they are considering purchasing using an information acquisition device equipped with a camera. The terminal reads the barcode or QR code attached to the product to obtain product information and transmits that information to a server via wireless communication technology. In this case, Wi-Fi is commonly used as the wireless communication technology.

[0069] The server retrieves product details from its internal database based on the received product information. Furthermore, it analyzes the impact of the product purchase on the user's financial situation using economic data that the user has previously authorized—specifically, information such as purchase history and monthly budget. A state-of-the-art generative AI model is used to process the data. An example of a prompt given to this model would be, "Based on the user's purchase history, analyze the impact of purchasing this product on the budget and generate advice."

[0070] The server generates the guidelines, which are then transmitted wirelessly to the terminal. The terminal uses augmented reality technology to visually display the guidelines. This allows the user to intuitively understand how the guidelines are presented by visually placing the product in the image, and use this information to inform their purchasing decisions.

[0071] As a concrete example, consider a scenario where a consumer is trying to purchase a new electronic appliance. When the user scans for a product, the server checks its price and compares it with the user's purchase history and budget to generate guidelines such as, "You may have a problem with your budget next month," or "Based on your previous purchase history, we recommend buying similar products during the next sale."

[0072] This system allows users to make appropriate purchasing decisions while considering their financial situation. It also functions as an effective means of improving quality of life and preventing wasteful spending.

[0073] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0074] Step 1:

[0075] The user scans the product using a camera-equipped device. Input includes the product's barcode or QR code. The device reads it and retrieves the product information. Specifically, the camera software within the device performs barcode scanning and obtains the product identification number.

[0076] Step 2:

[0077] The terminal transmits acquired product information to the server via wireless communication technology. Wi-Fi is commonly used in this case. The input includes the product identification number, which is then forwarded to the server as output. The communication module within the terminal operates and sends the product information as data packets to the server's receiving address.

[0078] Step 3:

[0079] The server retrieves the corresponding product details from its internal database based on the received product information. The input is a product identification number, and the output includes detailed information such as the product name, price, and category. A database query is executed, and product-related data is extracted.

[0080] Step 4:

[0081] The server integrates with user-authorized economic data. Inputs include the user's purchase history and financial information, which are combined with product details. The output is an integrated dataset. Specifically, the server uses CRUD (Create, Read, Uninstall, Delete) functionality to link existing user data to product details.

[0082] Step 5:

[0083] The server utilizes a generative AI model to generate guidelines based on an integrated dataset. Product and user economic data are used as input, and personalized purchasing advice is provided as output. The generative AI model is given prompts, and the AI ​​generates corresponding guidelines through analysis.

[0084] Step 6:

[0085] The server sends the generated guidance to the terminal. Wireless communication technology is used here as well. The input includes the generated guidance, and the output is the transmission of information to the terminal. The server's transmission protocol is activated, and the generated data is sent to the terminal's receiving address.

[0086] Step 7:

[0087] The device uses augmented reality technology to visually display the received guidance to the user. The input is guidance information received from a server, and the output is displayed on the screen. AR software within the device is activated, allowing the user to view the guidance along with the product displayed through the camera.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] In today's brick-and-mortar stores, it's difficult for consumers to immediately understand how a product they're considering purchasing will impact their finances and long-term life goals. Traditionally, consumers typically consider the impact of their spending after the fact, which can lead to inappropriate spending. Therefore, there's a need for a system that allows consumers to intuitively understand the impact of a product on their finances before purchasing it.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for collecting product data, means for generating advice by combining personal accounting data and product data, and means for visually displaying the generated advice using augmented reality technology. This allows consumers to instantly understand the impact a product will have on their financial situation before purchasing it in a physical store.

[0093] An "information acquisition terminal" is a device used to recognize product data and is used by users to identify products.

[0094] A "communication network" is a network system for transmitting and receiving information, used to exchange data between information acquisition terminals and central processing units.

[0095] A "central processing unit" is a system that has processing functions to generate advice by combining product data and individual accounting data.

[0096] Augmented reality technology is a technique used to visually display generated advice by overlaying computer-generated information onto images of the real world.

[0097] "Personal consumption history" refers to records of past spending by consumers and is data used to analyze consumption trends.

[0098] A "spending plan" is a plan for future spending that consumers anticipate, and is related to budgeting and financial goals.

[0099] "Lifestyle" refers to a consumer's patterns of living and preferences, and is a factor that influences their purchasing behavior.

[0100] This invention utilizes an information acquisition terminal, a communication network, and a central processing unit. A portable device such as a smartphone or tablet is used as the information acquisition terminal, and its camera function is used to acquire product data. For example, a user can quickly obtain product information by scanning the barcode of a product they are considering purchasing in a store. This product information is transmitted to the central processing unit via the communication network.

[0101] The central processing unit combines the received product information with personal accounting data such as the user's consumption history and spending plans, and generates personalized advice using a generative AI model. In this process, the generative AI model uses machine learning models such as Hugging Face Transformers to provide advice that helps the user make the best purchase decisions.

[0102] The generated advice is visually displayed on the information acquisition terminal using augmented reality technology. By utilizing augmented reality platforms such as ARKit and ARCore, the advice is overlaid next to the product, making it easy for the user to intuitively understand. This visual display allows users to instantly recognize how the product will affect their accounting situation and future spending.

[0103] As a concrete example, suppose a user is considering purchasing a new speaker. When the user scans this speaker with their device's camera, the central processing unit receives product information and, considering past purchase history and spending plans, generates specific advice such as "It fits this month's budget" and "Purchase recommended." This generated advice is then displayed on the information acquisition device's screen using augmented reality technology.

[0104] Examples of prompts for a generative AI model include:

[0105] User's monthly budget: 50,000 yen

[0106] User's current spending: 40,000 yen

[0107] Product I'm considering buying: Speakers (¥15,000)

[0108] Planned expenses for next month and beyond: Estimates based on normal living expenses.

[0109] Please generate advice to provide to the user.

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The user scans the product using the camera function of the information acquisition terminal. The input is image data of the product acquired by the camera. The output is product identification information, obtained by analyzing the product's barcode or QR code. Image processing libraries such as OpenCV are used for the analysis.

[0113] Step 2:

[0114] The terminal transmits product identification information to the central processing unit via the communication network. The input is the product identification information obtained in step 1. The output is a database query for product information on the server side. A REST API is used for communication to retrieve detailed information related to the product from the database.

[0115] Step 3:

[0116] The server retrieves product information from the database and combines it with the user's personal accounting data (e.g., consumption history and spending plan). The input consists of product information and the user's accounting data. The output is an integrated information set. This data processing uses libraries such as Pandas for data storage and calculations.

[0117] Step 4:

[0118] Based on the integrated information set obtained by the server, a generative AI model is used to generate personalized advice. The input is the integrated information set obtained in step 3. The output is personalized advice. Hugging Face Transformers are used as the generative AI model, and the machine learning model generates advice based on pre-configured prompt sentences.

[0119] Step 5:

[0120] The server sends the generated advice to the device. The input is the advice generated in step 4. The output is a visual representation using augmented reality technology. The REST API is used again to send data from the server, and ARKit or ARCore is used to overlay the advice onto the device's camera feed.

[0121] Step 6:

[0122] The user views advice overlaid next to the product on their device screen. The device utilizes augmented reality (AR) to intuitively display the generated advice. The input is visual data received from the server. The output is visual information of the advice organized in a way that is easy for the user to understand. This display allows the user to immediately grasp the impact the product will have on their financial situation.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention is a system that incorporates an emotion engine to recognize user emotions in order to support purchasing decisions for goods. When a user selects products in a store, product information is captured using a terminal. In this process, the terminal uses a camera function as an information acquisition device.

[0125] When a user scans a product selected through the camera, the device acquires the product information and transmits it to a server via the network. The server receives the product information and retrieves detailed information related to the product from its database. Furthermore, with the user's permission, the server collects the user's financial data based on purchase history and budget information.

[0126] This system incorporates an emotion engine, which allows the device to recognize emotions from the user's facial expressions and voice. This emotion data is sent to a server, which uses the emotion engine to analyze the user's emotional state. Based on the emotional state, the server uses generative AI to enhance personalized advice. In particular, if the user is feeling stressed, for example, it will generate advice encouraging them to reconsider their purchase.

[0127] The generated advice is sent from the server to the terminal, which uses augmented reality technology to visually overlay the advice next to the product the user is viewing. This allows the user to make purchasing decisions while simultaneously considering their emotions and financial situation at that moment. For example, when a user is about to purchase an expensive electronic product, the emotion engine may detect the user's state of excitement and provide advice to encourage a calmer decision.

[0128] Thus, the present invention is designed to enable users to make more balanced purchasing decisions by integrating user emotions and economic information to support product purchases. The system aims to improve the user's quality of life by providing real-time emotional and financial advice.

[0129] The following describes the processing flow.

[0130] Step 1:

[0131] The user scans the product they are considering purchasing in the store using the terminal's camera. The terminal acquires the product image or barcode and processes this information as digital data.

[0132] Step 2:

[0133] The terminal transmits the acquired product information to the server via the network. This product information includes the product ID and category information.

[0134] Step 3:

[0135] Based on the received product information, the server retrieves detailed information about the product from the database. This includes the product's price, brand, and reviews.

[0136] Step 4:

[0137] The server accesses the user's purchase history and budget information to retrieve their financial data. This data collection is conducted only within the scope of the user's prior permission.

[0138] Step 5:

[0139] The device uses a built-in emotion engine to recognize the user's emotions from their facial expressions and voice. This emotion data is sent from the device to a server, where it is used to analyze the user's current emotional state as digital data.

[0140] Step 6:

[0141] The server integrates emotional and financial data and uses generative AI to generate personalized advice for the user. For example, if a user is excited and considering purchasing an expensive item, a message encouraging calmness will be generated.

[0142] Step 7:

[0143] The generated advice is sent from the server to the terminal, and the terminal displays this advice overlaid on the camera image using AR technology.

[0144] Step 8:

[0145] Users use the provided advice to decide whether or not to continue purchasing the product. This advice includes recommendations based on their emotions and financial situation.

[0146] (Example 2)

[0147] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0148] Modern consumers must make purchasing decisions from a wide range of options, and emotions and economic circumstances can influence their decision-making. However, mechanisms for providing rational purchasing support that takes emotional states into account are not yet sufficiently developed. This can lead to inappropriate purchases and financial burdens, so there is a need for a system that can address these issues and enable consumers to make more balanced decisions.

[0149] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0150] In this invention, the server includes means for recognizing information about an item using an information acquisition device, means for transmitting the item information to the server via a network, means for generating personalized advice by combining the user's financial data and emotional data based on the item information using the server, means for recognizing the user's emotions using sensors and transmitting that data to the server, and means for visually displaying the generated advice using augmented reality technology. This enables consumers to make purchasing decisions in real time, taking into account their emotions and economic situation.

[0151] An "information acquisition device" is a device used to read information from an object and acquire it as data, and generally refers to a camera or sensor.

[0152] A "network" is a communication path that enables the sending and receiving of data, and includes the internet and dedicated lines.

[0153] A "server" refers to a computing system that processes data on a network and provides information to other devices.

[0154] "Item information" refers to detailed data about an item, including information such as product name, model number, and price.

[0155] "Financial data" refers to a user's economic information, specifically data related to income, expenses, budget, etc.

[0156] "Emotional data" refers to information about a user's emotional state, and is data analyzed by an emotion engine based on facial expressions, tone of voice, and other factors.

[0157] "Personalized advice" refers to individualized advice generated based on each user's emotional state and financial situation.

[0158] A "sensor" is a device used to detect a user's emotional state, and examples include cameras and microphones.

[0159] Augmented reality technology is a technology that overlays computer-generated information onto the real environment and displays it, and includes methods for visually displaying virtual information.

[0160] The system for implementing this invention uses a combination of an information acquisition device, a server, an emotion engine, a generative AI model, and augmented reality technology to support the user's purchasing decision-making.

[0161] When users select products using their mobile devices, they utilize the built-in camera function to scan product information. This information includes the product name, price, and model number, and the information collected by the device is transmitted to a server via the network.

[0162] The server retrieves detailed product information from a database and, after obtaining user permission, collects financial data such as purchase history and budget. Furthermore, the terminal uses a camera and microphone to collect the user's facial expressions and voice, and transmits emotional data to the server.

[0163] The server uses an emotion engine to analyze the user's emotional state and, taking financial data into consideration, generates personalized advice best suited to the user. A generative AI model is used to create advice that is appropriate to the user's emotional state, such as advice to reconsider a purchase.

[0164] For example, when a user is about to purchase an expensive electronic product, the terminal recognizes the user's excited facial expression, and the server provides advice to encourage a calm decision. This advice is overlaid on the user's field of view along with product information, using augmented reality technology.

[0165] Examples of input prompts for a generative AI model:

[0166] "User's emotional state: excited, Product: expensive electronic product, please generate purchase advice."

[0167] This system allows users to make balanced decisions in real time, taking into account both their emotions and economic circumstances.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The user selects an item of interest in the store and scans its information using the device's camera function. The camera reads data such as product labels and QR codes. The input is the product image captured by the camera, and the output is text data such as the product name, price, and model number.

[0171] Step 2:

[0172] The terminal transmits scanned product information to the server via the network. A common protocol (e.g., HTTP) is used for this communication. The input is the product information acquired by the terminal, and the output is the text data of the product transferred to the server.

[0173] Step 3:

[0174] The server queries the database based on the received product information to retrieve detailed product information. This step retrieves product descriptions, ratings, inventory information, and other relevant data as needed. The input is the product text data sent to the server, and the output is the detailed product information from the database.

[0175] Step 4:

[0176] The server collects financial data, such as purchase history and budget information, based on the user's consent. This data is used to form a user profile and generate personalized advice. The input is the user's permission and related information, and the output is the user's financial profile data.

[0177] Step 5:

[0178] The device recognizes the user's emotions using its camera and microphone, and analyzes their facial expressions and voice. The recognized emotion data is sent to the server in real time. The input is the user's facial expressions and voice data, and the output is text data of the emotional state generated by the emotion engine.

[0179] Step 6:

[0180] The server receives emotion data and uses an emotion engine to analyze the user's emotional state. Based on this analysis, it evaluates the user's current emotional state. The input is emotion data from the device, and the output is information about the analyzed emotional state.

[0181] Step 7:

[0182] The server uses a generative AI model to generate personalized advice based on the user's emotional state and financial data. For example, if the user is agitated, it will create advice that encourages calm decision-making. The input is the emotional state and financial data, and the output is the text of the generated advice.

[0183] Step 8:

[0184] The server sends the generated advice to the terminal over the network. The terminal uses augmented reality technology to visually overlay this advice next to the product the user is viewing. The input is the advice text sent from the server, and the output is the visual advice information displayed on the terminal.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] Modern consumers face a wide variety of product and service choices, and information overload and emotional bias can sometimes prevent them from making sound decisions, especially when purchasing. In this context, there is a need for systems that support consumers in making calm and balanced decisions. However, conventional systems have been unable to recognize users' emotions in real time and provide personalized advice based on those emotions.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes means for recognizing product information using an information acquisition component, means for sentiment analysis, and means for visually displaying advice generated using augmented reality technology. This makes it possible to provide advice that takes into account the user's emotional state in real time.

[0190] An "information acquisition component" is a device used to recognize information about a product, and typically includes cameras and sensors.

[0191] A "network" is the infrastructure used for communication to transmit data, and typically includes the internet and local area networks.

[0192] An "information processing device" is a computer or server that receives and analyzes data, and functions to provide appropriate advice to the user.

[0193] "Financial data" refers to economic information, including a user's budget information and past purchase history, and is used to support purchasing decisions.

[0194] "Advice" refers to the information and suggestions provided to users when selecting a product, and is generated based on sentiment analysis and financial data.

[0195] "Emotional analysis methods" are technologies that analyze a user's facial expressions and voice to recognize their emotional state, thereby enabling the personalization of the user experience.

[0196] Augmented reality technology is a technique that overlays computer-generated information onto the real world, and is used to visually display advice next to the product the user is viewing.

[0197] The system for carrying out this invention utilizes an information acquisition component, a network, an information processing device, emotion analysis means, and augmented reality technology. The user uses a terminal such as a smartphone or smart glasses to scan the product's barcode or QR code and acquire product information. The terminal transmits this information to the information processing device via the network. This information processing device acquires the user's financial data along with the product information, and further analyzes the user's facial expressions and voice through the terminal's camera and microphone using emotion analysis means.

[0198] On the information processing device, the user's emotional state is evaluated based on the acquired data, and personalized advice is generated that matches that state. A generative AI model is used to output the optimal advice from this data. As an example of a specific prompt, the AI ​​is instructed to "Generate advice to refrain from making a purchase decision when the user is stressed, based on the user's facial expression data."

[0199] Furthermore, augmented reality technology is used to visually display advice on the smart glasses' screen. This allows users to receive advice on product selection that takes into account their emotions and financial situation in real time. For example, when a user is about to purchase expensive sneakers, the information processing device can detect the user's excited state and provide advice to encourage calmer judgment and suggest considering other options.

[0200] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0201] Step 1:

[0202] The user uses the device's camera function to scan the product's barcode or QR code. Visual information of the product is obtained as input. Based on this information, the device extracts and displays the product's basic information. The output is the product information, which is then passed on to the next processing step.

[0203] Step 2:

[0204] The terminal transmits acquired product information to the server via the network. It uses product information as input and sends the information to the server according to a data transmission protocol. The output is the product information stored on the server.

[0205] Step 3:

[0206] The server retrieves the user's financial data from the database. It uses the user's identification information as input and cross-references it with the user's past purchase history and budget information. The output is the user's financial data, which is then used in the subsequent sentiment analysis.

[0207] Step 4:

[0208] The device acquires the user's facial expressions and voice using emotion analysis techniques. It uses camera and microphone data as input and analyzes the user's emotional state using an emotion recognition algorithm. The output is emotion data, which is sent to the server.

[0209] Step 5:

[0210] The server integrates emotional and financial data and uses a generative AI model to generate personalized advice based on the user's emotional state. The input consists of emotional and financial data, which are then processed by the AI ​​model to create the advice. The output is the generated advice.

[0211] Step 6:

[0212] The terminal displays generated advice received from the server within the user's field of view using augmented reality technology. It uses the received advice data as input and visually displays it next to the product the user is viewing. The output is the augmented reality display information.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] This invention is a system that visualizes in real time how a product purchased by a user in a commercial facility will affect their economic situation and financial goals. This system uses a terminal equipped with a camera as an information acquisition device to identify products that the user is interested in.

[0230] First, when a user scans a product using their device's camera, the device sends product information to a server. The server retrieves detailed information related to the product from its database and combines it with the user's financial data. Using the financial data the user has authorized—for example, purchase history and a set monthly budget—the server generates personalized advice using AI that aligns with the user's current lifestyle and long-term goals.

[0231] The generated advice is sent to the device and visually presented to the user using augmented reality technology. This allows the user to intuitively understand the advice displayed next to the item. This information includes the impact of the purchase on the monthly budget and considerations for future spending plans.

[0232] As a concrete example, consider a user who is looking to buy a new home appliance and scans the product in a store. In this case, the server checks the price of the product, compares it with the user's past purchase history of similar products, and analyzes their spending patterns based on their spending categories. As a result, the server provides the user with guidelines such as "This may exceed this month's budget" or "We recommend purchasing it next month or later."

[0233] In this way, the system aims to help users make more informed purchasing decisions and improve their quality of life. Through the system, users can take appropriate purchasing actions in real time that are tailored to their own financial situation.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] The user scans the items they are considering purchasing in the store using the camera on their device. The device then obtains an image or barcode of the items.

[0237] Step 2:

[0238] The terminal transmits the acquired product information to the server via the network. This includes data necessary for identifying the product (such as image information and barcode data).

[0239] Step 3:

[0240] The server uses the received product information to search the database and retrieve detailed product information. This allows the server to collect information such as the product's price and detailed specifications.

[0241] Step 4:

[0242] The server retrieves purchase history and budget information that the user has previously authorized. This prepares the server to utilize detailed data such as the user's purchasing patterns and monthly spending trends.

[0243] Step 5:

[0244] The server uses AI generation to analyze the impact of product purchases on the user's financial situation and generate personalized advice. For example, it can generate suggestions regarding the possibility of exceeding the budget or the timing of purchases.

[0245] Step 6:

[0246] The server sends the generated advice to the terminal.

[0247] Step 7:

[0248] The device uses the received advice information and displays the advice on the camera image using augmented reality technology. Users can visually confirm advice related to the product.

[0249] Step 8:

[0250] Users make purchasing decisions based on the advice provided. They choose to either postpone the purchase or proceed with the purchase based on the advice.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] A problem exists when consumers make purchasing decisions that are in line with their own economic situation and financial goals. Specifically, there are insufficient means to immediately understand how the cost of a desired purchase will affect their budget. As a result, consumers often cannot avoid overspending, which frequently hinders the achievement of long-term goals.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes means for identifying products using an information acquisition device, means for transmitting product information to a data processing device via wireless communication technology, means for creating guidelines based on the product information and integrated with the user's economic data using the data processing device, and means for visually displaying the created guidelines using augmented reality technology. This enables consumers to understand their financial situation in real time and make appropriate purchasing decisions.

[0256] An "information acquisition device" is a device used to identify products and has the function of acquiring information by reading product barcodes or QR codes.

[0257] "Wireless communication technology" refers to technologies used to transmit product information, such as Wi-Fi and Bluetooth, which send and receive data without using cables.

[0258] A "data processing device" is a device that performs necessary processing based on received information, such as a server or computer, and is used to analyze information and create guidelines.

[0259] "User financial data" is a general term for financial information related to individual users, and includes data necessary for economic decision-making, such as purchase history, budget information, and savings targets.

[0260] "Guidelines" are advice and guidance created to support users' purchasing activities, and they propose specific actions that are appropriate to their economic situation and goals.

[0261] Augmented reality technology is a technology that overlays digital information onto the physical real environment, merging reality and virtuality through the presentation of visual information.

[0262] This invention is a system that provides users with real-time information based on their economic situation and financial goals when they purchase goods at a commercial facility.

[0263] First, the user scans the product they are considering purchasing using an information acquisition device equipped with a camera. The terminal reads the barcode or QR code attached to the product to obtain product information and transmits that information to a server via wireless communication technology. In this case, Wi-Fi is commonly used as the wireless communication technology.

[0264] The server retrieves product details from its internal database based on the received product information. Furthermore, it analyzes the impact of the product purchase on the user's financial situation using economic data that the user has previously authorized—specifically, information such as purchase history and monthly budget. A state-of-the-art generative AI model is used to process the data. An example of a prompt given to this model would be, "Based on the user's purchase history, analyze the impact of purchasing this product on the budget and generate advice."

[0265] The server generates the guidelines, which are then transmitted wirelessly to the terminal. The terminal uses augmented reality technology to visually display the guidelines. This allows the user to intuitively understand how the guidelines are presented by visually placing the product in the image, and use this information to inform their purchasing decisions.

[0266] As a concrete example, consider a scenario where a consumer is trying to purchase a new electronic appliance. When the user scans for a product, the server checks its price and compares it with the user's purchase history and budget to generate guidelines such as, "You may have a problem with your budget next month," or "Based on your previous purchase history, we recommend buying similar products during the next sale."

[0267] This system allows users to make appropriate purchasing decisions while considering their financial situation. It also functions as an effective means of improving quality of life and preventing wasteful spending.

[0268] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0269] Step 1:

[0270] The user scans the product using a camera-equipped device. Input includes the product's barcode or QR code. The device reads it and retrieves the product information. Specifically, the camera software within the device performs barcode scanning and obtains the product identification number.

[0271] Step 2:

[0272] The terminal transmits acquired product information to the server via wireless communication technology. Wi-Fi is commonly used in this case. The input includes the product identification number, which is then forwarded to the server as output. The communication module within the terminal operates and sends the product information as data packets to the server's receiving address.

[0273] Step 3:

[0274] The server retrieves the corresponding product details from its internal database based on the received product information. The input is a product identification number, and the output includes detailed information such as the product name, price, and category. A database query is executed, and product-related data is extracted.

[0275] Step 4:

[0276] The server integrates with user-authorized economic data. Inputs include the user's purchase history and financial information, which are combined with product details. The output is an integrated dataset. Specifically, the server uses CRUD (Create, Read, Uninstall, Delete) functionality to link existing user data to product details.

[0277] Step 5:

[0278] The server utilizes a generative AI model to generate guidelines based on an integrated dataset. Product and user economic data are used as input, and personalized purchasing advice is provided as output. The generative AI model is given prompts, and the AI ​​generates corresponding guidelines through analysis.

[0279] Step 6:

[0280] The server sends the generated guidance to the terminal. Wireless communication technology is used here as well. The input includes the generated guidance, and the output is the transmission of information to the terminal. The server's transmission protocol is activated, and the generated data is sent to the terminal's receiving address.

[0281] Step 7:

[0282] The device uses augmented reality technology to visually display the received guidance to the user. The input is guidance information received from a server, and the output is displayed on the screen. AR software within the device is activated, allowing the user to view the guidance along with the product displayed through the camera.

[0283] (Application Example 1)

[0284] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0285] In modern physical stores, it is difficult for consumers to immediately understand how the products they are considering purchasing will affect their accounting situation and long-term life goals. In the conventional method, it is common for consumers to consider the impact of expenses after purchase, which may result in inappropriate expenses. Therefore, there is a need for a system that can intuitively understand the impact of products on the accounting situation before purchase.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0287] In this invention, the server includes means for collecting product data, means for generating advice by combining personal accounting data and product data, and means for visually displaying the generated advice using augmented reality technology. As a result, consumers can instantly grasp the impact of products on their financial situation before purchasing products in physical stores.

[0288] The "information acquisition terminal" is a device for recognizing product data and is used when the user identifies a product.

[0289] The "communication network" is a network system for transmitting and receiving information and is used for data exchange between the information acquisition terminal and the central processing unit.

[0290] The "central processing unit" is a system having a processing function for generating advice by combining product data and personal accounting data.

[0291] Augmented reality technology is a technique used to visually display generated advice by overlaying computer-generated information onto images of the real world.

[0292] "Personal consumption history" refers to records of past spending by consumers and is data used to analyze consumption trends.

[0293] A "spending plan" is a plan for future spending that consumers anticipate, and is related to budgeting and financial goals.

[0294] "Lifestyle" refers to a consumer's patterns of living and preferences, and is a factor that influences their purchasing behavior.

[0295] This invention utilizes an information acquisition terminal, a communication network, and a central processing unit. A portable device such as a smartphone or tablet is used as the information acquisition terminal, and its camera function is used to acquire product data. For example, a user can quickly obtain product information by scanning the barcode of a product they are considering purchasing in a store. This product information is transmitted to the central processing unit via the communication network.

[0296] The central processing unit combines the received product information with personal accounting data such as the user's consumption history and spending plans, and generates personalized advice using a generative AI model. In this process, the generative AI model uses machine learning models such as Hugging Face Transformers to provide advice that helps the user make the best purchase decisions.

[0297] The generated advice is visually displayed on the information acquisition terminal using augmented reality technology. By utilizing augmented reality platforms such as ARKit and ARCore, the advice is overlaid next to the product, making it easy for the user to intuitively understand. This visual display allows users to instantly recognize how the product will affect their accounting situation and future spending.

[0298] As a specific example, assume that a user is considering purchasing a new speaker. When the user scans the speaker with the camera of the terminal, the central processing unit receives the product information and generates specific advice such as "fits within this month's budget" or "recommended for purchase" considering the user's past consumption history and expenditure plan. This generated advice is displayed on the screen of the information acquisition terminal using augmented reality technology.

[0299] Examples of prompt sentences for the generation AI model are as follows:

[0300] User's monthly budget: 50,000 yen

[0301] User's current expenditure amount: 40,000 yen

[0302] Product under consideration for purchase: Speaker (15,000 yen)

[0303] Expected expenditure after next month: Estimate based on normal living expenses

[0304] Please generate advice to be provided to the user.

[0305] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0306] Step 1:

[0307] The user scans the product using the camera function of the information acquisition terminal. The input is the image data of the product acquired by the camera. The output is the identification information of the product, which is obtained by analyzing the product's barcode or QR code. For the analysis, an image processing library such as OpenCV is used.

[0308] Step 2:

[0309] The terminal transmits product identification information to the central processing unit via the communication network. The input is the product identification information obtained in step 1. The output is a database query for product information on the server side. A REST API is used for communication to retrieve detailed information related to the product from the database.

[0310] Step 3:

[0311] The server retrieves product information from the database and combines it with the user's personal accounting data (e.g., consumption history and spending plan). The input consists of product information and the user's accounting data. The output is an integrated information set. This data processing uses libraries such as Pandas for data storage and calculations.

[0312] Step 4:

[0313] Based on the integrated information set obtained by the server, a generative AI model is used to generate personalized advice. The input is the integrated information set obtained in step 3. The output is personalized advice. Hugging Face Transformers are used as the generative AI model, and the machine learning model generates advice based on pre-configured prompt sentences.

[0314] Step 5:

[0315] The server sends the generated advice to the device. The input is the advice generated in step 4. The output is a visual representation using augmented reality technology. The REST API is used again to send data from the server, and ARKit or ARCore is used to overlay the advice onto the device's camera feed.

[0316] Step 6:

[0317] The user views advice overlaid next to the product on their device screen. The device utilizes augmented reality (AR) to intuitively display the generated advice. The input is visual data received from the server. The output is visual information of the advice organized in a way that is easy for the user to understand. This display allows the user to immediately grasp the impact the product will have on their financial situation.

[0318] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0319] This invention is a system that incorporates an emotion engine to recognize user emotions in order to support purchasing decisions for goods. When a user selects products in a store, product information is captured using a terminal. In this process, the terminal uses a camera function as an information acquisition device.

[0320] When a user scans a product selected through the camera, the device acquires the product information and transmits it to a server via the network. The server receives the product information and retrieves detailed information related to the product from its database. Furthermore, with the user's permission, the server collects the user's financial data based on purchase history and budget information.

[0321] This system incorporates an emotion engine, which allows the device to recognize emotions from the user's facial expressions and voice. This emotion data is sent to a server, which uses the emotion engine to analyze the user's emotional state. Based on the emotional state, the server uses generative AI to enhance personalized advice. In particular, if the user is feeling stressed, for example, it will generate advice encouraging them to reconsider their purchase.

[0322] The generated advice is sent from the server to the terminal, which uses augmented reality technology to visually overlay the advice next to the product the user is viewing. This allows the user to make purchasing decisions while simultaneously considering their emotions and financial situation at that moment. For example, when a user is about to purchase an expensive electronic product, the emotion engine may detect the user's state of excitement and provide advice to encourage a calmer decision.

[0323] Thus, the present invention is designed to enable users to make more balanced purchasing decisions by integrating user emotions and economic information to support product purchases. The system aims to improve the user's quality of life by providing real-time emotional and financial advice.

[0324] The following describes the processing flow.

[0325] Step 1:

[0326] The user scans the product they are considering purchasing in the store using the terminal's camera. The terminal acquires the product image or barcode and processes this information as digital data.

[0327] Step 2:

[0328] The terminal transmits the acquired product information to the server via the network. This product information includes the product ID and category information.

[0329] Step 3:

[0330] Based on the received product information, the server retrieves detailed information about the product from the database. This includes the product's price, brand, and reviews.

[0331] Step 4:

[0332] The server accesses the user's purchase history and budget information to retrieve their financial data. This data collection is conducted only within the scope of the user's prior permission.

[0333] Step 5:

[0334] The device uses a built-in emotion engine to recognize the user's emotions from their facial expressions and voice. This emotion data is sent from the device to a server, where it is used to analyze the user's current emotional state as digital data.

[0335] Step 6:

[0336] The server integrates emotional and financial data and uses generative AI to generate personalized advice for the user. For example, if a user is excited and considering purchasing an expensive item, a message encouraging calmness will be generated.

[0337] Step 7:

[0338] The generated advice is sent from the server to the terminal, and the terminal displays this advice overlaid on the camera image using AR technology.

[0339] Step 8:

[0340] Users use the provided advice to decide whether or not to continue purchasing the product. This advice includes recommendations based on their emotions and financial situation.

[0341] (Example 2)

[0342] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0343] Modern consumers must make purchasing decisions from a wide range of options, and emotions and economic circumstances can influence their decision-making. However, mechanisms for providing rational purchasing support that takes emotional states into account are not yet sufficiently developed. This can lead to inappropriate purchases and financial burdens, so there is a need for a system that can address these issues and enable consumers to make more balanced decisions.

[0344] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0345] In this invention, the server includes means for recognizing information about an item using an information acquisition device, means for transmitting the item information to the server via a network, means for generating personalized advice by combining the user's financial data and emotional data based on the item information using the server, means for recognizing the user's emotions using sensors and transmitting that data to the server, and means for visually displaying the generated advice using augmented reality technology. This enables consumers to make purchasing decisions in real time, taking into account their emotions and economic situation.

[0346] An "information acquisition device" is a device used to read information from an object and acquire it as data, and generally refers to a camera or sensor.

[0347] A "network" is a communication path that enables the sending and receiving of data, and includes the internet and dedicated lines.

[0348] A "server" refers to a computing system that processes data on a network and provides information to other devices.

[0349] "Item information" refers to detailed data about an item, including information such as product name, model number, and price.

[0350] "Financial data" refers to a user's economic information, specifically data related to income, expenses, budget, etc.

[0351] "Emotional data" refers to information about a user's emotional state, and is data analyzed by an emotion engine based on facial expressions, tone of voice, and other factors.

[0352] "Personalized advice" refers to individualized advice generated based on each user's emotional state and financial situation.

[0353] A "sensor" is a device used to detect a user's emotional state, and examples include cameras and microphones.

[0354] Augmented reality technology is a technology that overlays computer-generated information onto the real environment and displays it, and includes methods for visually displaying virtual information.

[0355] The system for implementing this invention uses a combination of an information acquisition device, a server, an emotion engine, a generative AI model, and augmented reality technology to support the user's purchasing decision-making.

[0356] When users select products using their mobile devices, they utilize the built-in camera function to scan product information. This information includes the product name, price, and model number, and the information collected by the device is transmitted to a server via the network.

[0357] The server retrieves detailed product information from a database and, after obtaining user permission, collects financial data such as purchase history and budget. Furthermore, the terminal uses a camera and microphone to collect the user's facial expressions and voice, and transmits emotional data to the server.

[0358] The server uses an emotion engine to analyze the user's emotional state and, taking financial data into consideration, generates personalized advice best suited to the user. A generative AI model is used to create advice that is appropriate to the user's emotional state, such as advice to reconsider a purchase.

[0359] For example, when a user is about to purchase an expensive electronic product, the terminal recognizes the user's excited facial expression, and the server provides advice to encourage a calm decision. This advice is overlaid on the user's field of view along with product information, using augmented reality technology.

[0360] Examples of input prompts for a generative AI model:

[0361] "User's emotional state: excited, Product: expensive electronic product, please generate purchase advice."

[0362] This system allows users to make balanced decisions in real time, taking into account both their emotions and economic circumstances.

[0363] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0364] Step 1:

[0365] The user selects an item of interest in the store and scans its information using the device's camera function. The camera reads data such as product labels and QR codes. The input is the product image captured by the camera, and the output is text data such as the product name, price, and model number.

[0366] Step 2:

[0367] The terminal transmits scanned product information to the server via the network. A common protocol (e.g., HTTP) is used for this communication. The input is the product information acquired by the terminal, and the output is the text data of the product transferred to the server.

[0368] Step 3:

[0369] The server queries the database based on the received product information to retrieve detailed product information. This step retrieves product descriptions, ratings, inventory information, and other relevant data as needed. The input is the product text data sent to the server, and the output is the detailed product information from the database.

[0370] Step 4:

[0371] The server collects financial data, such as purchase history and budget information, based on the user's consent. This data is used to form a user profile and generate personalized advice. The input is the user's permission and related information, and the output is the user's financial profile data.

[0372] Step 5:

[0373] The device recognizes the user's emotions using its camera and microphone, and analyzes their facial expressions and voice. The recognized emotion data is sent to the server in real time. The input is the user's facial expressions and voice data, and the output is text data of the emotional state generated by the emotion engine.

[0374] Step 6:

[0375] The server receives emotion data and uses an emotion engine to analyze the user's emotional state. Based on this analysis, it evaluates the user's current emotional state. The input is emotion data from the device, and the output is information about the analyzed emotional state.

[0376] Step 7:

[0377] The server uses a generative AI model to generate personalized advice based on the user's emotional state and financial data. For example, if the user is agitated, it will create advice that encourages calm decision-making. The input is the emotional state and financial data, and the output is the text of the generated advice.

[0378] Step 8:

[0379] The server sends the generated advice to the terminal over the network. The terminal uses augmented reality technology to visually overlay this advice next to the product the user is viewing. The input is the advice text sent from the server, and the output is the visual advice information displayed on the terminal.

[0380] (Application Example 2)

[0381] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0382] Modern consumers face a wide variety of product and service choices, and information overload and emotional bias can sometimes prevent them from making sound decisions, especially when purchasing. In this context, there is a need for systems that support consumers in making calm and balanced decisions. However, conventional systems have been unable to recognize users' emotions in real time and provide personalized advice based on those emotions.

[0383] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0384] In this invention, the server includes means for recognizing product information using an information acquisition component, means for sentiment analysis, and means for visually displaying advice generated using augmented reality technology. This makes it possible to provide advice that takes into account the user's emotional state in real time.

[0385] An "information acquisition component" is a device used to recognize information about a product, and typically includes cameras and sensors.

[0386] A "network" is the infrastructure used for communication to transmit data, and typically includes the internet and local area networks.

[0387] An "information processing device" is a computer or server that receives and analyzes data, and functions to provide appropriate advice to the user.

[0388] "Financial data" refers to economic information, including a user's budget information and past purchase history, and is used to support purchasing decisions.

[0389] "Advice" refers to the information and suggestions provided to users when selecting a product, and is generated based on sentiment analysis and financial data.

[0390] "Emotional analysis methods" are technologies that analyze a user's facial expressions and voice to recognize their emotional state, thereby enabling the personalization of the user experience.

[0391] Augmented reality technology is a technique that overlays computer-generated information onto the real world, and is used to visually display advice next to the product the user is viewing.

[0392] The system for carrying out this invention utilizes an information acquisition component, a network, an information processing device, emotion analysis means, and augmented reality technology. The user uses a terminal such as a smartphone or smart glasses to scan the product's barcode or QR code and acquire product information. The terminal transmits this information to the information processing device via the network. This information processing device acquires the user's financial data along with the product information, and further analyzes the user's facial expressions and voice through the terminal's camera and microphone using emotion analysis means.

[0393] On the information processing device, the user's emotional state is evaluated based on the acquired data, and personalized advice is generated that matches that state. A generative AI model is used to output the optimal advice from this data. As an example of a specific prompt, the AI ​​is instructed to "Generate advice to refrain from making a purchase decision when the user is stressed, based on the user's facial expression data."

[0394] Furthermore, augmented reality technology is used to visually display advice on the smart glasses' screen. This allows users to receive advice on product selection that takes into account their emotions and financial situation in real time. For example, when a user is about to purchase expensive sneakers, the information processing device can detect the user's excited state and provide advice to encourage calmer judgment and suggest considering other options.

[0395] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0396] Step 1:

[0397] The user uses the device's camera function to scan the product's barcode or QR code. Visual information of the product is obtained as input. Based on this information, the device extracts and displays the product's basic information. The output is the product information, which is then passed on to the next processing step.

[0398] Step 2:

[0399] The terminal transmits acquired product information to the server via the network. It uses product information as input and sends the information to the server according to a data transmission protocol. The output is the product information stored on the server.

[0400] Step 3:

[0401] The server retrieves the user's financial data from the database. It uses the user's identification information as input and cross-references it with the user's past purchase history and budget information. The output is the user's financial data, which is then used in the subsequent sentiment analysis.

[0402] Step 4:

[0403] The device acquires the user's facial expressions and voice using emotion analysis techniques. It uses camera and microphone data as input and analyzes the user's emotional state using an emotion recognition algorithm. The output is emotion data, which is sent to the server.

[0404] Step 5:

[0405] The server integrates emotional and financial data and uses a generative AI model to generate personalized advice based on the user's emotional state. The input consists of emotional and financial data, which are then processed by the AI ​​model to create the advice. The output is the generated advice.

[0406] Step 6:

[0407] The terminal displays generated advice received from the server within the user's field of view using augmented reality technology. It uses the received advice data as input and visually displays it next to the product the user is viewing. The output is the augmented reality display information.

[0408] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0409] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0410] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0411] [Third Embodiment]

[0412] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0413] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0414] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0415] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0416] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0417] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0418] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0419] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0420] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0421] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0422] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0423] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0424] This invention is a system that visualizes in real time how a product purchased by a user in a commercial facility will affect their economic situation and financial goals. This system uses a terminal equipped with a camera as an information acquisition device to identify products that the user is interested in.

[0425] First, when a user scans a product using their device's camera, the device sends product information to a server. The server retrieves detailed information related to the product from its database and combines it with the user's financial data. Using the financial data the user has authorized—for example, purchase history and a set monthly budget—the server generates personalized advice using AI that aligns with the user's current lifestyle and long-term goals.

[0426] The generated advice is sent to the device and visually presented to the user using augmented reality technology. This allows the user to intuitively understand the advice displayed next to the item. This information includes the impact of the purchase on the monthly budget and considerations for future spending plans.

[0427] As a concrete example, consider a user who is looking to buy a new home appliance and scans the product in a store. In this case, the server checks the price of the product, compares it with the user's past purchase history of similar products, and analyzes their spending patterns based on their spending categories. As a result, the server provides the user with guidelines such as "This may exceed this month's budget" or "We recommend purchasing it next month or later."

[0428] In this way, the system aims to help users make more informed purchasing decisions and improve their quality of life. Through the system, users can take appropriate purchasing actions in real time that are tailored to their own financial situation.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The user scans the items they are considering purchasing in the store using the camera on their device. The device then obtains an image or barcode of the items.

[0432] Step 2:

[0433] The terminal transmits the acquired product information to the server via the network. This includes data necessary for identifying the product (such as image information and barcode data).

[0434] Step 3:

[0435] The server uses the received product information to search the database and retrieve detailed product information. This allows the server to collect information such as the product's price and detailed specifications.

[0436] Step 4:

[0437] The server retrieves purchase history and budget information that the user has previously authorized. This prepares the server to utilize detailed data such as the user's purchasing patterns and monthly spending trends.

[0438] Step 5:

[0439] The server uses AI generation to analyze the impact of product purchases on the user's financial situation and generate personalized advice. For example, it can generate suggestions regarding the possibility of exceeding the budget or the timing of purchases.

[0440] Step 6:

[0441] The server sends the generated advice to the terminal.

[0442] Step 7:

[0443] The device uses the received advice information and displays the advice on the camera image using augmented reality technology. Users can visually confirm advice related to the product.

[0444] Step 8:

[0445] Users make purchasing decisions based on the advice provided. They choose to either postpone the purchase or proceed with the purchase based on the advice.

[0446] (Example 1)

[0447] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0448] A problem exists when consumers make purchasing decisions that are in line with their own economic situation and financial goals. Specifically, there are insufficient means to immediately understand how the cost of a desired purchase will affect their budget. As a result, consumers often cannot avoid overspending, which frequently hinders the achievement of long-term goals.

[0449] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0450] In this invention, the server includes means for identifying products using an information acquisition device, means for transmitting product information to a data processing device via wireless communication technology, means for creating guidelines based on the product information and integrated with the user's economic data using the data processing device, and means for visually displaying the created guidelines using augmented reality technology. This enables consumers to understand their financial situation in real time and make appropriate purchasing decisions.

[0451] An "information acquisition device" is a device used to identify products and has the function of acquiring information by reading product barcodes or QR codes.

[0452] "Wireless communication technology" refers to technologies used to transmit product information, such as Wi-Fi and Bluetooth, which send and receive data without using cables.

[0453] A "data processing device" is a device that performs necessary processing based on received information, such as a server or computer, and is used to analyze information and create guidelines.

[0454] "User financial data" is a general term for financial information related to individual users, and includes data necessary for economic decision-making, such as purchase history, budget information, and savings targets.

[0455] "Guidelines" are advice and guidance created to support users' purchasing activities, and they propose specific actions that are appropriate to their economic situation and goals.

[0456] Augmented reality technology is a technology that overlays digital information onto the physical real environment, merging reality and virtuality through the presentation of visual information.

[0457] This invention is a system that provides users with real-time information based on their economic situation and financial goals when they purchase goods at a commercial facility.

[0458] First, the user scans the product they are considering purchasing using an information acquisition device equipped with a camera. The terminal reads the barcode or QR code attached to the product to obtain product information and transmits that information to a server via wireless communication technology. In this case, Wi-Fi is commonly used as the wireless communication technology.

[0459] The server retrieves product details from its internal database based on the received product information. Furthermore, it analyzes the impact of the product purchase on the user's financial situation using economic data that the user has previously authorized—specifically, information such as purchase history and monthly budget. A state-of-the-art generative AI model is used to process the data. An example of a prompt given to this model would be, "Based on the user's purchase history, analyze the impact of purchasing this product on the budget and generate advice."

[0460] The server generates the guidelines, which are then transmitted wirelessly to the terminal. The terminal uses augmented reality technology to visually display the guidelines. This allows the user to intuitively understand how the guidelines are presented by visually placing the product in the image, and use this information to inform their purchasing decisions.

[0461] As a concrete example, consider a scenario where a consumer is trying to purchase a new electronic appliance. When the user scans for a product, the server checks its price and compares it with the user's purchase history and budget to generate guidelines such as, "You may have a problem with your budget next month," or "Based on your previous purchase history, we recommend buying similar products during the next sale."

[0462] This system allows users to make appropriate purchasing decisions while considering their financial situation. It also functions as an effective means of improving quality of life and preventing wasteful spending.

[0463] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0464] Step 1:

[0465] The user scans the product using a camera-equipped device. Input includes the product's barcode or QR code. The device reads it and retrieves the product information. Specifically, the camera software within the device performs barcode scanning and obtains the product identification number.

[0466] Step 2:

[0467] The terminal transmits acquired product information to the server via wireless communication technology. Wi-Fi is commonly used in this case. The input includes the product identification number, which is then forwarded to the server as output. The communication module within the terminal operates and sends the product information as data packets to the server's receiving address.

[0468] Step 3:

[0469] The server retrieves the corresponding product details from its internal database based on the received product information. The input is a product identification number, and the output includes detailed information such as the product name, price, and category. A database query is executed, and product-related data is extracted.

[0470] Step 4:

[0471] The server integrates with user-authorized economic data. Inputs include the user's purchase history and financial information, which are combined with product details. The output is an integrated dataset. Specifically, the server uses CRUD (Create, Read, Uninstall, Delete) functionality to link existing user data to product details.

[0472] Step 5:

[0473] The server utilizes a generative AI model to generate guidelines based on an integrated dataset. Product and user economic data are used as input, and personalized purchasing advice is provided as output. The generative AI model is given prompts, and the AI ​​generates corresponding guidelines through analysis.

[0474] Step 6:

[0475] The server sends the generated guidance to the terminal. Wireless communication technology is used here as well. The input includes the generated guidance, and the output is the transmission of information to the terminal. The server's transmission protocol is activated, and the generated data is sent to the terminal's receiving address.

[0476] Step 7:

[0477] The device uses augmented reality technology to visually display the received guidance to the user. The input is guidance information received from a server, and the output is displayed on the screen. AR software within the device is activated, allowing the user to view the guidance along with the product displayed through the camera.

[0478] (Application Example 1)

[0479] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] In today's brick-and-mortar stores, it's difficult for consumers to immediately understand how a product they're considering purchasing will impact their finances and long-term life goals. Traditionally, consumers typically consider the impact of their spending after the fact, which can lead to inappropriate spending. Therefore, there's a need for a system that allows consumers to intuitively understand the impact of a product on their finances before purchasing it.

[0481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0482] In this invention, the server includes means for collecting product data, means for generating advice by combining personal accounting data and product data, and means for visually displaying the generated advice using augmented reality technology. This allows consumers to instantly understand the impact a product will have on their financial situation before purchasing it in a physical store.

[0483] An "information acquisition terminal" is a device used to recognize product data and is used by users to identify products.

[0484] A "communication network" is a network system for transmitting and receiving information, used to exchange data between information acquisition terminals and central processing units.

[0485] A "central processing unit" is a system that has processing functions to generate advice by combining product data and individual accounting data.

[0486] Augmented reality technology is a technique used to visually display generated advice by overlaying computer-generated information onto images of the real world.

[0487] "Personal consumption history" refers to records of past spending by consumers and is data used to analyze consumption trends.

[0488] A "spending plan" is a plan for future spending that consumers anticipate, and is related to budgeting and financial goals.

[0489] "Lifestyle" refers to a consumer's patterns of living and preferences, and is a factor that influences their purchasing behavior.

[0490] This invention utilizes an information acquisition terminal, a communication network, and a central processing unit. A portable device such as a smartphone or tablet is used as the information acquisition terminal, and its camera function is used to acquire product data. For example, a user can quickly obtain product information by scanning the barcode of a product they are considering purchasing in a store. This product information is transmitted to the central processing unit via the communication network.

[0491] The central processing unit combines the received product information with personal accounting data such as the user's consumption history and spending plans, and generates personalized advice using a generative AI model. In this process, the generative AI model uses machine learning models such as Hugging Face Transformers to provide advice that helps the user make the best purchase decisions.

[0492] The generated advice is visually displayed on the information acquisition terminal using augmented reality technology. By utilizing augmented reality platforms such as ARKit and ARCore, the advice is overlaid next to the product, making it easy for the user to intuitively understand. This visual display allows users to instantly recognize how the product will affect their accounting situation and future spending.

[0493] As a concrete example, suppose a user is considering purchasing a new speaker. When the user scans this speaker with their device's camera, the central processing unit receives product information and, considering past purchase history and spending plans, generates specific advice such as "It fits this month's budget" and "Purchase recommended." This generated advice is then displayed on the information acquisition device's screen using augmented reality technology.

[0494] Examples of prompts for a generative AI model include:

[0495] User's monthly budget: 50,000 yen

[0496] User's current spending: 40,000 yen

[0497] Product I'm considering buying: Speakers (¥15,000)

[0498] Planned expenses for next month and beyond: Estimates based on normal living expenses.

[0499] Please generate advice to provide to the user.

[0500] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0501] Step 1:

[0502] The user scans the product using the camera function of the information acquisition terminal. The input is image data of the product acquired by the camera. The output is product identification information, obtained by analyzing the product's barcode or QR code. Image processing libraries such as OpenCV are used for the analysis.

[0503] Step 2:

[0504] The terminal transmits product identification information to the central processing unit via the communication network. The input is the product identification information obtained in step 1. The output is a database query for product information on the server side. A REST API is used for communication to retrieve detailed information related to the product from the database.

[0505] Step 3:

[0506] The server retrieves product information from the database and combines it with the user's personal accounting data (e.g., consumption history and spending plan). The input consists of product information and the user's accounting data. The output is an integrated information set. This data processing uses libraries such as Pandas for data storage and calculations.

[0507] Step 4:

[0508] Based on the integrated information set obtained by the server, a generative AI model is used to generate personalized advice. The input is the integrated information set obtained in step 3. The output is personalized advice. Hugging Face Transformers are used as the generative AI model, and the machine learning model generates advice based on pre-configured prompt sentences.

[0509] Step 5:

[0510] The server sends the generated advice to the device. The input is the advice generated in step 4. The output is a visual representation using augmented reality technology. The REST API is used again to send data from the server, and ARKit or ARCore is used to overlay the advice onto the device's camera feed.

[0511] Step 6:

[0512] The user views advice overlaid next to the product on their device screen. The device utilizes augmented reality (AR) to intuitively display the generated advice. The input is visual data received from the server. The output is visual information of the advice organized in a way that is easy for the user to understand. This display allows the user to immediately grasp the impact the product will have on their financial situation.

[0513] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0514] This invention is a system that incorporates an emotion engine to recognize user emotions in order to support purchasing decisions for goods. When a user selects products in a store, product information is captured using a terminal. In this process, the terminal uses a camera function as an information acquisition device.

[0515] When a user scans a product selected through the camera, the device acquires the product information and transmits it to a server via the network. The server receives the product information and retrieves detailed information related to the product from its database. Furthermore, with the user's permission, the server collects the user's financial data based on purchase history and budget information.

[0516] This system incorporates an emotion engine, which allows the device to recognize emotions from the user's facial expressions and voice. This emotion data is sent to a server, which uses the emotion engine to analyze the user's emotional state. Based on the emotional state, the server uses generative AI to enhance personalized advice. In particular, if the user is feeling stressed, for example, it will generate advice encouraging them to reconsider their purchase.

[0517] The generated advice is sent from the server to the terminal, which uses augmented reality technology to visually overlay the advice next to the product the user is viewing. This allows the user to make purchasing decisions while simultaneously considering their emotions and financial situation at that moment. For example, when a user is about to purchase an expensive electronic product, the emotion engine may detect the user's state of excitement and provide advice to encourage a calmer decision.

[0518] Thus, the present invention is designed to enable users to make more balanced purchasing decisions by integrating user emotions and economic information to support product purchases. The system aims to improve the user's quality of life by providing real-time emotional and financial advice.

[0519] The following describes the processing flow.

[0520] Step 1:

[0521] The user scans the product they are considering purchasing in the store using the terminal's camera. The terminal acquires the product image or barcode and processes this information as digital data.

[0522] Step 2:

[0523] The terminal transmits the acquired product information to the server via the network. This product information includes the product ID and category information.

[0524] Step 3:

[0525] Based on the received product information, the server retrieves detailed information about the product from the database. This includes the product's price, brand, and reviews.

[0526] Step 4:

[0527] The server accesses the user's purchase history and budget information to retrieve their financial data. This data collection is conducted only within the scope of the user's prior permission.

[0528] Step 5:

[0529] The device uses a built-in emotion engine to recognize the user's emotions from their facial expressions and voice. This emotion data is sent from the device to a server, where it is used to analyze the user's current emotional state as digital data.

[0530] Step 6:

[0531] The server integrates emotional and financial data and uses generative AI to generate personalized advice for the user. For example, if a user is excited and considering purchasing an expensive item, a message encouraging calmness will be generated.

[0532] Step 7:

[0533] The generated advice is sent from the server to the terminal, and the terminal displays this advice overlaid on the camera image using AR technology.

[0534] Step 8:

[0535] Users use the provided advice to decide whether or not to continue purchasing the product. This advice includes recommendations based on their emotions and financial situation.

[0536] (Example 2)

[0537] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0538] Modern consumers must make purchasing decisions from a wide range of options, and emotions and economic circumstances can influence their decision-making. However, mechanisms for providing rational purchasing support that takes emotional states into account are not yet sufficiently developed. This can lead to inappropriate purchases and financial burdens, so there is a need for a system that can address these issues and enable consumers to make more balanced decisions.

[0539] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0540] In this invention, the server includes means for recognizing information about an item using an information acquisition device, means for transmitting the item information to the server via a network, means for generating personalized advice by combining the user's financial data and emotional data based on the item information using the server, means for recognizing the user's emotions using sensors and transmitting that data to the server, and means for visually displaying the generated advice using augmented reality technology. This enables consumers to make purchasing decisions in real time, taking into account their emotions and economic situation.

[0541] An "information acquisition device" is a device used to read information from an object and acquire it as data, and generally refers to a camera or sensor.

[0542] A "network" is a communication path that enables the sending and receiving of data, and includes the internet and dedicated lines.

[0543] A "server" refers to a computing system that processes data on a network and provides information to other devices.

[0544] "Item information" refers to detailed data about an item, including information such as product name, model number, and price.

[0545] "Financial data" refers to a user's economic information, specifically data related to income, expenses, budget, etc.

[0546] "Emotional data" refers to information about a user's emotional state, and is data analyzed by an emotion engine based on facial expressions, tone of voice, and other factors.

[0547] "Personalized advice" refers to individualized advice generated based on each user's emotional state and financial situation.

[0548] A "sensor" is a device used to detect a user's emotional state, and examples include cameras and microphones.

[0549] Augmented reality technology is a technology that overlays computer-generated information onto the real environment and displays it, and includes methods for visually displaying virtual information.

[0550] The system for implementing this invention uses a combination of an information acquisition device, a server, an emotion engine, a generative AI model, and augmented reality technology to support the user's purchasing decision-making.

[0551] When users select products using their mobile devices, they utilize the built-in camera function to scan product information. This information includes the product name, price, and model number, and the information collected by the device is transmitted to a server via the network.

[0552] The server retrieves detailed product information from a database and, after obtaining user permission, collects financial data such as purchase history and budget. Furthermore, the terminal uses a camera and microphone to collect the user's facial expressions and voice, and transmits emotional data to the server.

[0553] The server uses an emotion engine to analyze the user's emotional state and, taking financial data into consideration, generates personalized advice best suited to the user. A generative AI model is used to create advice that is appropriate to the user's emotional state, such as advice to reconsider a purchase.

[0554] For example, when a user is about to purchase an expensive electronic product, the terminal recognizes the user's excited facial expression, and the server provides advice to encourage a calm decision. This advice is overlaid on the user's field of view along with product information, using augmented reality technology.

[0555] Examples of input prompts for a generative AI model:

[0556] "User's emotional state: excited, Product: expensive electronic product, please generate purchase advice."

[0557] This system allows users to make balanced decisions in real time, taking into account both their emotions and economic circumstances.

[0558] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0559] Step 1:

[0560] The user selects an item of interest in the store and scans its information using the device's camera function. The camera reads data such as product labels and QR codes. The input is the product image captured by the camera, and the output is text data such as the product name, price, and model number.

[0561] Step 2:

[0562] The terminal transmits scanned product information to the server via the network. A common protocol (e.g., HTTP) is used for this communication. The input is the product information acquired by the terminal, and the output is the text data of the product transferred to the server.

[0563] Step 3:

[0564] The server queries the database based on the received product information to retrieve detailed product information. This step retrieves product descriptions, ratings, inventory information, and other relevant data as needed. The input is the product text data sent to the server, and the output is the detailed product information from the database.

[0565] Step 4:

[0566] The server collects financial data, such as purchase history and budget information, based on the user's consent. This data is used to form a user profile and generate personalized advice. The input is the user's permission and related information, and the output is the user's financial profile data.

[0567] Step 5:

[0568] The device recognizes the user's emotions using its camera and microphone, and analyzes their facial expressions and voice. The recognized emotion data is sent to the server in real time. The input is the user's facial expressions and voice data, and the output is text data of the emotional state generated by the emotion engine.

[0569] Step 6:

[0570] The server receives emotion data and uses an emotion engine to analyze the user's emotional state. Based on this analysis, it evaluates the user's current emotional state. The input is emotion data from the device, and the output is information about the analyzed emotional state.

[0571] Step 7:

[0572] The server uses a generative AI model to generate personalized advice based on the user's emotional state and financial data. For example, if the user is agitated, it will create advice that encourages calm decision-making. The input is the emotional state and financial data, and the output is the text of the generated advice.

[0573] Step 8:

[0574] The server sends the generated advice to the terminal over the network. The terminal uses augmented reality technology to visually overlay this advice next to the product the user is viewing. The input is the advice text sent from the server, and the output is the visual advice information displayed on the terminal.

[0575] (Application Example 2)

[0576] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0577] Modern consumers face a wide variety of product and service choices, and information overload and emotional bias can sometimes prevent them from making sound decisions, especially when purchasing. In this context, there is a need for systems that support consumers in making calm and balanced decisions. However, conventional systems have been unable to recognize users' emotions in real time and provide personalized advice based on those emotions.

[0578] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0579] In this invention, the server includes means for recognizing product information using an information acquisition component, means for sentiment analysis, and means for visually displaying advice generated using augmented reality technology. This makes it possible to provide advice that takes into account the user's emotional state in real time.

[0580] An "information acquisition component" is a device used to recognize information about a product, and typically includes cameras and sensors.

[0581] A "network" is the infrastructure used for communication to transmit data, and typically includes the internet and local area networks.

[0582] An "information processing device" is a computer or server that receives and analyzes data, and functions to provide appropriate advice to the user.

[0583] "Financial data" refers to economic information, including a user's budget information and past purchase history, and is used to support purchasing decisions.

[0584] "Advice" refers to the information and suggestions provided to users when selecting a product, and is generated based on sentiment analysis and financial data.

[0585] "Emotional analysis methods" are technologies that analyze a user's facial expressions and voice to recognize their emotional state, thereby enabling the personalization of the user experience.

[0586] Augmented reality technology is a technique that overlays computer-generated information onto the real world, and is used to visually display advice next to the product the user is viewing.

[0587] The system for carrying out this invention utilizes an information acquisition component, a network, an information processing device, emotion analysis means, and augmented reality technology. The user uses a terminal such as a smartphone or smart glasses to scan the product's barcode or QR code and acquire product information. The terminal transmits this information to the information processing device via the network. This information processing device acquires the user's financial data along with the product information, and further analyzes the user's facial expressions and voice through the terminal's camera and microphone using emotion analysis means.

[0588] On the information processing device, the user's emotional state is evaluated based on the acquired data, and personalized advice is generated that matches that state. A generative AI model is used to output the optimal advice from this data. As an example of a specific prompt, the AI ​​is instructed to "Generate advice to refrain from making a purchase decision when the user is stressed, based on the user's facial expression data."

[0589] Furthermore, augmented reality technology is used to visually display advice on the smart glasses' screen. This allows users to receive advice on product selection that takes into account their emotions and financial situation in real time. For example, when a user is about to purchase expensive sneakers, the information processing device can detect the user's excited state and provide advice to encourage calmer judgment and suggest considering other options.

[0590] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0591] Step 1:

[0592] The user uses the device's camera function to scan the product's barcode or QR code. Visual information of the product is obtained as input. Based on this information, the device extracts and displays the product's basic information. The output is the product information, which is then passed on to the next processing step.

[0593] Step 2:

[0594] The terminal transmits acquired product information to the server via the network. It uses product information as input and sends the information to the server according to a data transmission protocol. The output is the product information stored on the server.

[0595] Step 3:

[0596] The server retrieves the user's financial data from the database. It uses the user's identification information as input and cross-references it with the user's past purchase history and budget information. The output is the user's financial data, which is then used in the subsequent sentiment analysis.

[0597] Step 4:

[0598] The device acquires the user's facial expressions and voice using emotion analysis techniques. It uses camera and microphone data as input and analyzes the user's emotional state using an emotion recognition algorithm. The output is emotion data, which is sent to the server.

[0599] Step 5:

[0600] The server integrates emotional and financial data and uses a generative AI model to generate personalized advice based on the user's emotional state. The input consists of emotional and financial data, which are then processed by the AI ​​model to create the advice. The output is the generated advice.

[0601] Step 6:

[0602] The terminal displays generated advice received from the server within the user's field of view using augmented reality technology. It uses the received advice data as input and visually displays it next to the product the user is viewing. The output is the augmented reality display information.

[0603] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0604] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0605] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0606] [Fourth Embodiment]

[0607] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0608] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0609] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0610] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0611] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0613] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0614] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0615] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0616] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0617] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0618] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0619] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] This invention is a system that visualizes in real time how a product purchased by a user in a commercial facility will affect their economic situation and financial goals. This system uses a terminal equipped with a camera as an information acquisition device to identify products that the user is interested in.

[0621] First, when a user scans a product using their device's camera, the device sends product information to a server. The server retrieves detailed information related to the product from its database and combines it with the user's financial data. Using the financial data the user has authorized—for example, purchase history and a set monthly budget—the server generates personalized advice using AI that aligns with the user's current lifestyle and long-term goals.

[0622] The generated advice is sent to the device and visually presented to the user using augmented reality technology. This allows the user to intuitively understand the advice displayed next to the item. This information includes the impact of the purchase on the monthly budget and considerations for future spending plans.

[0623] As a concrete example, consider a user who is looking to buy a new home appliance and scans the product in a store. In this case, the server checks the price of the product, compares it with the user's past purchase history of similar products, and analyzes their spending patterns based on their spending categories. As a result, the server provides the user with guidelines such as "This may exceed this month's budget" or "We recommend purchasing it next month or later."

[0624] In this way, the system aims to help users make more informed purchasing decisions and improve their quality of life. Through the system, users can take appropriate purchasing actions in real time that are tailored to their own financial situation.

[0625] The following describes the processing flow.

[0626] Step 1:

[0627] The user scans the items they are considering purchasing in the store using the camera on their device. The device then obtains an image or barcode of the items.

[0628] Step 2:

[0629] The terminal transmits the acquired product information to the server via the network. This includes data necessary for identifying the product (such as image information and barcode data).

[0630] Step 3:

[0631] The server uses the received product information to search the database and retrieve detailed product information. This allows the server to collect information such as the product's price and detailed specifications.

[0632] Step 4:

[0633] The server retrieves purchase history and budget information that the user has previously authorized. This prepares the server to utilize detailed data such as the user's purchasing patterns and monthly spending trends.

[0634] Step 5:

[0635] The server uses AI generation to analyze the impact of product purchases on the user's financial situation and generate personalized advice. For example, it can generate suggestions regarding the possibility of exceeding the budget or the timing of purchases.

[0636] Step 6:

[0637] The server sends the generated advice to the terminal.

[0638] Step 7:

[0639] The device uses the received advice information and displays the advice on the camera image using augmented reality technology. Users can visually confirm advice related to the product.

[0640] Step 8:

[0641] Users make purchasing decisions based on the advice provided. They choose to either postpone the purchase or proceed with the purchase based on the advice.

[0642] (Example 1)

[0643] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] A problem exists when consumers make purchasing decisions that are in line with their own economic situation and financial goals. Specifically, there are insufficient means to immediately understand how the cost of a desired purchase will affect their budget. As a result, consumers often cannot avoid overspending, which frequently hinders the achievement of long-term goals.

[0645] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0646] In this invention, the server includes means for identifying products using an information acquisition device, means for transmitting product information to a data processing device via wireless communication technology, means for creating guidelines based on the product information and integrated with the user's economic data using the data processing device, and means for visually displaying the created guidelines using augmented reality technology. This enables consumers to understand their financial situation in real time and make appropriate purchasing decisions.

[0647] An "information acquisition device" is a device used to identify products and has the function of acquiring information by reading product barcodes or QR codes.

[0648] "Wireless communication technology" refers to technologies used to transmit product information, such as Wi-Fi and Bluetooth, which send and receive data without using cables.

[0649] A "data processing device" is a device that performs necessary processing based on received information, such as a server or computer, and is used to analyze information and create guidelines.

[0650] "User financial data" is a general term for financial information related to individual users, and includes data necessary for economic decision-making, such as purchase history, budget information, and savings targets.

[0651] "Guidelines" are advice and guidance created to support users' purchasing activities, and they propose specific actions that are appropriate to their economic situation and goals.

[0652] Augmented reality technology is a technology that overlays digital information onto the physical real environment, merging reality and virtuality through the presentation of visual information.

[0653] This invention is a system that provides users with real-time information based on their economic situation and financial goals when they purchase goods at a commercial facility.

[0654] First, the user scans the product they are considering purchasing using an information acquisition device equipped with a camera. The terminal reads the barcode or QR code attached to the product to obtain product information and transmits that information to a server via wireless communication technology. In this case, Wi-Fi is commonly used as the wireless communication technology.

[0655] The server retrieves product details from its internal database based on the received product information. Furthermore, it analyzes the impact of the product purchase on the user's financial situation using economic data that the user has previously authorized—specifically, information such as purchase history and monthly budget. A state-of-the-art generative AI model is used to process the data. An example of a prompt given to this model would be, "Based on the user's purchase history, analyze the impact of purchasing this product on the budget and generate advice."

[0656] The server generates the guidelines, which are then transmitted wirelessly to the terminal. The terminal uses augmented reality technology to visually display the guidelines. This allows the user to intuitively understand how the guidelines are presented by visually placing the product in the image, and use this information to inform their purchasing decisions.

[0657] As a concrete example, consider a scenario where a consumer is trying to purchase a new electronic appliance. When the user scans for a product, the server checks its price and compares it with the user's purchase history and budget to generate guidelines such as, "You may have a problem with your budget next month," or "Based on your previous purchase history, we recommend buying similar products during the next sale."

[0658] This system allows users to make appropriate purchasing decisions while considering their financial situation. It also functions as an effective means of improving quality of life and preventing wasteful spending.

[0659] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0660] Step 1:

[0661] The user scans the product using a camera-equipped device. Input includes the product's barcode or QR code. The device reads it and retrieves the product information. Specifically, the camera software within the device performs barcode scanning and obtains the product identification number.

[0662] Step 2:

[0663] The terminal transmits acquired product information to the server via wireless communication technology. Wi-Fi is commonly used in this case. The input includes the product identification number, which is then forwarded to the server as output. The communication module within the terminal operates and sends the product information as data packets to the server's receiving address.

[0664] Step 3:

[0665] The server retrieves the corresponding product details from its internal database based on the received product information. The input is a product identification number, and the output includes detailed information such as the product name, price, and category. A database query is executed, and product-related data is extracted.

[0666] Step 4:

[0667] The server integrates with user-authorized economic data. Inputs include the user's purchase history and financial information, which are combined with product details. The output is an integrated dataset. Specifically, the server uses CRUD (Create, Read, Uninstall, Delete) functionality to link existing user data to product details.

[0668] Step 5:

[0669] The server utilizes a generative AI model to generate guidelines based on an integrated dataset. Product and user economic data are used as input, and personalized purchasing advice is provided as output. The generative AI model is given prompts, and the AI ​​generates corresponding guidelines through analysis.

[0670] Step 6:

[0671] The server sends the generated guidance to the terminal. Wireless communication technology is used here as well. The input includes the generated guidance, and the output is the transmission of information to the terminal. The server's transmission protocol is activated, and the generated data is sent to the terminal's receiving address.

[0672] Step 7:

[0673] The device uses augmented reality technology to visually display the received guidance to the user. The input is guidance information received from a server, and the output is displayed on the screen. AR software within the device is activated, allowing the user to view the guidance along with the product displayed through the camera.

[0674] (Application Example 1)

[0675] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] In today's brick-and-mortar stores, it's difficult for consumers to immediately understand how a product they're considering purchasing will impact their finances and long-term life goals. Traditionally, consumers typically consider the impact of their spending after the fact, which can lead to inappropriate spending. Therefore, there's a need for a system that allows consumers to intuitively understand the impact of a product on their finances before purchasing it.

[0677] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0678] In this invention, the server includes means for collecting product data, means for generating advice by combining personal accounting data and product data, and means for visually displaying the generated advice using augmented reality technology. This allows consumers to instantly understand the impact a product will have on their financial situation before purchasing it in a physical store.

[0679] An "information acquisition terminal" is a device used to recognize product data and is used by users to identify products.

[0680] A "communication network" is a network system for transmitting and receiving information, used to exchange data between information acquisition terminals and central processing units.

[0681] A "central processing unit" is a system that has processing functions to generate advice by combining product data and individual accounting data.

[0682] Augmented reality technology is a technique used to visually display generated advice by overlaying computer-generated information onto images of the real world.

[0683] "Personal consumption history" refers to records of past spending by consumers and is data used to analyze consumption trends.

[0684] A "spending plan" is a plan for future spending that consumers anticipate, and is related to budgeting and financial goals.

[0685] "Lifestyle" refers to a consumer's patterns of living and preferences, and is a factor that influences their purchasing behavior.

[0686] This invention utilizes an information acquisition terminal, a communication network, and a central processing unit. A portable device such as a smartphone or tablet is used as the information acquisition terminal, and its camera function is used to acquire product data. For example, a user can quickly obtain product information by scanning the barcode of a product they are considering purchasing in a store. This product information is transmitted to the central processing unit via the communication network.

[0687] The central processing unit combines the received product information with personal accounting data such as the user's consumption history and spending plans, and generates personalized advice using a generative AI model. In this process, the generative AI model uses machine learning models such as Hugging Face Transformers to provide advice that helps the user make the best purchase decisions.

[0688] The generated advice is visually displayed on the information acquisition terminal using augmented reality technology. By utilizing augmented reality platforms such as ARKit and ARCore, the advice is overlaid next to the product, making it easy for the user to intuitively understand. This visual display allows users to instantly recognize how the product will affect their accounting situation and future spending.

[0689] As a concrete example, suppose a user is considering purchasing a new speaker. When the user scans this speaker with their device's camera, the central processing unit receives product information and, considering past purchase history and spending plans, generates specific advice such as "It fits this month's budget" and "Purchase recommended." This generated advice is then displayed on the information acquisition device's screen using augmented reality technology.

[0690] Examples of prompts for a generative AI model include:

[0691] User's monthly budget: 50,000 yen

[0692] User's current spending: 40,000 yen

[0693] Product I'm considering buying: Speakers (¥15,000)

[0694] Planned expenses for next month and beyond: Estimates based on normal living expenses.

[0695] Please generate advice to provide to the user.

[0696] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0697] Step 1:

[0698] The user scans the product using the camera function of the information acquisition terminal. The input is image data of the product acquired by the camera. The output is product identification information, obtained by analyzing the product's barcode or QR code. Image processing libraries such as OpenCV are used for the analysis.

[0699] Step 2:

[0700] The terminal transmits product identification information to the central processing unit via the communication network. The input is the product identification information obtained in step 1. The output is a database query for product information on the server side. A REST API is used for communication to retrieve detailed information related to the product from the database.

[0701] Step 3:

[0702] The server retrieves product information from the database and combines it with the user's personal accounting data (e.g., consumption history and spending plan). The input consists of product information and the user's accounting data. The output is an integrated information set. This data processing uses libraries such as Pandas for data storage and calculations.

[0703] Step 4:

[0704] Based on the integrated information set obtained by the server, a generative AI model is used to generate personalized advice. The input is the integrated information set obtained in step 3. The output is personalized advice. Hugging Face Transformers are used as the generative AI model, and the machine learning model generates advice based on pre-configured prompt sentences.

[0705] Step 5:

[0706] The server sends the generated advice to the device. The input is the advice generated in step 4. The output is a visual representation using augmented reality technology. The REST API is used again to send data from the server, and ARKit or ARCore is used to overlay the advice onto the device's camera feed.

[0707] Step 6:

[0708] The user views advice overlaid next to the product on their device screen. The device utilizes augmented reality (AR) to intuitively display the generated advice. The input is visual data received from the server. The output is visual information of the advice organized in a way that is easy for the user to understand. This display allows the user to immediately grasp the impact the product will have on their financial situation.

[0709] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0710] This invention is a system that incorporates an emotion engine to recognize user emotions in order to support purchasing decisions for goods. When a user selects products in a store, product information is captured using a terminal. In this process, the terminal uses a camera function as an information acquisition device.

[0711] When a user scans a product selected through the camera, the device acquires the product information and transmits it to a server via the network. The server receives the product information and retrieves detailed information related to the product from its database. Furthermore, with the user's permission, the server collects the user's financial data based on purchase history and budget information.

[0712] This system incorporates an emotion engine, which allows the device to recognize emotions from the user's facial expressions and voice. This emotion data is sent to a server, which uses the emotion engine to analyze the user's emotional state. Based on the emotional state, the server uses generative AI to enhance personalized advice. In particular, if the user is feeling stressed, for example, it will generate advice encouraging them to reconsider their purchase.

[0713] The generated advice is sent from the server to the terminal, which uses augmented reality technology to visually overlay the advice next to the product the user is viewing. This allows the user to make purchasing decisions while simultaneously considering their emotions and financial situation at that moment. For example, when a user is about to purchase an expensive electronic product, the emotion engine may detect the user's state of excitement and provide advice to encourage a calmer decision.

[0714] Thus, the present invention is designed to enable users to make more balanced purchasing decisions by integrating user emotions and economic information to support product purchases. The system aims to improve the user's quality of life by providing real-time emotional and financial advice.

[0715] The following describes the processing flow.

[0716] Step 1:

[0717] The user scans the product they are considering purchasing in the store using the terminal's camera. The terminal acquires the product image or barcode and processes this information as digital data.

[0718] Step 2:

[0719] The terminal transmits the acquired product information to the server via the network. This product information includes the product ID and category information.

[0720] Step 3:

[0721] Based on the received product information, the server retrieves detailed information about the product from the database. This includes the product's price, brand, and reviews.

[0722] Step 4:

[0723] The server accesses the user's purchase history and budget information to retrieve their financial data. This data collection is conducted only within the scope of the user's prior permission.

[0724] Step 5:

[0725] The device uses a built-in emotion engine to recognize the user's emotions from their facial expressions and voice. This emotion data is sent from the device to a server, where it is used to analyze the user's current emotional state as digital data.

[0726] Step 6:

[0727] The server integrates emotional and financial data and uses generative AI to generate personalized advice for the user. For example, if a user is excited and considering purchasing an expensive item, a message encouraging calmness will be generated.

[0728] Step 7:

[0729] The generated advice is sent from the server to the terminal, and the terminal displays this advice overlaid on the camera image using AR technology.

[0730] Step 8:

[0731] Users use the provided advice to decide whether or not to continue purchasing the product. This advice includes recommendations based on their emotions and financial situation.

[0732] (Example 2)

[0733] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0734] Modern consumers must make purchasing decisions from a wide range of options, and emotions and economic circumstances can influence their decision-making. However, mechanisms for providing rational purchasing support that takes emotional states into account are not yet sufficiently developed. This can lead to inappropriate purchases and financial burdens, so there is a need for a system that can address these issues and enable consumers to make more balanced decisions.

[0735] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0736] In this invention, the server includes means for recognizing information about an item using an information acquisition device, means for transmitting the item information to the server via a network, means for generating personalized advice by combining the user's financial data and emotional data based on the item information using the server, means for recognizing the user's emotions using sensors and transmitting that data to the server, and means for visually displaying the generated advice using augmented reality technology. This enables consumers to make purchasing decisions in real time, taking into account their emotions and economic situation.

[0737] An "information acquisition device" is a device used to read information from an object and acquire it as data, and generally refers to a camera or sensor.

[0738] A "network" is a communication path that enables the sending and receiving of data, and includes the internet and dedicated lines.

[0739] A "server" refers to a computing system that processes data on a network and provides information to other devices.

[0740] "Item information" refers to detailed data about an item, including information such as product name, model number, and price.

[0741] "Financial data" refers to a user's economic information, specifically data related to income, expenses, budget, etc.

[0742] "Emotional data" refers to information about a user's emotional state, and is data analyzed by an emotion engine based on facial expressions, tone of voice, and other factors.

[0743] "Personalized advice" refers to individualized advice generated based on each user's emotional state and financial situation.

[0744] A "sensor" is a device used to detect a user's emotional state, and examples include cameras and microphones.

[0745] Augmented reality technology is a technology that overlays computer-generated information onto the real environment and displays it, and includes methods for visually displaying virtual information.

[0746] The system for implementing this invention uses a combination of an information acquisition device, a server, an emotion engine, a generative AI model, and augmented reality technology to support the user's purchasing decision-making.

[0747] When users select products using their mobile devices, they utilize the built-in camera function to scan product information. This information includes the product name, price, and model number, and the information collected by the device is transmitted to a server via the network.

[0748] The server retrieves detailed product information from a database and, after obtaining user permission, collects financial data such as purchase history and budget. Furthermore, the terminal uses a camera and microphone to collect the user's facial expressions and voice, and transmits emotional data to the server.

[0749] The server uses an emotion engine to analyze the user's emotional state and, taking financial data into consideration, generates personalized advice best suited to the user. A generative AI model is used to create advice that is appropriate to the user's emotional state, such as advice to reconsider a purchase.

[0750] For example, when a user is about to purchase an expensive electronic product, the terminal recognizes the user's excited facial expression, and the server provides advice to encourage a calm decision. This advice is overlaid on the user's field of view along with product information, using augmented reality technology.

[0751] Examples of input prompts for a generative AI model:

[0752] "User's emotional state: excited, Product: expensive electronic product, please generate purchase advice."

[0753] This system allows users to make balanced decisions in real time, taking into account both their emotions and economic circumstances.

[0754] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0755] Step 1:

[0756] The user selects an item of interest in the store and scans its information using the device's camera function. The camera reads data such as product labels and QR codes. The input is the product image captured by the camera, and the output is text data such as the product name, price, and model number.

[0757] Step 2:

[0758] The terminal transmits scanned product information to the server via the network. A common protocol (e.g., HTTP) is used for this communication. The input is the product information acquired by the terminal, and the output is the text data of the product transferred to the server.

[0759] Step 3:

[0760] The server queries the database based on the received product information to retrieve detailed product information. This step retrieves product descriptions, ratings, inventory information, and other relevant data as needed. The input is the product text data sent to the server, and the output is the detailed product information from the database.

[0761] Step 4:

[0762] The server collects financial data, such as purchase history and budget information, based on the user's consent. This data is used to form a user profile and generate personalized advice. The input is the user's permission and related information, and the output is the user's financial profile data.

[0763] Step 5:

[0764] The device recognizes the user's emotions using its camera and microphone, and analyzes their facial expressions and voice. The recognized emotion data is sent to the server in real time. The input is the user's facial expressions and voice data, and the output is text data of the emotional state generated by the emotion engine.

[0765] Step 6:

[0766] The server receives emotion data and uses an emotion engine to analyze the user's emotional state. Based on this analysis, it evaluates the user's current emotional state. The input is emotion data from the device, and the output is information about the analyzed emotional state.

[0767] Step 7:

[0768] The server uses a generative AI model to generate personalized advice based on the user's emotional state and financial data. For example, if the user is agitated, it will create advice that encourages calm decision-making. The input is the emotional state and financial data, and the output is the text of the generated advice.

[0769] Step 8:

[0770] The server sends the generated advice to the terminal over the network. The terminal uses augmented reality technology to visually overlay this advice next to the product the user is viewing. The input is the advice text sent from the server, and the output is the visual advice information displayed on the terminal.

[0771] (Application Example 2)

[0772] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] Modern consumers face a wide variety of product and service choices, and information overload and emotional bias can sometimes prevent them from making sound decisions, especially when purchasing. In this context, there is a need for systems that support consumers in making calm and balanced decisions. However, conventional systems have been unable to recognize users' emotions in real time and provide personalized advice based on those emotions.

[0774] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0775] In this invention, the server includes means for recognizing product information using an information acquisition component, means for sentiment analysis, and means for visually displaying advice generated using augmented reality technology. This makes it possible to provide advice that takes into account the user's emotional state in real time.

[0776] An "information acquisition component" is a device used to recognize information about a product, and typically includes cameras and sensors.

[0777] A "network" is the infrastructure used for communication to transmit data, and typically includes the internet and local area networks.

[0778] An "information processing device" is a computer or server that receives and analyzes data, and functions to provide appropriate advice to the user.

[0779] "Financial data" refers to economic information, including a user's budget information and past purchase history, and is used to support purchasing decisions.

[0780] "Advice" refers to the information and suggestions provided to users when selecting a product, and is generated based on sentiment analysis and financial data.

[0781] "Emotional analysis methods" are technologies that analyze a user's facial expressions and voice to recognize their emotional state, thereby enabling the personalization of the user experience.

[0782] Augmented reality technology is a technique that overlays computer-generated information onto the real world, and is used to visually display advice next to the product the user is viewing.

[0783] The system for carrying out this invention utilizes an information acquisition component, a network, an information processing device, emotion analysis means, and augmented reality technology. The user uses a terminal such as a smartphone or smart glasses to scan the product's barcode or QR code and acquire product information. The terminal transmits this information to the information processing device via the network. This information processing device acquires the user's financial data along with the product information, and further analyzes the user's facial expressions and voice through the terminal's camera and microphone using emotion analysis means.

[0784] On the information processing device, the user's emotional state is evaluated based on the acquired data, and personalized advice is generated that matches that state. A generative AI model is used to output the optimal advice from this data. As an example of a specific prompt, the AI ​​is instructed to "Generate advice to refrain from making a purchase decision when the user is stressed, based on the user's facial expression data."

[0785] Furthermore, augmented reality technology is used to visually display advice on the smart glasses' screen. This allows users to receive advice on product selection that takes into account their emotions and financial situation in real time. For example, when a user is about to purchase expensive sneakers, the information processing device can detect the user's excited state and provide advice to encourage calmer judgment and suggest considering other options.

[0786] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0787] Step 1:

[0788] The user uses the device's camera function to scan the product's barcode or QR code. Visual information of the product is obtained as input. Based on this information, the device extracts and displays the product's basic information. The output is the product information, which is then passed on to the next processing step.

[0789] Step 2:

[0790] The terminal transmits acquired product information to the server via the network. It uses product information as input and sends the information to the server according to a data transmission protocol. The output is the product information stored on the server.

[0791] Step 3:

[0792] The server retrieves the user's financial data from the database. It uses the user's identification information as input and cross-references it with the user's past purchase history and budget information. The output is the user's financial data, which is then used in the subsequent sentiment analysis.

[0793] Step 4:

[0794] The device acquires the user's facial expressions and voice using emotion analysis techniques. It uses camera and microphone data as input and analyzes the user's emotional state using an emotion recognition algorithm. The output is emotion data, which is sent to the server.

[0795] Step 5:

[0796] The server integrates emotional and financial data and uses a generative AI model to generate personalized advice based on the user's emotional state. The input consists of emotional and financial data, which are then processed by the AI ​​model to create the advice. The output is the generated advice.

[0797] Step 6:

[0798] The terminal displays generated advice received from the server within the user's field of view using augmented reality technology. It uses the received advice data as input and visually displays it next to the product the user is viewing. The output is the augmented reality display information.

[0799] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0800] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0801] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0802] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0803] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0804] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0805] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0806] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0807] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0808] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0809] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0810] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0811] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0812] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0813] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0814] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0815] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0816] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0817] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0818] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0819] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0820] The following is further disclosed regarding the embodiments described above.

[0821] (Claim 1)

[0822] A means for recognizing information about an item using an information acquisition device,

[0823] A means of transmitting item information to a server via a network,

[0824] A means of generating advice based on item information using a server, combined with the user's financial data,

[0825] A means of visually displaying advice generated using augmented reality technology,

[0826] A system that includes this.

[0827] (Claim 2)

[0828] The system according to claim 1 for obtaining a user's purchase history and budget information.

[0829] (Claim 3)

[0830] The system according to claim 1, wherein the generated advice is based on the user's lifestyle and goals.

[0831] "Example 1"

[0832] (Claim 1)

[0833] A means for identifying products using an information acquisition device,

[0834] A means for transmitting product information to a data processing device via wireless communication technology,

[0835] A means of creating guidelines that integrate product information with the user's economic data using a data processing device,

[0836] A means of visually displaying guidelines created using augmented reality technology,

[0837] A system that includes this.

[0838] (Claim 2)

[0839] The system according to claim 1 for obtaining a user's purchase history and budget information.

[0840] (Claim 3)

[0841] The system according to claim 1, wherein the guidelines created are based on the user's lifestyle and goals.

[0842] "Application Example 1"

[0843] (Claim 1)

[0844] A means of recognizing product data using an information acquisition terminal,

[0845] A means for transmitting product data to a central processing unit via a communication network,

[0846] A means of generating advice based on product data using a central processing unit, combined with individual accounting data,

[0847] A means of visually displaying advice generated using augmented reality technology,

[0848] A system that includes this.

[0849] (Claim 2)

[0850] The system according to claim 1 for obtaining an individual's consumption history and spending plan.

[0851] (Claim 3)

[0852] The system according to claim 1, wherein the generated advice is based on an individual's lifestyle and goals, and operates using a smart device.

[0853] "Example 2 of combining an emotion engine"

[0854] (Claim 1)

[0855] A means for recognizing information about an item using an information acquisition device,

[0856] A means of transmitting item information to a server via a network,

[0857] A method for generating personalized advice by combining user financial data and emotional data based on item information using a server,

[0858] A means of recognizing a user's emotions using sensors and transmitting that data to a server,

[0859] A means of visually displaying the generated advice using augmented reality technology,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] The system according to claim 1, which acquires a user's purchase history and budget information and evaluates their emotional state.

[0863] (Claim 3)

[0864] The system according to claim 1, wherein the generated advice is based on the user's emotional state and financial situation.

[0865] "Application example 2 when combining with an emotional engine"

[0866] (Claim 1)

[0867] A means for recognizing product information using an information acquisition component,

[0868] Means for transmitting product information to an information processing device via a network,

[0869] A means of generating advice based on product information using an information processing device, combined with the user's financial data,

[0870] A means of sentiment analysis for detecting the user's emotional state and personalizing advice based on that emotional state,

[0871] A means of visually displaying advice generated using augmented reality technology,

[0872] A system that includes this.

[0873] (Claim 2)

[0874] The system according to claim 1, which obtains a user's past purchase history and budget information.

[0875] (Claim 3)

[0876] The system according to claim 1, wherein the generated advice is based on the user's lifestyle and goals, and takes into account the user's emotional state. [Explanation of Symbols]

[0877] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for recognizing information about an item using an information acquisition device, A means of transmitting item information to a server via a network, A means of generating advice based on item information using a server, combined with the user's financial data, A means of visually displaying advice generated using augmented reality technology, A system that includes this.

2. The system according to claim 1, which acquires a user's purchase history and budget information.

3. The system according to claim 1, wherein the generated advice is based on the user's lifestyle and goals.

Images