system

Abstract

We provide the system. [Solution] The information processing device includes means for analyzing the user's past purchase history and preferences and generating calculation rules for suggesting the optimal product, A means for generating product suggestions based on the user's current location and situation, based on the aforementioned calculation rules, A means of presenting generated product suggestions to users on mobile devices using augmented reality technology, A method for recognizing products using a camera and displaying them as superimposed visual information, A means for obtaining the user's choice behavior and updating the calculation rules based thereon, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When consumers go shopping, there is a problem that it takes time to select products and they cannot quickly obtain desired product information. Also, there is a problem that it is difficult for retailers to make effective product proposals based on consumers' preferences. This invention aims to solve these problems by providing a system that individualizes consumers' purchase experiences and makes efficient product proposals.

Means for Solving the Problems

[0005] The system according to the present invention provides a means for analyzing a consumer's past purchase history and preferences using an information processing device and generating an optimal product suggestion algorithm based on that data. Furthermore, it can suggest products according to the consumer's current location and situation, and present these suggestions to a mobile device using augmented reality technology. In addition, by acquiring consumer selection behavior in real time and improving the suggestion algorithm based on the acquired data, a more accurate personalized shopping experience is achieved. This allows consumers to quickly and easily grasp the necessary information, and enables retailers to make product suggestions that match consumer preferences.

[0006] An "information processing device" is a computer system that has the ability to collect and analyze user data and generate appropriate product recommendations.

[0007] "User" refers to an individual or legal entity that uses this system to purchase goods.

[0008] "Purchase history" refers to records of products and services that a user has purchased in the past.

[0009] "Preferences" refer to the individual tastes and tendencies of users regarding products and services they particularly like.

[0010] An "algorithm" is a set of steps or computational rules designed to perform a specific task.

[0011] "Mobile devices" refer to electronic devices that users can easily carry with them, such as mobile phones, smartphones, or tablets.

[0012] Augmented reality technology is a technology that overlays digital information onto the real world.

[0013] "Selection behavior" refers to the specific reactions and choices that users make in response to proposed products.

[0014] The "proposal algorithm" is a calculation method for selecting and presenting appropriate product information according to the interests and needs of users.

[0015] "Product proposal" is an act of providing product information and promotions judged to be beneficial to users.

Brief Explanation of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Modes for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0020] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention relates to an information processing system for providing consumers with a personalized shopping experience. The system includes an information processing device, a mobile terminal, and a communication network for exchanging data between them.

[0038] Information processing device (server)

[0039] The server has the function of collecting and analyzing users' past purchase history and preference information. Based on this, it learns the user's purchasing patterns using machine learning algorithms. Through this learning, a customized product recommendation algorithm is generated for each user. Furthermore, the server obtains the latest product and promotion information from retail stores and product providers and evaluates its relevance to the user's preferences.

[0040] Mobile devices

[0041] Mobile devices (such as smartphones and AR glasses) play a role in providing users with suggested information received from the server in real time. Specifically, when a product in a store comes into view, it displays the product's characteristics, discount information, and similarity to purchase history overlaid using augmented reality technology.

[0042] User operation

[0043] Users view product information suggested during shopping via their mobile devices and provide feedback on items that interest them. This includes product selection, evaluation, and whether or not they make a purchase. This feedback data is sent from the device to the server and used to improve the product suggestion algorithm for future purchases, resulting in more refined personalization.

[0044] Specific example

[0045] For example, imagine a user is using their smartphone in a supermarket. The server analyzes the user's past purchase history of health-conscious foods and selects newly arrived organic products and related items on sale. This information is displayed via the smartphone's AR function, associated with the products the user is viewing on the shelf. When the user selects items they are interested in and adds them to their cart, the selection information is sent to the server and used to improve the accuracy of future recommendations.

[0046] Thus, the present invention makes the consumer shopping experience more efficient and personalized, resulting in time savings and increased satisfaction.

[0047] The following describes the processing flow.

[0048] Step 1:

[0049] The server collects users' past purchase history and preference data from a database and inputs this data into a machine learning algorithm for analysis. Through this analysis, it identifies users' purchasing patterns and preferences and generates a product recommendation algorithm based on them.

[0050] Step 2:

[0051] The server retrieves the latest product and promotional information from retail stores and product providers, compares it with analyzed preference data, and creates a product list tailored to the user. This information is then compiled into data packets and sent to the user's mobile device.

[0052] Step 3:

[0053] The terminal analyzes product suggestion information received from the server and associates it with products in the store where the user is currently located. When the user views a specific product through the terminal, additional information and promotions related to that product are displayed using augmented reality technology.

[0054] Step 4:

[0055] Users use their devices to view product information that interests them. They also provide feedback through actions such as selecting products, adding them to their cart, and providing ratings. The data collected during this process serves as a valuable indicator of purchasing intent and preferences.

[0056] Step 5:

[0057] The device sends feedback data about the user's choices to the server. This allows the server to update its suggestion algorithm, further improving the accuracy of future suggestions. Through this cyclical process, continuous personalization is achieved.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] Traditional shopping systems struggle to efficiently provide personalized product recommendations to users, resulting in a lack of timely information delivery. Furthermore, users cannot access detailed information about products they are interested in from the suggested items in real time, leaving a lack of methods to provide a highly satisfying shopping experience.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] This invention includes a server that collects the user's past purchase history and preference information and generates an optimal product suggestion algorithm through data analysis; a server that generates personalized product suggestions for the user and transmits them to a mobile terminal; and a mobile terminal that uses augmented reality technology to present the received product suggestion information to the user in real time and visually displays product characteristics and discount information. As a result, the user can receive efficient and personalized product suggestions, check detailed product information in real time, and achieve a highly satisfying shopping experience.

[0063] An "information processing device" is a computer or the entire system used to collect, analyze, and process data.

[0064] "Purchase history" refers to records of products that a user has purchased in the past, as well as related information.

[0065] "Preference information" refers to data that indicates a user's preferences and interests in products.

[0066] A "product suggestion algorithm" is a mathematical or programmatic method for suggesting the most suitable products based on a user's purchase history and preference information.

[0067] A "mobile device" refers to a portable electronic device capable of receiving and displaying information, and specifically includes smartphones and augmented reality devices.

[0068] Augmented reality technology is a technology that overlays digital information onto the real world, providing information through the user's sight or hearing.

[0069] "Real-time" refers to a time frame in which information is processed almost immediately after it is generated or received, with virtually no delay.

[0070] "Latest product inventory" refers to the most up-to-date data on the current status and quantity of products at the seller.

[0071] "Promotional information" refers to marketing data related to product sales, discounts, and campaigns.

[0072] A "generative AI model" is a trained model that uses artificial intelligence technology to provide the optimal output for a given input.

[0073] A "prompt" refers to an instruction or input text given to a generative AI model.

[0074] The present invention is a system that provides users with a personalized shopping experience through a system including an information processing device, a mobile terminal, and a communication network.

[0075] First, the server functions as an information processing device, collecting users' past purchase history and preference information. This information is stored in a database and used through data analysis. The server uses programming languages ​​such as Python and R, as well as machine learning libraries (e.g., TENSORFLOW® and scikit-learn), to analyze user data and generate an optimal product recommendation algorithm. This algorithm provides different recommendations for each user.

[0076] The server also retrieves the latest inventory and promotional information from retailers in real time via the internet. APIs are used for this retrieval, and the data includes new product information and discount information. This information, along with a product recommendation algorithm, is then presented to the user.

[0077] Next, the server sends the generated product suggestions to a mobile device (e.g., a smartphone). HTTPS is used as the communication protocol here, ensuring that data is exchanged securely.

[0078] The device displays received product suggestion information to the user using augmented reality (AR) technology. The device has apps and frameworks (e.g., ARCore and ARKit) installed to support AR technology. When the user visually checks products in a physical store, the device overlays digital information onto the product. This provides the user with product characteristics and discount information, saving time and increasing satisfaction.

[0079] As a concrete example, when a user is shopping at a supermarket, the server analyzes the user's purchasing patterns and suggests new organic products and current sale items to users who prefer health-conscious foods. This information is displayed on the shelves via the AR function of the mobile device. If the user becomes interested in a product, selects it, and adds it to their cart, that information is sent to the server in real time and used to improve the accuracy of future suggestions.

[0080] An example of a prompt message is, "Based on past purchase history, please suggest health-conscious products."

[0081] In this way, the present invention efficiently provides personalized product suggestions tailored to the user's preferences, thereby realizing a comfortable shopping experience.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The server collects the user's past purchase history and preference information from a database. The user's ID is provided as input, and relevant historical data is retrieved based on it. This data includes purchased items, purchase date and time, ratings, etc., and is then cleansed to remove noise and prepared for analysis.

[0085] Step 2:

[0086] The server analyzes collected purchase history and preference information. The input data includes cleansed purchase history, which the server analyzes using a machine learning library (e.g., TensorFlow). This analysis identifies user purchasing patterns and extracts features. The output generates a preference model for each user.

[0087] Step 3:

[0088] The server creates a product suggestion algorithm based on the generated preference model. The preference model and product data are used as input, and the algorithm is programmed using libraries such as scikit-learn. The output is a product list optimized for a specific user.

[0089] Step 4:

[0090] The server retrieves the latest product inventory and promotional information from retailers via API. The input includes the access key for the vendor's API. This retrieval updates new product and special offer information, which is then integrated into the suggestion algorithm. The output is a list of product suggestions reflecting the latest information.

[0091] Step 5:

[0092] The server sends optimized product recommendations to the mobile device. Inputs include the generated product list and the user's device information, and the data is securely transmitted via the HTTPS protocol. The device receives this information in real time and notifies the user. Output is a notification or display on the device.

[0093] Step 6:

[0094] The terminal displays received product suggestion information using augmented reality technology. The input data includes a list of product suggestions. The terminal utilizes ARCore and ARKit to overlay the suggestions onto products visible in the user's field of view. This allows the user to simultaneously view digital information while viewing physical products. The output is a visual presentation of product information using augmented reality.

[0095] Step 7:

[0096] The user reviews product information and selects items that interest them. The information displayed on the terminal serves as input, and based on this, the user adds items to their cart or leaves a review. The selected data is sent from the terminal to the server as feedback, and the output is feedback information used to improve the next suggestion algorithm.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] Modern consumers have access to a wide variety of product information, but conversely, they find it difficult to find products that suit them from the vast number of options. Especially in physical stores, there is a demand for personalized product suggestions based on the customer's purchase history and preferences, but systems that reflect individual preferences and past history in real time have not yet been fully realized. Therefore, it is necessary to provide an efficient and personalized shopping experience.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating calculation rules for suggesting the optimal product; means for generating product suggestions based on the user's current location and situation; and means for presenting the generated product suggestions to the user on a mobile terminal using augmented reality technology. This makes it possible for the user to accurately obtain information that matches their preferences when selecting a product.

[0102] An "information processing device" is a device that performs data processing to analyze a user's purchase history and preferences, and to generate calculation rules for product recommendations.

[0103] "Operation rules" refer to sequences of numbers or logical procedures used to select and suggest the most suitable products based on the user's behavioral history.

[0104] A "portable device" is an electronic device that can be carried by the user and has the function of visually displaying product suggestions.

[0105] Augmented reality technology is a technology that overlays virtual information onto visual information from the real world.

[0106] A "camera" is a device that uses optical sensors to record images of real objects and environments.

[0107] "Visualized information" refers to information that has been converted from digital data into a form that can be visually perceived by the user.

[0108] "Selection behavior" refers to a series of actions taken by users to choose products that interest them.

[0109] A "retail store" is a physical business facility for selling goods directly to consumers.

[0110] "Product information" refers to information about a product's specifications and price that consumers use as a reference when considering a purchase.

[0111] "Goods" refers to physical products that are the subject of a transaction.

[0112] The system for implementing this invention combines a server as an information processing device with a smartphone or AR-enabled device as a mobile terminal. The server analyzes the user's purchase history and preference information stored in a database and generates computational rules using a generated AI model based on this analysis. Specifically, it utilizes machine learning platforms such as TensorFlow to analyze purchase patterns and suggest products.

[0113] The terminal uses augmented reality technologies such as ARCore to overlay product suggestion information received from the server onto the user's environment. When a smartphone camera scans a product on a shelf, sales promotion data is displayed in real time as visualized information. This allows users to efficiently select products based on visually presented information within the store.

[0114] As a concrete example, when a user uses their smartphone in the beverage section of a grocery store, the device recommends low-calorie drinks based on their past purchase history and displays them using augmented reality (AR). In this case, an example of a prompt message that would be processed on the server might be, "Based on the product shelf scanned by the user, please suggest low-calorie beverages using AR, taking into account their past purchase history and health-conscious preferences."

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The server retrieves user purchase history and preference information from the database. Given a user ID as input, it extracts past purchase transaction data via SQL queries. Based on this data, a machine learning model analyzes purchase patterns and creates algorithmic rules to generate new product suggestions.

[0118] Step 2:

[0119] Based on the generated calculation rules, the server provides optimal product recommendations by comparing them with the user's current location information. Using GPS data to determine the user's location, it retrieves the latest product and promotional information from the relevant store and creates a customized product recommendation list for the user. The output is a set of product information tailored to the user's preferences.

[0120] Step 3:

[0121] The terminal overlays product suggestion information received from the server onto the in-store video captured by the camera. When a user scans a shelf using their smartphone camera, the terminal receives that video data as input, and product suggestions are displayed in real time using AR technology. As output, a visual augmented reality display is generated and presented to the user.

[0122] Step 4:

[0123] The user makes a selection of products that interest them. The terminal retrieves this selection information and sends it back to the server. The server receives the identifiers and evaluation data of the selected products as input and feeds this information back into the database to be used for future purchase suggestions. As output, a more refined user profile is updated.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention is a system for highly personalized user purchasing experiences, and includes an information processing device (server), a mobile terminal, and communication means for exchanging data between them. Furthermore, by incorporating an emotion engine, it enables product recommendations that take into account the user's emotional state.

[0126] server

[0127] The server collects and analyzes users' past purchase history and preference information. Based on the analysis results, it generates an optimal product recommendation algorithm using machine learning algorithms. This algorithm enables personalized product recommendations for each user. In addition, the server dynamically adjusts the recommendations based on user sentiment data acquired in real time.

[0128] Mobile devices

[0129] The mobile device plays the role of presenting product suggestions received from the server to the user in real time. Furthermore, it is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. Based on this information, it adjusts the content and order of the products presented, creating an optimal purchasing experience tailored to the user's emotions.

[0130] User operation

[0131] Users access product information through augmented reality technology using their mobile devices. While providing general feedback such as product selection and purchasing behavior, emotional data is also collected in real time. This allows the system to continuously suggest products tailored to the user's emotional state.

[0132] Specific example

[0133] For example, imagine a user visiting a supermarket and checking product information on their mobile device. The emotion engine analyzes the user's facial expressions and prioritizes displaying detailed information and related products for items the user finds interesting. Furthermore, if it detects the user is tired, it suggests relaxing products and promotions to provide a more comfortable shopping experience.

[0134] Thus, the present invention provides a system that enables highly personalized product recommendations that take into account the user's emotional state, thereby improving and optimizing the shopping experience.

[0135] The following describes the processing flow.

[0136] Step 1:

[0137] The server collects users' past purchase history and preference information from a database. This data is then analyzed using machine learning algorithms to generate a product recommendation algorithm optimized for each user.

[0138] Step 2:

[0139] The server retrieves the latest product and promotional information from retail stores, combines it with analyzed preference data, and creates a ranked product list to provide to the user. This list is then sent to the mobile device.

[0140] Step 3:

[0141] The terminal receives a product list sent from the server and uses an emotion engine to acquire emotional data in real time from the user's facial expressions and voice. This allows the terminal to analyze the user's emotional state and adjust product recommendations accordingly.

[0142] Step 4:

[0143] The device uses augmented reality technology to display information and promotions related to the products the user is viewing. Based on the user's emotional state, it prioritizes displaying product information that is likely to interest the user.

[0144] Step 5:

[0145] Users select and evaluate products and input feedback via a device. The device then sends this feedback information and acquired sentiment data to a server.

[0146] Step 6:

[0147] The server analyzes the emotional data and feedback sent from the terminal and updates the product recommendation algorithm. This results in more personalized and optimized recommendations for subsequent uses, tailored to the user's emotional state.

[0148] (Example 2)

[0149] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0150] In today's information society, personalizing the consumer purchasing experience is crucial. However, conventional systems rely solely on past history and preference information to suggest products, failing to consider consumers' real-time emotional states and thus hindering the provision of optimal recommendations. Furthermore, the inability to quickly update recommendations to reflect consumers' latest needs and emotions limits the potential for improving customer satisfaction.

[0151] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0152] In this invention, the server includes means for analyzing the user's past purchase history and preferences and generating an algorithm for making personalized product suggestions using machine learning; means for analyzing the user's emotional state in real time and adjusting product suggestions accordingly; and means for updating the algorithm based on emotional data collected by a mobile terminal. This enables highly customized product suggestions that are tailored to the user's emotions and environment.

[0153] An "information processing device" is a computer system used to collect and analyze data and generate and execute algorithms based on specific purposes.

[0154] "Machine learning" is a type of artificial intelligence technology that automatically learns patterns and relationships from large amounts of data to make predictions and decisions.

[0155] "Emotional state" refers to information about the user's psychological state and mood at that time, obtained from their facial expressions, voice, etc.

[0156] An "algorithm" refers to a set of procedures or computational methods for solving a specific problem, and in this context, it refers to the processing procedures for making a product proposal.

[0157] Augmented reality technology is a technique that overlays computer-generated information onto images of the real world.

[0158] A "mobile device" refers to an electronic device that is portable by the user and capable of communication.

[0159] "Product suggestions" refer to the purchase options and related information presented to the user, and are usually structured based on the user's preferences and circumstances.

[0160] "Emotional data" refers to information about the emotional state of users, and is collected using sensors and analytical technologies.

[0161] This invention is a system for personalizing the user's purchasing experience and includes a server, terminals, and communication means to link them. First, the server collects the user's past purchase history and preferences and stores them in a database. For example, necessary information can be extracted from the database using SQL. The collected data is input into a machine learning algorithm using Python or similar to generate a model for product recommendations optimized for the user. Clustering and regression analysis may be performed using the scikit-learn library.

[0162] The terminal displays product suggestions received from the server to the user in real time. The terminal also incorporates an emotion engine that uses a camera and microphone to detect the user's facial expressions and voice, analyzing their emotional state. Facial recognition can be implemented using TensorFlow or OpenCV. This emotion data is sent to the server and used for immediate adjustments to product suggestions.

[0163] Users can view product information using augmented reality technology via their devices. During this process, users can select products they are interested in, and their feedback is collected by the device. This feedback data is sent to a server and used to improve the product recommendation algorithm.

[0164] For example, suppose a user visits a supermarket and uses a terminal to view product information. The emotion engine detects the user's interest and prioritizes displaying detailed information about specific products. If the system determines that the user is tired, it suggests promotions for products with relaxing effects. A specific prompt might be: "If the emotion engine detects that a user shopping at a supermarket is tired, please explain, with product examples, how product suggestions should be presented."

[0165] In this way, the system optimizes the purchasing experience by taking into account the user's real-time emotions and providing personalized suggestions.

[0166] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0167] Step 1:

[0168] The server collects the user's past purchase history and preferences from a database. The user ID is provided as input, and the relevant information is extracted from the database using an SQL query. The output is a dataset of the user's past purchase history and preference information. This dataset is preprocessed for analysis, including standardization and handling of missing values.

[0169] Step 2:

[0170] The server trains a machine learning algorithm using the collected dataset to generate a personalized product recommendation model. A pre-processed dataset is used as input. The scikit-learn library in Python is used for data analysis and algorithm training, including clustering and regression analysis. The output is a product recommendation model optimized for the user, preparing the system for product recommendations based on user preferences.

[0171] Step 3:

[0172] The terminal presents product suggestions to the user based on those received from the server. Product information is displayed on the terminal as output. Optimized product suggestions from the server are included as input. The terminal uses augmented reality technology to provide product information to the user visually. When the user shows interest in a product, the camera and AR functions are used to display detailed product information.

[0173] Step 4:

[0174] The device analyzes the user's emotions in real time. Data from the device's built-in camera and microphone is used as input. The device processes this data using an emotion analysis engine, employing TensorFlow and OpenCV for facial recognition. The output is data indicating the user's emotional state. This data is used to adjust the content and priority of product recommendations.

[0175] Step 5:

[0176] The server dynamically adjusts product recommendations based on real-time sentiment data. It uses sentiment state data as input. The server updates its product recommendation model and selects products appropriate to the user's current situation. The output is the updated product recommendations, which are then resent to the terminal. This creates a shopping experience that harmonizes with the user's emotions.

[0177] Step 6:

[0178] Users select products through their devices, and their behavioral data is collected in real time. User actions are captured as input. The device sends the selected product information to a server, which is used as feedback data to improve the algorithm. The output is feedback data, contributing to improved accuracy in product recommendations.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] Modern retail stores are required to provide a shopping experience optimized for consumers' diverse preferences and dynamic emotional changes. However, conventional product recommendation systems lack the ability to suggest products in real time while considering consumer emotions, making it difficult to provide a truly satisfying service to individual consumers. Therefore, a new system is needed to increase consumer purchasing intent and improve the in-store experience.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating an algorithm for suggesting optimal products; means for analyzing the user's emotional state using an emotion recognition engine and dynamically adjusting the product suggestions; and means for presenting the generated product suggestions to the user using augmented reality technology via a communication device. This enables personalized product suggestions that respond to the consumer's emotions, thereby improving the purchasing experience at retail stores.

[0184] An "information processing device" is a device that performs analysis and generates algorithms based on user data, and also enables data communication with external parties.

[0185] An "algorithm" is a series of calculation procedures designed to provide optimal product recommendations by taking into account the user's purchase history and preferences.

[0186] An "emotion recognition engine" is a system that analyzes a user's facial expressions and voice to evaluate their emotional state in real time.

[0187] A "communication device" is a device that sends and receives data between a server and other devices.

[0188] Augmented reality technology is a technology that overlays digital information onto the real world, enabling users to experience three-dimensional and interactive information presentations.

[0189] "Product recommendations" refer to information about products that are presented to users based on analyzed data and are recommended for purchase.

[0190] A "consumer" is a customer who purchases goods at a retail store.

[0191] In order to implement this invention, it is necessary to build a system by combining three main elements: a server, a communication terminal, and an emotion recognition engine.

[0192] The server is responsible for analyzing users' past purchase history and preferences. Specifically, it retrieves purchase history data from the database and uses machine learning algorithms to generate product recommendation algorithms optimized for each user. Possible software options include machine learning frameworks such as TensorFlow and PyTorch.

[0193] A communication terminal is a device that presents proposed product information to the user through augmented reality technology. Examples include smart glasses and smartphones. These devices receive information from a server and simultaneously overlay digital information onto the user's real-world field of view. Specific hardware examples include devices such as the Oculus Quest.

[0194] The emotion recognition engine analyzes the user's facial expressions and voice to evaluate their emotional state in real time. This allows the server to dynamically adjust product recommendations based on this emotional data. Technologies used include services such as Microsoft® Azure® Face Recognition API and Google® Cloud Vision API.

[0195] Users use communication terminals within retail stores to view product information presented using augmented reality technology. The suggestions are tailored to the user's emotions, improving shopping satisfaction. For example, when a user is in the book section, smart glasses display information on related books and offer new suggestions based on their emotional response.

[0196] An example of a prompt sentence to input into a generative AI model would be, "When a user is looking at an interesting product, what are some effective ways to suggest related product information?"

[0197] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0198] Step 1:

[0199] The server retrieves user purchase history and preference data from a database. This data is then analyzed using a machine learning algorithm to generate a product recommendation algorithm optimized for each user. The output is a customized product recommendation algorithm tailored to each individual user.

[0200] Step 2:

[0201] The server retrieves the latest product information from an online database and incorporates it into a product suggestion algorithm generated for each user. During this process, the algorithm is optimized to include product inventory status and promotional information. The output of this process is updated product suggestion information.

[0202] Step 3:

[0203] The communication terminal receives product suggestion information transmitted from the server. The terminal uses this as input and presents it to the user as appropriate visual content using augmented reality technology. As output, product information is displayed integrated into the user's field of view.

[0204] Step 4:

[0205] The emotion recognition engine acquires the user's facial expressions and voice in real time from the camera and microphone, and uses this data as input to analyze their emotional state. This allows the system to output emotional information such as whether the user is interested or tired.

[0206] Step 5:

[0207] The server dynamically updates product recommendation information based on emotional information from the emotion recognition engine. This process involves inference by a generative AI model, reconstructing product recommendations based on prompt text. The output is the most up-to-date product recommendation tailored to the user's emotions.

[0208] Step 6:

[0209] Users access related products and additional information based on the product information displayed via their communication terminal. As output, feedback regarding product selection is sent to the server and incorporated into future suggestions.

[0210] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0211] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0212] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0213] [Second Embodiment]

[0214] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0215] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0216] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0217] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0218] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0219] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0220] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0221] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0222] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0223] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0224] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0225] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0226] This invention relates to an information processing system for providing consumers with a personalized shopping experience. The system includes an information processing device, a mobile terminal, and a communication network for exchanging data between them.

[0227] Information processing device (server)

[0228] The server has the function of collecting and analyzing users' past purchase history and preference information. Based on this, it learns the user's purchasing patterns using machine learning algorithms. Through this learning, a customized product recommendation algorithm is generated for each user. Furthermore, the server obtains the latest product and promotion information from retail stores and product providers and evaluates its relevance to the user's preferences.

[0229] Mobile devices

[0230] Mobile devices (such as smartphones and AR glasses) play a role in providing users with suggested information received from the server in real time. Specifically, when a product in a store comes into view, it displays the product's characteristics, discount information, and similarity to purchase history overlaid using augmented reality technology.

[0231] User operation

[0232] Users view product information suggested during shopping via their mobile devices and provide feedback on items that interest them. This includes product selection, evaluation, and whether or not they make a purchase. This feedback data is sent from the device to the server and used to improve the product suggestion algorithm for future purchases, resulting in more refined personalization.

[0233] Specific example

[0234] For example, imagine a user is using their smartphone in a supermarket. The server analyzes the user's past purchase history of health-conscious foods and selects newly arrived organic products and related items on sale. This information is displayed via the smartphone's AR function, associated with the products the user is viewing on the shelf. When the user selects items they are interested in and adds them to their cart, the selection information is sent to the server and used to improve the accuracy of future recommendations.

[0235] Thus, the present invention makes the consumer shopping experience more efficient and personalized, resulting in time savings and increased satisfaction.

[0236] The following describes the processing flow.

[0237] Step 1:

[0238] The server collects users' past purchase history and preference data from a database and inputs this data into a machine learning algorithm for analysis. Through this analysis, it identifies users' purchasing patterns and preferences and generates a product recommendation algorithm based on them.

[0239] Step 2:

[0240] The server retrieves the latest product and promotional information from retail stores and product providers, compares it with analyzed preference data, and creates a product list tailored to the user. This information is then compiled into data packets and sent to the user's mobile device.

[0241] Step 3:

[0242] The terminal analyzes product suggestion information received from the server and associates it with products in the store where the user is currently located. When the user views a specific product through the terminal, additional information and promotions related to that product are displayed using augmented reality technology.

[0243] Step 4:

[0244] Users use their devices to view product information that interests them. They also provide feedback through actions such as selecting products, adding them to their cart, and providing ratings. The data collected during this process serves as a valuable indicator of purchasing intent and preferences.

[0245] Step 5:

[0246] The device sends feedback data about the user's choices to the server. This allows the server to update its suggestion algorithm, further improving the accuracy of future suggestions. Through this cyclical process, continuous personalization is achieved.

[0247] (Example 1)

[0248] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0249] Traditional shopping systems struggle to efficiently provide personalized product recommendations to users, resulting in a lack of timely information delivery. Furthermore, users cannot access detailed information about products they are interested in from the suggested items in real time, leaving a lack of methods to provide a highly satisfying shopping experience.

[0250] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0251] This invention includes a server that collects the user's past purchase history and preference information and generates an optimal product suggestion algorithm through data analysis; a server that generates personalized product suggestions for the user and transmits them to a mobile terminal; and a mobile terminal that uses augmented reality technology to present the received product suggestion information to the user in real time and visually displays product characteristics and discount information. As a result, the user can receive efficient and personalized product suggestions, check detailed product information in real time, and achieve a highly satisfying shopping experience.

[0252] An "information processing device" is a computer or the entire system used to collect, analyze, and process data.

[0253] "Purchase history" refers to records of products that a user has purchased in the past, as well as related information.

[0254] "Preference information" refers to data that indicates a user's preferences and interests in products.

[0255] A "product suggestion algorithm" is a mathematical or programmatic method for suggesting the most suitable products based on a user's purchase history and preference information.

[0256] A "mobile device" refers to a portable electronic device capable of receiving and displaying information, and specifically includes smartphones and augmented reality devices.

[0257] Augmented reality technology is a technology that overlays digital information onto the real world, providing information through the user's sight or hearing.

[0258] "Real-time" refers to a time frame in which information is processed almost immediately after it is generated or received, with virtually no delay.

[0259] "Latest product inventory" refers to the most up-to-date data on the current status and quantity of products at the seller.

[0260] "Promotional information" refers to marketing data related to product sales, discounts, and campaigns.

[0261] A "generative AI model" is a trained model that uses artificial intelligence technology to provide the optimal output for a given input.

[0262] A "prompt" refers to an instruction or input text given to a generative AI model.

[0263] The present invention is a system that provides users with a personalized shopping experience through a system including an information processing device, a mobile terminal, and a communication network.

[0264] First, the server functions as an information processing device, collecting users' past purchase history and preference information. This information is stored in a database and used through data analysis. The server uses programming languages ​​such as Python and R, as well as machine learning libraries (e.g., TensorFlow and scikit-learn), to analyze user data and generate an optimal product recommendation algorithm. This algorithm provides different recommendations for each user.

[0265] The server also retrieves the latest inventory and promotional information from retailers in real time via the internet. APIs are used for this retrieval, and the data includes new product information and discount information. This information, along with a product recommendation algorithm, is then presented to the user.

[0266] Next, the server sends the generated product suggestions to a mobile device (e.g., a smartphone). HTTPS is used as the communication protocol here, ensuring that data is exchanged securely.

[0267] The device displays received product suggestion information to the user using augmented reality (AR) technology. The device has apps and frameworks (e.g., ARCore and ARKit) installed to support AR technology. When the user visually checks products in a physical store, the device overlays digital information onto the product. This provides the user with product characteristics and discount information, saving time and increasing satisfaction.

[0268] As a concrete example, when a user is shopping at a supermarket, the server analyzes the user's purchasing patterns and suggests new organic products and current sale items to users who prefer health-conscious foods. This information is displayed on the shelves via the AR function of the mobile device. If the user becomes interested in a product, selects it, and adds it to their cart, that information is sent to the server in real time and used to improve the accuracy of future suggestions.

[0269] An example of a prompt message is, "Based on past purchase history, please suggest health-conscious products."

[0270] In this way, the present invention efficiently provides personalized product suggestions tailored to the user's preferences, thereby realizing a comfortable shopping experience.

[0271] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0272] Step 1:

[0273] The server collects the user's past purchase history and preference information from a database. The user's ID is provided as input, and relevant historical data is retrieved based on it. This data includes purchased items, purchase date and time, ratings, etc., and is then cleansed to remove noise and prepared for analysis.

[0274] Step 2:

[0275] The server analyzes collected purchase history and preference information. The input data includes cleansed purchase history, which the server analyzes using a machine learning library (e.g., TensorFlow). This analysis identifies user purchasing patterns and extracts features. The output generates a preference model for each user.

[0276] Step 3:

[0277] The server creates a product suggestion algorithm based on the generated preference model. The preference model and product data are used as input, and the algorithm is programmed using libraries such as scikit-learn. The output is a product list optimized for a specific user.

[0278] Step 4:

[0279] The server obtains the latest product inventory and promotion information from retailers through an API. The input includes the access key of the merchant API. This acquisition updates new products, special offers, etc., which are integrated into the recommendation algorithm. The output is a list of product recommendations reflecting the latest information.

[0280] Step 5:

[0281] The server sends the optimized product recommendations to the mobile device. The input includes the generated product list and the user's terminal information, and the data is securely transmitted via the HTTPS protocol. The terminal receives this information in real-time and notifies the user. The output is a notification or display on the terminal.

[0282] Step 6:

[0283] The terminal displays the received product recommendation information using augmented reality technology. The input data includes the product recommendation list. The terminal utilizes ARCore or ARKit to overlay the recommendation content on the visible products in the field of view. This enables the user to simultaneously view digital information while looking at physical products. The output is a visual presentation of product information via augmented reality.

[0284] Step 7:

[0285] The user checks the product information and selects the products they are interested in. The input is the information displayed on the terminal, based on which the user adds products to the cart or makes evaluations. The selected data is sent from the terminal to the server as feedback, and the output is feedback information that is utilized to improve the next recommendation algorithm.

[0286] (Application Example 1)

[0287] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0288] Modern consumers have access to a wide variety of product information, but conversely, they find it difficult to find products that suit them from the vast number of options. Especially in physical stores, there is a demand for personalized product suggestions based on the customer's purchase history and preferences, but systems that reflect individual preferences and past history in real time have not yet been fully realized. Therefore, it is necessary to provide an efficient and personalized shopping experience.

[0289] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0290] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating calculation rules for suggesting the optimal product; means for generating product suggestions based on the user's current location and situation; and means for presenting the generated product suggestions to the user on a mobile terminal using augmented reality technology. This makes it possible for the user to accurately obtain information that matches their preferences when selecting a product.

[0291] An "information processing device" is a device that performs data processing to analyze a user's purchase history and preferences, and to generate calculation rules for product recommendations.

[0292] "Operation rules" refer to sequences of numbers or logical procedures used to select and suggest the most suitable products based on the user's behavioral history.

[0293] A "portable device" is an electronic device that can be carried by the user and has the function of visually displaying product suggestions.

[0294] Augmented reality technology is a technology that overlays virtual information onto visual information from the real world.

[0295] A "camera" is a device that uses optical sensors to record images of real objects and environments.

[0296] "Visualized information" refers to information that has been converted from digital data into a form that can be visually perceived by the user.

[0297] "Selection behavior" refers to a series of actions taken by users to choose products that interest them.

[0298] A "retail store" is a physical business facility for selling goods directly to consumers.

[0299] "Product information" refers to information about a product's specifications and price that consumers use as a reference when considering a purchase.

[0300] "Goods" refers to physical products that are the subject of a transaction.

[0301] The system for implementing this invention combines a server as an information processing device with a smartphone or AR-enabled device as a mobile terminal. The server analyzes the user's purchase history and preference information stored in a database and generates computational rules using a generated AI model based on this analysis. Specifically, it utilizes machine learning platforms such as TensorFlow to analyze purchase patterns and suggest products.

[0302] The terminal uses augmented reality technologies such as ARCore to overlay product suggestion information received from the server onto the user's environment. When a smartphone camera scans a product on a shelf, sales promotion data is displayed in real time as visualized information. This allows users to efficiently select products based on visually presented information within the store.

[0303] As a concrete example, when a user uses their smartphone in the beverage section of a grocery store, the device recommends low-calorie drinks based on their past purchase history and displays them using augmented reality (AR). In this case, an example of a prompt message that would be processed on the server might be, "Based on the product shelf scanned by the user, please suggest low-calorie beverages using AR, taking into account their past purchase history and health-conscious preferences."

[0304] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0305] Step 1:

[0306] The server acquires the purchase history and preference information of the user from the database. The user ID is given as input, and the past purchase transaction history data is extracted through an SQL query. Based on this data, the machine learning model analyzes the purchase pattern and creates an operation rule for generating new product proposals.

[0307] Step 2:

[0308] Based on the generated operation rule, the server makes an optimal product proposal in comparison with the user's current location information. Location identification is performed using GPS data, and the latest product information and promotion information of the corresponding store are retrieved to create a customized product proposal list for the user. As output, a set of product information considering the user's preferences is generated.

[0309] Step 3:

[0310] The terminal superimposes and displays the product proposal information received from the server on the video of the actual store captured by the camera. When the user scans the shelf using the smartphone camera on the spot, the video data is received as input, and the product proposal is displayed in real time by utilizing AR technology. As output, a visual augmented reality display is generated and presented to the user.

[0311] Step 4:

[0312] The user makes a selection action for the product that interests them. The terminal acquires this selection information and sends it back to the server. The identifier and evaluation data of the selected product are received as input, and feedback is provided to the database to reflect it in the next purchase proposal. As output, a more refined user profile is updated.

[0313] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0314] This invention is a system for highly personalized user purchasing experiences, and includes an information processing device (server), a mobile terminal, and communication means for exchanging data between them. Furthermore, by incorporating an emotion engine, it enables product recommendations that take into account the user's emotional state.

[0315] server

[0316] The server collects and analyzes users' past purchase history and preference information. Based on the analysis results, it generates an optimal product recommendation algorithm using machine learning algorithms. This algorithm enables personalized product recommendations for each user. In addition, the server dynamically adjusts the recommendations based on user sentiment data acquired in real time.

[0317] Mobile devices

[0318] The mobile device plays the role of presenting product suggestions received from the server to the user in real time. Furthermore, it is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. Based on this information, it adjusts the content and order of the products presented, creating an optimal purchasing experience tailored to the user's emotions.

[0319] User operation

[0320] Users access product information through augmented reality technology using their mobile devices. While providing general feedback such as product selection and purchasing behavior, emotional data is also collected in real time. This allows the system to continuously suggest products tailored to the user's emotional state.

[0321] Specific example

[0322] For example, imagine a user visiting a supermarket and checking product information on their mobile device. The emotion engine analyzes the user's facial expressions and prioritizes displaying detailed information and related products for items the user finds interesting. Furthermore, if it detects the user is tired, it suggests relaxing products and promotions to provide a more comfortable shopping experience.

[0323] Thus, the present invention provides a system that enables highly personalized product recommendations that take into account the user's emotional state, thereby improving and optimizing the shopping experience.

[0324] The following describes the processing flow.

[0325] Step 1:

[0326] The server collects users' past purchase history and preference information from a database. This data is then analyzed using machine learning algorithms to generate a product recommendation algorithm optimized for each user.

[0327] Step 2:

[0328] The server retrieves the latest product and promotional information from retail stores, combines it with analyzed preference data, and creates a ranked product list to provide to the user. This list is then sent to the mobile device.

[0329] Step 3:

[0330] The terminal receives a product list sent from the server and uses an emotion engine to acquire emotional data in real time from the user's facial expressions and voice. This allows the terminal to analyze the user's emotional state and adjust product recommendations accordingly.

[0331] Step 4:

[0332] The device uses augmented reality technology to display information and promotions related to the products the user is viewing. Based on the user's emotional state, it prioritizes displaying product information that is likely to interest the user.

[0333] Step 5:

[0334] Users select and evaluate products and input feedback via a device. The device then sends this feedback information and acquired sentiment data to a server.

[0335] Step 6:

[0336] The server analyzes the emotional data and feedback sent from the terminal and updates the product recommendation algorithm. This results in more personalized and optimized recommendations for subsequent uses, tailored to the user's emotional state.

[0337] (Example 2)

[0338] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0339] In today's information society, personalizing the consumer purchasing experience is crucial. However, conventional systems rely solely on past history and preference information to suggest products, failing to consider consumers' real-time emotional states and thus hindering the provision of optimal recommendations. Furthermore, the inability to quickly update recommendations to reflect consumers' latest needs and emotions limits the potential for improving customer satisfaction.

[0340] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0341] In this invention, the server includes means for analyzing the user's past purchase history and preferences and generating an algorithm for making personalized product suggestions using machine learning; means for analyzing the user's emotional state in real time and adjusting product suggestions accordingly; and means for updating the algorithm based on emotional data collected by a mobile terminal. This enables highly customized product suggestions that are tailored to the user's emotions and environment.

[0342] An "information processing device" is a computer system used to collect and analyze data and generate and execute algorithms based on specific purposes.

[0343] "Machine learning" is a type of artificial intelligence technology that automatically learns patterns and relationships from large amounts of data to make predictions and decisions.

[0344] "Emotional state" refers to information about the user's psychological state and mood at that time, obtained from their facial expressions, voice, etc.

[0345] An "algorithm" refers to a set of procedures or computational methods for solving a specific problem, and in this context, it refers to the processing procedures for making a product proposal.

[0346] Augmented reality technology is a technique that overlays computer-generated information onto images of the real world.

[0347] A "mobile device" refers to an electronic device that is portable by the user and capable of communication.

[0348] "Product suggestions" refer to the purchase options and related information presented to the user, and are usually structured based on the user's preferences and circumstances.

[0349] "Emotional data" refers to information about the emotional state of users, and is collected using sensors and analytical technologies.

[0350] This invention is a system for personalizing the user's purchasing experience and includes a server, terminals, and communication means to link them. First, the server collects the user's past purchase history and preferences and stores them in a database. For example, necessary information can be extracted from the database using SQL. The collected data is input into a machine learning algorithm using Python or similar to generate a model for product recommendations optimized for the user. Clustering and regression analysis may be performed using the scikit-learn library.

[0351] The terminal displays product suggestions received from the server to the user in real time. The terminal also incorporates an emotion engine that uses a camera and microphone to detect the user's facial expressions and voice, analyzing their emotional state. Facial recognition can be implemented using TensorFlow or OpenCV. This emotion data is sent to the server and used for immediate adjustments to product suggestions.

[0352] Users can view product information using augmented reality technology via their devices. During this process, users can select products they are interested in, and their feedback is collected by the device. This feedback data is sent to a server and used to improve the product recommendation algorithm.

[0353] For example, suppose a user visits a supermarket and uses a terminal to view product information. The emotion engine detects the user's interest and prioritizes displaying detailed information about specific products. If the system determines that the user is tired, it suggests promotions for products with relaxing effects. A specific prompt might be: "If the emotion engine detects that a user shopping at a supermarket is tired, please explain, with product examples, how product suggestions should be presented."

[0354] In this way, the system optimizes the purchasing experience by taking into account the user's real-time emotions and providing personalized suggestions.

[0355] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0356] Step 1:

[0357] The server collects the user's past purchase history and preferences from a database. The user ID is provided as input, and the relevant information is extracted from the database using an SQL query. The output is a dataset of the user's past purchase history and preference information. This dataset is preprocessed for analysis, including standardization and handling of missing values.

[0358] Step 2:

[0359] The server trains a machine learning algorithm using the collected dataset to generate a personalized product recommendation model. A pre-processed dataset is used as input. The scikit-learn library in Python is used for data analysis and algorithm training, including clustering and regression analysis. The output is a product recommendation model optimized for the user, preparing the system for product recommendations based on user preferences.

[0360] Step 3:

[0361] The terminal presents product suggestions to the user based on those received from the server. Product information is displayed on the terminal as output. Optimized product suggestions from the server are included as input. The terminal uses augmented reality technology to provide product information to the user visually. When the user shows interest in a product, the camera and AR functions are used to display detailed product information.

[0362] Step 4:

[0363] The device analyzes the user's emotions in real time. Data from the device's built-in camera and microphone is used as input. The device processes this data using an emotion analysis engine, employing TensorFlow and OpenCV for facial recognition. The output is data indicating the user's emotional state. This data is used to adjust the content and priority of product recommendations.

[0364] Step 5:

[0365] The server dynamically adjusts product recommendations based on real-time sentiment data. It uses sentiment state data as input. The server updates its product recommendation model and selects products appropriate to the user's current situation. The output is the updated product recommendations, which are then resent to the terminal. This creates a shopping experience that harmonizes with the user's emotions.

[0366] Step 6:

[0367] Users select products through their devices, and their behavioral data is collected in real time. User actions are captured as input. The device sends the selected product information to a server, which is used as feedback data to improve the algorithm. The output is feedback data, contributing to improved accuracy in product recommendations.

[0368] (Application Example 2)

[0369] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0370] Modern retail stores are required to provide a shopping experience optimized for consumers' diverse preferences and dynamic emotional changes. However, conventional product recommendation systems lack the ability to suggest products in real time while considering consumer emotions, making it difficult to provide a truly satisfying service to individual consumers. Therefore, a new system is needed to increase consumer purchasing intent and improve the in-store experience.

[0371] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0372] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating an algorithm for suggesting optimal products; means for analyzing the user's emotional state using an emotion recognition engine and dynamically adjusting the product suggestions; and means for presenting the generated product suggestions to the user using augmented reality technology via a communication device. This enables personalized product suggestions that respond to the consumer's emotions, thereby improving the purchasing experience at retail stores.

[0373] An "information processing device" is a device that performs analysis and generates algorithms based on user data, and also enables data communication with external parties.

[0374] An "algorithm" is a series of calculation procedures designed to provide optimal product recommendations by taking into account the user's purchase history and preferences.

[0375] An "emotion recognition engine" is a system that analyzes a user's facial expressions and voice to evaluate their emotional state in real time.

[0376] A "communication device" is a device that sends and receives data between a server and other devices.

[0377] Augmented reality technology is a technology that overlays digital information onto the real world, enabling users to experience three-dimensional and interactive information presentations.

[0378] "Product recommendations" refer to information about products that are presented to users based on analyzed data and are recommended for purchase.

[0379] A "consumer" is a customer who purchases goods at a retail store.

[0380] In order to implement this invention, it is necessary to build a system by combining three main elements: a server, a communication terminal, and an emotion recognition engine.

[0381] The server is responsible for analyzing users' past purchase history and preferences. Specifically, it retrieves purchase history data from the database and uses machine learning algorithms to generate product recommendation algorithms optimized for each user. Possible software options include machine learning frameworks such as TensorFlow and PyTorch.

[0382] A communication terminal is a device that presents proposed product information to the user through augmented reality technology. Examples include smart glasses and smartphones. These devices receive information from a server and simultaneously overlay digital information onto the user's real-world field of view. Specific hardware examples include devices such as the Oculus Quest.

[0383] The emotion recognition engine analyzes the user's facial expressions and voice to evaluate their emotional state in real time. This allows the server to dynamically adjust product recommendations based on this emotional data. Technologies used include services such as Microsoft Azure Facial Recognition API and Google Cloud Vision API.

[0384] Users use communication terminals within retail stores to view product information presented using augmented reality technology. The suggestions are tailored to the user's emotions, improving shopping satisfaction. For example, when a user is in the book section, smart glasses display information on related books and offer new suggestions based on their emotional response.

[0385] An example of a prompt sentence to input into a generative AI model would be, "When a user is looking at an interesting product, what are some effective ways to suggest related product information?"

[0386] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0387] Step 1:

[0388] The server retrieves user purchase history and preference data from a database. This data is then analyzed using a machine learning algorithm to generate a product recommendation algorithm optimized for each user. The output is a customized product recommendation algorithm tailored to each individual user.

[0389] Step 2:

[0390] The server retrieves the latest product information from an online database and incorporates it into a product suggestion algorithm generated for each user. During this process, the algorithm is optimized to include product inventory status and promotional information. The output of this process is updated product suggestion information.

[0391] Step 3:

[0392] The communication terminal receives product suggestion information transmitted from the server. The terminal uses this as input and presents it to the user as appropriate visual content using augmented reality technology. As output, product information is displayed integrated into the user's field of view.

[0393] Step 4:

[0394] The emotion recognition engine acquires the user's facial expressions and voice in real time from the camera and microphone, and uses this data as input to analyze their emotional state. This allows the system to output emotional information such as whether the user is interested or tired.

[0395] Step 5:

[0396] The server dynamically updates product recommendation information based on emotional information from the emotion recognition engine. This process involves inference by a generative AI model, reconstructing product recommendations based on prompt text. The output is the most up-to-date product recommendation tailored to the user's emotions.

[0397] Step 6:

[0398] Users access related products and additional information based on the product information displayed via their communication terminal. As output, feedback regarding product selection is sent to the server and incorporated into future suggestions.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0400] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0402] [Third Embodiment]

[0403] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0404] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0405] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0407] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0409] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0410] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0411] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0414] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0415] This invention relates to an information processing system for providing consumers with a personalized shopping experience. The system includes an information processing device, a mobile terminal, and a communication network for exchanging data between them.

[0416] Information processing device (server)

[0417] The server has the function of collecting and analyzing users' past purchase history and preference information. Based on this, it learns the user's purchasing patterns using machine learning algorithms. Through this learning, a customized product recommendation algorithm is generated for each user. Furthermore, the server obtains the latest product and promotion information from retail stores and product providers and evaluates its relevance to the user's preferences.

[0418] Mobile devices

[0419] Mobile devices (such as smartphones and AR glasses) play a role in providing users with suggested information received from the server in real time. Specifically, when a product in a store comes into view, it displays the product's characteristics, discount information, and similarity to purchase history overlaid using augmented reality technology.

[0420] User operation

[0421] Users view product information suggested during shopping via their mobile devices and provide feedback on items that interest them. This includes product selection, evaluation, and whether or not they make a purchase. This feedback data is sent from the device to the server and used to improve the product suggestion algorithm for future purchases, resulting in more refined personalization.

[0422] Specific example

[0423] For example, imagine a user is using their smartphone in a supermarket. The server analyzes the user's past purchase history of health-conscious foods and selects newly arrived organic products and related items on sale. This information is displayed via the smartphone's AR function, associated with the products the user is viewing on the shelf. When the user selects items they are interested in and adds them to their cart, the selection information is sent to the server and used to improve the accuracy of future recommendations.

[0424] Thus, the present invention makes the consumer shopping experience more efficient and personalized, resulting in time savings and increased satisfaction.

[0425] The following describes the processing flow.

[0426] Step 1:

[0427] The server collects users' past purchase history and preference data from a database and inputs this data into a machine learning algorithm for analysis. Through this analysis, it identifies users' purchasing patterns and preferences and generates a product recommendation algorithm based on them.

[0428] Step 2:

[0429] The server retrieves the latest product and promotional information from retail stores and product providers, compares it with analyzed preference data, and creates a product list tailored to the user. This information is then compiled into data packets and sent to the user's mobile device.

[0430] Step 3:

[0431] The terminal analyzes product suggestion information received from the server and associates it with products in the store where the user is currently located. When the user views a specific product through the terminal, additional information and promotions related to that product are displayed using augmented reality technology.

[0432] Step 4:

[0433] Users use their devices to view product information that interests them. They also provide feedback through actions such as selecting products, adding them to their cart, and providing ratings. The data collected during this process serves as a valuable indicator of purchasing intent and preferences.

[0434] Step 5:

[0435] The device sends feedback data about the user's choices to the server. This allows the server to update its suggestion algorithm, further improving the accuracy of future suggestions. Through this cyclical process, continuous personalization is achieved.

[0436] (Example 1)

[0437] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0438] Traditional shopping systems struggle to efficiently provide personalized product recommendations to users, resulting in a lack of timely information delivery. Furthermore, users cannot access detailed information about products they are interested in from the suggested items in real time, leaving a lack of methods to provide a highly satisfying shopping experience.

[0439] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0440] This invention includes a server that collects the user's past purchase history and preference information and generates an optimal product suggestion algorithm through data analysis; a server that generates personalized product suggestions for the user and transmits them to a mobile terminal; and a mobile terminal that uses augmented reality technology to present the received product suggestion information to the user in real time and visually displays product characteristics and discount information. As a result, the user can receive efficient and personalized product suggestions, check detailed product information in real time, and achieve a highly satisfying shopping experience.

[0441] An "information processing device" is a computer or the entire system used to collect, analyze, and process data.

[0442] "Purchase history" refers to records of products that a user has purchased in the past, as well as related information.

[0443] "Preference information" refers to data that indicates a user's preferences and interests in products.

[0444] A "product suggestion algorithm" is a mathematical or programmatic method for suggesting the most suitable products based on a user's purchase history and preference information.

[0445] A "mobile device" refers to a portable electronic device capable of receiving and displaying information, and specifically includes smartphones and augmented reality devices.

[0446] Augmented reality technology is a technology that overlays digital information onto the real world, providing information through the user's sight or hearing.

[0447] "Real-time" refers to a time frame in which information is processed almost immediately after it is generated or received, with virtually no delay.

[0448] "Latest product inventory" refers to the most up-to-date data on the current status and quantity of products at the seller.

[0449] "Promotional information" refers to marketing data related to product sales, discounts, and campaigns.

[0450] A "generative AI model" is a trained model that uses artificial intelligence technology to provide the optimal output for a given input.

[0451] A "prompt" refers to an instruction or input text given to a generative AI model.

[0452] The present invention is a system that provides users with a personalized shopping experience through a system including an information processing device, a mobile terminal, and a communication network.

[0453] First, the server functions as an information processing device, collecting users' past purchase history and preference information. This information is stored in a database and used through data analysis. The server uses programming languages ​​such as Python and R, as well as machine learning libraries (e.g., TensorFlow and scikit-learn), to analyze user data and generate an optimal product recommendation algorithm. This algorithm provides different recommendations for each user.

[0454] The server also retrieves the latest inventory and promotional information from retailers in real time via the internet. APIs are used for this retrieval, and the data includes new product information and discount information. This information, along with a product recommendation algorithm, is then presented to the user.

[0455] Next, the server sends the generated product suggestions to a mobile device (e.g., a smartphone). HTTPS is used as the communication protocol here, ensuring that data is exchanged securely.

[0456] The device displays received product suggestion information to the user using augmented reality (AR) technology. The device has apps and frameworks (e.g., ARCore and ARKit) installed to support AR technology. When the user visually checks products in a physical store, the device overlays digital information onto the product. This provides the user with product characteristics and discount information, saving time and increasing satisfaction.

[0457] As a concrete example, when a user is shopping at a supermarket, the server analyzes the user's purchasing patterns and suggests new organic products and current sale items to users who prefer health-conscious foods. This information is displayed on the shelves via the AR function of the mobile device. If the user becomes interested in a product, selects it, and adds it to their cart, that information is sent to the server in real time and used to improve the accuracy of future suggestions.

[0458] An example of a prompt message is, "Based on past purchase history, please suggest health-conscious products."

[0459] In this way, the present invention efficiently provides personalized product suggestions tailored to the user's preferences, thereby realizing a comfortable shopping experience.

[0460] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0461] Step 1:

[0462] The server collects the user's past purchase history and preference information from a database. The user's ID is provided as input, and relevant historical data is retrieved based on it. This data includes purchased items, purchase date and time, ratings, etc., and is then cleansed to remove noise and prepared for analysis.

[0463] Step 2:

[0464] The server analyzes collected purchase history and preference information. The input data includes cleansed purchase history, which the server analyzes using a machine learning library (e.g., TensorFlow). This analysis identifies user purchasing patterns and extracts features. The output generates a preference model for each user.

[0465] Step 3:

[0466] The server creates a product suggestion algorithm based on the generated preference model. The preference model and product data are used as input, and the algorithm is programmed using libraries such as scikit-learn. The output is a product list optimized for a specific user.

[0467] Step 4:

[0468] The server retrieves the latest product inventory and promotional information from retailers via API. The input includes the access key for the vendor's API. This retrieval updates new product and special offer information, which is then integrated into the suggestion algorithm. The output is a list of product suggestions reflecting the latest information.

[0469] Step 5:

[0470] The server sends optimized product recommendations to the mobile device. Inputs include the generated product list and the user's device information, and the data is securely transmitted via the HTTPS protocol. The device receives this information in real time and notifies the user. Output is a notification or display on the device.

[0471] Step 6:

[0472] The terminal displays received product suggestion information using augmented reality technology. The input data includes a list of product suggestions. The terminal utilizes ARCore and ARKit to overlay the suggestions onto products visible in the user's field of view. This allows the user to simultaneously view digital information while viewing physical products. The output is a visual presentation of product information using augmented reality.

[0473] Step 7:

[0474] The user reviews product information and selects items that interest them. The information displayed on the terminal serves as input, and based on this, the user adds items to their cart or leaves a review. The selected data is sent from the terminal to the server as feedback, and the output is feedback information used to improve the next suggestion algorithm.

[0475] (Application Example 1)

[0476] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0477] Modern consumers have access to a wide variety of product information, but conversely, they find it difficult to find products that suit them from the vast number of options. Especially in physical stores, there is a demand for personalized product suggestions based on the customer's purchase history and preferences, but systems that reflect individual preferences and past history in real time have not yet been fully realized. Therefore, it is necessary to provide an efficient and personalized shopping experience.

[0478] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0479] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating calculation rules for suggesting the optimal product; means for generating product suggestions based on the user's current location and situation; and means for presenting the generated product suggestions to the user on a mobile terminal using augmented reality technology. This makes it possible for the user to accurately obtain information that matches their preferences when selecting a product.

[0480] An "information processing device" is a device that performs data processing to analyze a user's purchase history and preferences, and to generate calculation rules for product recommendations.

[0481] "Operation rules" refer to sequences of numbers or logical procedures used to select and suggest the most suitable products based on the user's behavioral history.

[0482] A "portable device" is an electronic device that can be carried by the user and has the function of visually displaying product suggestions.

[0483] Augmented reality technology is a technology that overlays virtual information onto visual information from the real world.

[0484] A "camera" is a device that uses optical sensors to record images of real objects and environments.

[0485] "Visualized information" refers to information that has been converted from digital data into a form that can be visually perceived by the user.

[0486] "Selection behavior" refers to a series of actions taken by users to choose products that interest them.

[0487] A "retail store" is a physical business facility for selling goods directly to consumers.

[0488] "Product information" refers to information about a product's specifications and price that consumers use as a reference when considering a purchase.

[0489] "Goods" refers to physical products that are the subject of a transaction.

[0490] The system for implementing this invention combines a server as an information processing device with a smartphone or AR-enabled device as a mobile terminal. The server analyzes the user's purchase history and preference information stored in a database and generates computational rules using a generated AI model based on this analysis. Specifically, it utilizes machine learning platforms such as TensorFlow to analyze purchase patterns and suggest products.

[0491] The terminal uses augmented reality technologies such as ARCore to overlay product suggestion information received from the server onto the user's environment. When a smartphone camera scans a product on a shelf, sales promotion data is displayed in real time as visualized information. This allows users to efficiently select products based on visually presented information within the store.

[0492] As a concrete example, when a user uses their smartphone in the beverage section of a grocery store, the device recommends low-calorie drinks based on their past purchase history and displays them using augmented reality (AR). In this case, an example of a prompt message that would be processed on the server might be, "Based on the product shelf scanned by the user, please suggest low-calorie beverages using AR, taking into account their past purchase history and health-conscious preferences."

[0493] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0494] Step 1:

[0495] The server retrieves user purchase history and preference information from the database. Given a user ID as input, it extracts past purchase transaction data via SQL queries. Based on this data, a machine learning model analyzes purchase patterns and creates algorithmic rules to generate new product suggestions.

[0496] Step 2:

[0497] Based on the generated calculation rules, the server provides optimal product recommendations by comparing them with the user's current location information. Using GPS data to determine the user's location, it retrieves the latest product and promotional information from the relevant store and creates a customized product recommendation list for the user. The output is a set of product information tailored to the user's preferences.

[0498] Step 3:

[0499] The terminal overlays product suggestion information received from the server onto the in-store video captured by the camera. When a user scans a shelf using their smartphone camera, the terminal receives that video data as input, and product suggestions are displayed in real time using AR technology. As output, a visual augmented reality display is generated and presented to the user.

[0500] Step 4:

[0501] The user makes a selection of products that interest them. The terminal retrieves this selection information and sends it back to the server. The server receives the identifiers and evaluation data of the selected products as input and feeds this information back into the database to be used for future purchase suggestions. As output, a more refined user profile is updated.

[0502] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0503] This invention is a system for highly personalized user purchasing experiences, and includes an information processing device (server), a mobile terminal, and communication means for exchanging data between them. Furthermore, by incorporating an emotion engine, it enables product recommendations that take into account the user's emotional state.

[0504] server

[0505] The server collects and analyzes users' past purchase history and preference information. Based on the analysis results, it generates an optimal product recommendation algorithm using machine learning algorithms. This algorithm enables personalized product recommendations for each user. In addition, the server dynamically adjusts the recommendations based on user sentiment data acquired in real time.

[0506] Mobile devices

[0507] The mobile device plays the role of presenting product suggestions received from the server to the user in real time. Furthermore, it is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. Based on this information, it adjusts the content and order of the products presented, creating an optimal purchasing experience tailored to the user's emotions.

[0508] User operation

[0509] Users access product information through augmented reality technology using their mobile devices. While providing general feedback such as product selection and purchasing behavior, emotional data is also collected in real time. This allows the system to continuously suggest products tailored to the user's emotional state.

[0510] Specific example

[0511] For example, imagine a user visiting a supermarket and checking product information on their mobile device. The emotion engine analyzes the user's facial expressions and prioritizes displaying detailed information and related products for items the user finds interesting. Furthermore, if it detects the user is tired, it suggests relaxing products and promotions to provide a more comfortable shopping experience.

[0512] Thus, the present invention provides a system that enables highly personalized product recommendations that take into account the user's emotional state, thereby improving and optimizing the shopping experience.

[0513] The following describes the processing flow.

[0514] Step 1:

[0515] The server collects users' past purchase history and preference information from a database. This data is then analyzed using machine learning algorithms to generate a product recommendation algorithm optimized for each user.

[0516] Step 2:

[0517] The server retrieves the latest product and promotional information from retail stores, combines it with analyzed preference data, and creates a ranked product list to provide to the user. This list is then sent to the mobile device.

[0518] Step 3:

[0519] The terminal receives a product list sent from the server and uses an emotion engine to acquire emotional data in real time from the user's facial expressions and voice. This allows the terminal to analyze the user's emotional state and adjust product recommendations accordingly.

[0520] Step 4:

[0521] The device uses augmented reality technology to display information and promotions related to the products the user is viewing. Based on the user's emotional state, it prioritizes displaying product information that is likely to interest the user.

[0522] Step 5:

[0523] Users select and evaluate products and input feedback via a device. The device then sends this feedback information and acquired sentiment data to a server.

[0524] Step 6:

[0525] The server analyzes the emotional data and feedback sent from the terminal and updates the product recommendation algorithm. This results in more personalized and optimized recommendations for subsequent uses, tailored to the user's emotional state.

[0526] (Example 2)

[0527] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0528] In today's information society, personalizing the consumer purchasing experience is crucial. However, conventional systems rely solely on past history and preference information to suggest products, failing to consider consumers' real-time emotional states and thus hindering the provision of optimal recommendations. Furthermore, the inability to quickly update recommendations to reflect consumers' latest needs and emotions limits the potential for improving customer satisfaction.

[0529] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0530] In this invention, the server includes means for analyzing the user's past purchase history and preferences and generating an algorithm for making personalized product suggestions using machine learning; means for analyzing the user's emotional state in real time and adjusting product suggestions accordingly; and means for updating the algorithm based on emotional data collected by a mobile terminal. This enables highly customized product suggestions that are tailored to the user's emotions and environment.

[0531] An "information processing device" is a computer system used to collect and analyze data and generate and execute algorithms based on specific purposes.

[0532] "Machine learning" is a type of artificial intelligence technology that automatically learns patterns and relationships from large amounts of data to make predictions and decisions.

[0533] "Emotional state" refers to information about the user's psychological state and mood at that time, obtained from their facial expressions, voice, etc.

[0534] An "algorithm" refers to a set of procedures or computational methods for solving a specific problem, and in this context, it refers to the processing procedures for making a product proposal.

[0535] Augmented reality technology is a technique that overlays computer-generated information onto images of the real world.

[0536] A "mobile device" refers to an electronic device that is portable by the user and capable of communication.

[0537] "Product suggestions" refer to the purchase options and related information presented to the user, and are usually structured based on the user's preferences and circumstances.

[0538] "Emotional data" refers to information about the emotional state of users, and is collected using sensors and analytical technologies.

[0539] This invention is a system for personalizing the user's purchasing experience and includes a server, terminals, and communication means to link them. First, the server collects the user's past purchase history and preferences and stores them in a database. For example, necessary information can be extracted from the database using SQL. The collected data is input into a machine learning algorithm using Python or similar to generate a model for product recommendations optimized for the user. Clustering and regression analysis may be performed using the scikit-learn library.

[0540] The terminal displays product suggestions received from the server to the user in real time. The terminal also incorporates an emotion engine that uses a camera and microphone to detect the user's facial expressions and voice, analyzing their emotional state. Facial recognition can be implemented using TensorFlow or OpenCV. This emotion data is sent to the server and used for immediate adjustments to product suggestions.

[0541] Users can view product information using augmented reality technology via their devices. During this process, users can select products they are interested in, and their feedback is collected by the device. This feedback data is sent to a server and used to improve the product recommendation algorithm.

[0542] For example, suppose a user visits a supermarket and uses a terminal to view product information. The emotion engine detects the user's interest and prioritizes displaying detailed information about specific products. If the system determines that the user is tired, it suggests promotions for products with relaxing effects. A specific prompt might be: "If the emotion engine detects that a user shopping at a supermarket is tired, please explain, with product examples, how product suggestions should be presented."

[0543] In this way, the system optimizes the purchasing experience by taking into account the user's real-time emotions and providing personalized suggestions.

[0544] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0545] Step 1:

[0546] The server collects the user's past purchase history and preferences from a database. The user ID is provided as input, and the relevant information is extracted from the database using an SQL query. The output is a dataset of the user's past purchase history and preference information. This dataset is preprocessed for analysis, including standardization and handling of missing values.

[0547] Step 2:

[0548] The server trains a machine learning algorithm using the collected dataset to generate a personalized product recommendation model. A pre-processed dataset is used as input. The scikit-learn library in Python is used for data analysis and algorithm training, including clustering and regression analysis. The output is a product recommendation model optimized for the user, preparing the system for product recommendations based on user preferences.

[0549] Step 3:

[0550] The terminal presents product suggestions to the user based on those received from the server. Product information is displayed on the terminal as output. Optimized product suggestions from the server are included as input. The terminal uses augmented reality technology to provide product information to the user visually. When the user shows interest in a product, the camera and AR functions are used to display detailed product information.

[0551] Step 4:

[0552] The device analyzes the user's emotions in real time. Data from the device's built-in camera and microphone is used as input. The device processes this data using an emotion analysis engine, employing TensorFlow and OpenCV for facial recognition. The output is data indicating the user's emotional state. This data is used to adjust the content and priority of product recommendations.

[0553] Step 5:

[0554] The server dynamically adjusts product recommendations based on real-time sentiment data. It uses sentiment state data as input. The server updates its product recommendation model and selects products appropriate to the user's current situation. The output is the updated product recommendations, which are then resent to the terminal. This creates a shopping experience that harmonizes with the user's emotions.

[0555] Step 6:

[0556] Users select products through their devices, and their behavioral data is collected in real time. User actions are captured as input. The device sends the selected product information to a server, which is used as feedback data to improve the algorithm. The output is feedback data, contributing to improved accuracy in product recommendations.

[0557] (Application Example 2)

[0558] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0559] Modern retail stores are required to provide a shopping experience optimized for consumers' diverse preferences and dynamic emotional changes. However, conventional product recommendation systems lack the ability to suggest products in real time while considering consumer emotions, making it difficult to provide a truly satisfying service to individual consumers. Therefore, a new system is needed to increase consumer purchasing intent and improve the in-store experience.

[0560] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0561] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating an algorithm for suggesting optimal products; means for analyzing the user's emotional state using an emotion recognition engine and dynamically adjusting the product suggestions; and means for presenting the generated product suggestions to the user using augmented reality technology via a communication device. This enables personalized product suggestions that respond to the consumer's emotions, thereby improving the purchasing experience at retail stores.

[0562] An "information processing device" is a device that performs analysis and generates algorithms based on user data, and also enables data communication with external parties.

[0563] An "algorithm" is a series of calculation procedures designed to provide optimal product recommendations by taking into account the user's purchase history and preferences.

[0564] An "emotion recognition engine" is a system that analyzes a user's facial expressions and voice to evaluate their emotional state in real time.

[0565] A "communication device" is a device that sends and receives data between a server and other devices.

[0566] Augmented reality technology is a technology that overlays digital information onto the real world, enabling users to experience three-dimensional and interactive information presentations.

[0567] "Product recommendations" refer to information about products that are presented to users based on analyzed data and are recommended for purchase.

[0568] A "consumer" is a customer who purchases goods at a retail store.

[0569] In order to implement this invention, it is necessary to build a system by combining three main elements: a server, a communication terminal, and an emotion recognition engine.

[0570] The server is responsible for analyzing users' past purchase history and preferences. Specifically, it retrieves purchase history data from the database and uses machine learning algorithms to generate product recommendation algorithms optimized for each user. Possible software options include machine learning frameworks such as TensorFlow and PyTorch.

[0571] A communication terminal is a device that presents proposed product information to the user through augmented reality technology. Examples include smart glasses and smartphones. These devices receive information from a server and simultaneously overlay digital information onto the user's real-world field of view. Specific hardware examples include devices such as the Oculus Quest.

[0572] The emotion recognition engine analyzes the user's facial expressions and voice to evaluate their emotional state in real time. This allows the server to dynamically adjust product recommendations based on this emotional data. Technologies used include services such as Microsoft Azure Facial Recognition API and Google Cloud Vision API.

[0573] Users use communication terminals within retail stores to view product information presented using augmented reality technology. The suggestions are tailored to the user's emotions, improving shopping satisfaction. For example, when a user is in the book section, smart glasses display information on related books and offer new suggestions based on their emotional response.

[0574] An example of a prompt sentence to input into a generative AI model would be, "When a user is looking at an interesting product, what are some effective ways to suggest related product information?"

[0575] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0576] Step 1:

[0577] The server retrieves user purchase history and preference data from a database. This data is then analyzed using a machine learning algorithm to generate a product recommendation algorithm optimized for each user. The output is a customized product recommendation algorithm tailored to each individual user.

[0578] Step 2:

[0579] The server retrieves the latest product information from an online database and incorporates it into a product suggestion algorithm generated for each user. During this process, the algorithm is optimized to include product inventory status and promotional information. The output of this process is updated product suggestion information.

[0580] Step 3:

[0581] The communication terminal receives product suggestion information transmitted from the server. The terminal uses this as input and presents it to the user as appropriate visual content using augmented reality technology. As output, product information is displayed integrated into the user's field of view.

[0582] Step 4:

[0583] The emotion recognition engine acquires the user's facial expressions and voice in real time from the camera and microphone, and uses this data as input to analyze their emotional state. This allows the system to output emotional information such as whether the user is interested or tired.

[0584] Step 5:

[0585] The server dynamically updates product recommendation information based on emotional information from the emotion recognition engine. This process involves inference by a generative AI model, reconstructing product recommendations based on prompt text. The output is the most up-to-date product recommendation tailored to the user's emotions.

[0586] Step 6:

[0587] Users access related products and additional information based on the product information displayed via their communication terminal. As output, feedback regarding product selection is sent to the server and incorporated into future suggestions.

[0588] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0589] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0590] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0591] [Fourth Embodiment]

[0592] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0593] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0594] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0595] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0596] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0597] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0598] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0599] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0600] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0601] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0602] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0603] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0604] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0605] This invention relates to an information processing system for providing consumers with a personalized shopping experience. The system includes an information processing device, a mobile terminal, and a communication network for exchanging data between them.

[0606] Information processing device (server)

[0607] The server has the function of collecting and analyzing users' past purchase history and preference information. Based on this, it learns the user's purchasing patterns using machine learning algorithms. Through this learning, a customized product recommendation algorithm is generated for each user. Furthermore, the server obtains the latest product and promotion information from retail stores and product providers and evaluates its relevance to the user's preferences.

[0608] Mobile devices

[0609] Mobile devices (such as smartphones and AR glasses) play a role in providing users with suggested information received from the server in real time. Specifically, when a product in a store comes into view, it displays the product's characteristics, discount information, and similarity to purchase history overlaid using augmented reality technology.

[0610] User operation

[0611] Users view product information suggested during shopping via their mobile devices and provide feedback on items that interest them. This includes product selection, evaluation, and whether or not they make a purchase. This feedback data is sent from the device to the server and used to improve the product suggestion algorithm for future purchases, resulting in more refined personalization.

[0612] Specific example

[0613] For example, imagine a user is using their smartphone in a supermarket. The server analyzes the user's past purchase history of health-conscious foods and selects newly arrived organic products and related items on sale. This information is displayed via the smartphone's AR function, associated with the products the user is viewing on the shelf. When the user selects items they are interested in and adds them to their cart, the selection information is sent to the server and used to improve the accuracy of future recommendations.

[0614] Thus, the present invention makes the consumer shopping experience more efficient and personalized, resulting in time savings and increased satisfaction.

[0615] The following describes the processing flow.

[0616] Step 1:

[0617] The server collects users' past purchase history and preference data from a database and inputs this data into a machine learning algorithm for analysis. Through this analysis, it identifies users' purchasing patterns and preferences and generates a product recommendation algorithm based on them.

[0618] Step 2:

[0619] The server retrieves the latest product and promotional information from retail stores and product providers, compares it with analyzed preference data, and creates a product list tailored to the user. This information is then compiled into data packets and sent to the user's mobile device.

[0620] Step 3:

[0621] The terminal analyzes product suggestion information received from the server and associates it with products in the store where the user is currently located. When the user views a specific product through the terminal, additional information and promotions related to that product are displayed using augmented reality technology.

[0622] Step 4:

[0623] Users use their devices to view product information that interests them. They also provide feedback through actions such as selecting products, adding them to their cart, and providing ratings. The data collected during this process serves as a valuable indicator of purchasing intent and preferences.

[0624] Step 5:

[0625] The device sends feedback data about the user's choices to the server. This allows the server to update its suggestion algorithm, further improving the accuracy of future suggestions. Through this cyclical process, continuous personalization is achieved.

[0626] (Example 1)

[0627] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0628] Traditional shopping systems struggle to efficiently provide personalized product recommendations to users, resulting in a lack of timely information delivery. Furthermore, users cannot access detailed information about products they are interested in from the suggested items in real time, leaving a lack of methods to provide a highly satisfying shopping experience.

[0629] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0630] This invention includes a server that collects the user's past purchase history and preference information and generates an optimal product suggestion algorithm through data analysis; a server that generates personalized product suggestions for the user and transmits them to a mobile terminal; and a mobile terminal that uses augmented reality technology to present the received product suggestion information to the user in real time and visually displays product characteristics and discount information. As a result, the user can receive efficient and personalized product suggestions, check detailed product information in real time, and achieve a highly satisfying shopping experience.

[0631] An "information processing device" is a computer or the entire system used to collect, analyze, and process data.

[0632] "Purchase history" refers to records of products that a user has purchased in the past, as well as related information.

[0633] "Preference information" refers to data that indicates a user's preferences and interests in products.

[0634] A "product suggestion algorithm" is a mathematical or programmatic method for suggesting the most suitable products based on a user's purchase history and preference information.

[0635] A "mobile device" refers to a portable electronic device capable of receiving and displaying information, and specifically includes smartphones and augmented reality devices.

[0636] Augmented reality technology is a technology that overlays digital information onto the real world, providing information through the user's sight or hearing.

[0637] "Real-time" refers to a time frame in which information is processed almost immediately after it is generated or received, with virtually no delay.

[0638] "Latest product inventory" refers to the most up-to-date data on the current status and quantity of products at the seller.

[0639] "Promotional information" refers to marketing data related to product sales, discounts, and campaigns.

[0640] A "generative AI model" is a trained model that uses artificial intelligence technology to provide the optimal output for a given input.

[0641] A "prompt" refers to an instruction or input text given to a generative AI model.

[0642] The present invention is a system that provides users with a personalized shopping experience through a system including an information processing device, a mobile terminal, and a communication network.

[0643] First, the server functions as an information processing device, collecting users' past purchase history and preference information. This information is stored in a database and used through data analysis. The server uses programming languages ​​such as Python and R, as well as machine learning libraries (e.g., TensorFlow and scikit-learn), to analyze user data and generate an optimal product recommendation algorithm. This algorithm provides different recommendations for each user.

[0644] The server also retrieves the latest inventory and promotional information from retailers in real time via the internet. APIs are used for this retrieval, and the data includes new product information and discount information. This information, along with a product recommendation algorithm, is then presented to the user.

[0645] Next, the server sends the generated product suggestions to a mobile device (e.g., a smartphone). HTTPS is used as the communication protocol here, ensuring that data is exchanged securely.

[0646] The device displays received product suggestion information to the user using augmented reality (AR) technology. The device has apps and frameworks (e.g., ARCore and ARKit) installed to support AR technology. When the user visually checks products in a physical store, the device overlays digital information onto the product. This provides the user with product characteristics and discount information, saving time and increasing satisfaction.

[0647] As a concrete example, when a user is shopping at a supermarket, the server analyzes the user's purchasing patterns and suggests new organic products and current sale items to users who prefer health-conscious foods. This information is displayed on the shelves via the AR function of the mobile device. If the user becomes interested in a product, selects it, and adds it to their cart, that information is sent to the server in real time and used to improve the accuracy of future suggestions.

[0648] An example of a prompt message is, "Based on past purchase history, please suggest health-conscious products."

[0649] In this way, the present invention efficiently provides personalized product suggestions tailored to the user's preferences, thereby realizing a comfortable shopping experience.

[0650] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0651] Step 1:

[0652] The server collects the user's past purchase history and preference information from a database. The user's ID is provided as input, and relevant historical data is retrieved based on it. This data includes purchased items, purchase date and time, ratings, etc., and is then cleansed to remove noise and prepared for analysis.

[0653] Step 2:

[0654] The server analyzes collected purchase history and preference information. The input data includes cleansed purchase history, which the server analyzes using a machine learning library (e.g., TensorFlow). This analysis identifies user purchasing patterns and extracts features. The output generates a preference model for each user.

[0655] Step 3:

[0656] The server creates a product suggestion algorithm based on the generated preference model. The preference model and product data are used as input, and the algorithm is programmed using libraries such as scikit-learn. The output is a product list optimized for a specific user.

[0657] Step 4:

[0658] The server retrieves the latest product inventory and promotional information from retailers via API. The input includes the access key for the vendor's API. This retrieval updates new product and special offer information, which is then integrated into the suggestion algorithm. The output is a list of product suggestions reflecting the latest information.

[0659] Step 5:

[0660] The server sends optimized product recommendations to the mobile device. Inputs include the generated product list and the user's device information, and the data is securely transmitted via the HTTPS protocol. The device receives this information in real time and notifies the user. Output is a notification or display on the device.

[0661] Step 6:

[0662] The terminal displays received product suggestion information using augmented reality technology. The input data includes a list of product suggestions. The terminal utilizes ARCore and ARKit to overlay the suggestions onto products visible in the user's field of view. This allows the user to simultaneously view digital information while viewing physical products. The output is a visual presentation of product information using augmented reality.

[0663] Step 7:

[0664] The user reviews product information and selects items that interest them. The information displayed on the terminal serves as input, and based on this, the user adds items to their cart or leaves a review. The selected data is sent from the terminal to the server as feedback, and the output is feedback information used to improve the next suggestion algorithm.

[0665] (Application Example 1)

[0666] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0667] Modern consumers have access to a wide variety of product information, but conversely, they find it difficult to find products that suit them from the vast number of options. Especially in physical stores, there is a demand for personalized product suggestions based on the customer's purchase history and preferences, but systems that reflect individual preferences and past history in real time have not yet been fully realized. Therefore, it is necessary to provide an efficient and personalized shopping experience.

[0668] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0669] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating calculation rules for suggesting the optimal product; means for generating product suggestions based on the user's current location and situation; and means for presenting the generated product suggestions to the user on a mobile terminal using augmented reality technology. This makes it possible for the user to accurately obtain information that matches their preferences when selecting a product.

[0670] An "information processing device" is a device that performs data processing to analyze a user's purchase history and preferences, and to generate calculation rules for product recommendations.

[0671] "Operation rules" refer to sequences of numbers or logical procedures used to select and suggest the most suitable products based on the user's behavioral history.

[0672] A "portable device" is an electronic device that can be carried by the user and has the function of visually displaying product suggestions.

[0673] Augmented reality technology is a technology that overlays virtual information onto visual information from the real world.

[0674] A "camera" is a device that uses optical sensors to record images of real objects and environments.

[0675] "Visualized information" refers to information that has been converted from digital data into a form that can be visually perceived by the user.

[0676] "Selection behavior" refers to a series of actions taken by users to choose products that interest them.

[0677] A "retail store" is a physical business facility for selling goods directly to consumers.

[0678] "Product information" refers to information about a product's specifications and price that consumers use as a reference when considering a purchase.

[0679] "Goods" refers to physical products that are the subject of a transaction.

[0680] The system for implementing this invention combines a server as an information processing device with a smartphone or AR-enabled device as a mobile terminal. The server analyzes the user's purchase history and preference information stored in a database and generates computational rules using a generated AI model based on this analysis. Specifically, it utilizes machine learning platforms such as TensorFlow to analyze purchase patterns and suggest products.

[0681] The terminal uses augmented reality technologies such as ARCore to overlay product suggestion information received from the server onto the user's environment. When a smartphone camera scans a product on a shelf, sales promotion data is displayed in real time as visualized information. This allows users to efficiently select products based on visually presented information within the store.

[0682] As a concrete example, when a user uses their smartphone in the beverage section of a grocery store, the device recommends low-calorie drinks based on their past purchase history and displays them using augmented reality (AR). In this case, an example of a prompt message that would be processed on the server might be, "Based on the product shelf scanned by the user, please suggest low-calorie beverages using AR, taking into account their past purchase history and health-conscious preferences."

[0683] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0684] Step 1:

[0685] The server retrieves user purchase history and preference information from the database. Given a user ID as input, it extracts past purchase transaction data via SQL queries. Based on this data, a machine learning model analyzes purchase patterns and creates algorithmic rules to generate new product suggestions.

[0686] Step 2:

[0687] Based on the generated calculation rules, the server provides optimal product recommendations by comparing them with the user's current location information. Using GPS data to determine the user's location, it retrieves the latest product and promotional information from the relevant store and creates a customized product recommendation list for the user. The output is a set of product information tailored to the user's preferences.

[0688] Step 3:

[0689] The terminal overlays product suggestion information received from the server onto the in-store video captured by the camera. When a user scans a shelf using their smartphone camera, the terminal receives that video data as input, and product suggestions are displayed in real time using AR technology. As output, a visual augmented reality display is generated and presented to the user.

[0690] Step 4:

[0691] The user makes a selection of products that interest them. The terminal retrieves this selection information and sends it back to the server. The server receives the identifiers and evaluation data of the selected products as input and feeds this information back into the database to be used for future purchase suggestions. As output, a more refined user profile is updated.

[0692] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0693] This invention is a system for highly personalized user purchasing experiences, and includes an information processing device (server), a mobile terminal, and communication means for exchanging data between them. Furthermore, by incorporating an emotion engine, it enables product recommendations that take into account the user's emotional state.

[0694] server

[0695] The server collects and analyzes users' past purchase history and preference information. Based on the analysis results, it generates an optimal product recommendation algorithm using machine learning algorithms. This algorithm enables personalized product recommendations for each user. In addition, the server dynamically adjusts the recommendations based on user sentiment data acquired in real time.

[0696] Mobile devices

[0697] The mobile device plays the role of presenting product suggestions received from the server to the user in real time. Furthermore, it is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. Based on this information, it adjusts the content and order of the products presented, creating an optimal purchasing experience tailored to the user's emotions.

[0698] User operation

[0699] Users access product information through augmented reality technology using their mobile devices. While providing general feedback such as product selection and purchasing behavior, emotional data is also collected in real time. This allows the system to continuously suggest products tailored to the user's emotional state.

[0700] Specific example

[0701] For example, imagine a user visiting a supermarket and checking product information on their mobile device. The emotion engine analyzes the user's facial expressions and prioritizes displaying detailed information and related products for items the user finds interesting. Furthermore, if it detects the user is tired, it suggests relaxing products and promotions to provide a more comfortable shopping experience.

[0702] Thus, the present invention provides a system that enables highly personalized product recommendations that take into account the user's emotional state, thereby improving and optimizing the shopping experience.

[0703] The following describes the processing flow.

[0704] Step 1:

[0705] The server collects users' past purchase history and preference information from a database. This data is then analyzed using machine learning algorithms to generate a product recommendation algorithm optimized for each user.

[0706] Step 2:

[0707] The server retrieves the latest product and promotional information from retail stores, combines it with analyzed preference data, and creates a ranked product list to provide to the user. This list is then sent to the mobile device.

[0708] Step 3:

[0709] The terminal receives a product list sent from the server and uses an emotion engine to acquire emotional data in real time from the user's facial expressions and voice. This allows the terminal to analyze the user's emotional state and adjust product recommendations accordingly.

[0710] Step 4:

[0711] The device uses augmented reality technology to display information and promotions related to the products the user is viewing. Based on the user's emotional state, it prioritizes displaying product information that is likely to interest the user.

[0712] Step 5:

[0713] Users select and evaluate products and input feedback via a device. The device then sends this feedback information and acquired sentiment data to a server.

[0714] Step 6:

[0715] The server analyzes the emotional data and feedback sent from the terminal and updates the product recommendation algorithm. This results in more personalized and optimized recommendations for subsequent uses, tailored to the user's emotional state.

[0716] (Example 2)

[0717] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0718] In today's information society, personalizing the consumer purchasing experience is crucial. However, conventional systems rely solely on past history and preference information to suggest products, failing to consider consumers' real-time emotional states and thus hindering the provision of optimal recommendations. Furthermore, the inability to quickly update recommendations to reflect consumers' latest needs and emotions limits the potential for improving customer satisfaction.

[0719] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0720] In this invention, the server includes means for analyzing the user's past purchase history and preferences and generating an algorithm for making personalized product suggestions using machine learning; means for analyzing the user's emotional state in real time and adjusting product suggestions accordingly; and means for updating the algorithm based on emotional data collected by a mobile terminal. This enables highly customized product suggestions that are tailored to the user's emotions and environment.

[0721] An "information processing device" is a computer system used to collect and analyze data and generate and execute algorithms based on specific purposes.

[0722] "Machine learning" is a type of artificial intelligence technology that automatically learns patterns and relationships from large amounts of data to make predictions and decisions.

[0723] "Emotional state" refers to information about the user's psychological state and mood at that time, obtained from their facial expressions, voice, etc.

[0724] An "algorithm" refers to a set of procedures or computational methods for solving a specific problem, and in this context, it refers to the processing procedures for making a product proposal.

[0725] Augmented reality technology is a technique that overlays computer-generated information onto images of the real world.

[0726] A "mobile device" refers to an electronic device that is portable by the user and capable of communication.

[0727] "Product suggestions" refer to the purchase options and related information presented to the user, and are usually structured based on the user's preferences and circumstances.

[0728] "Emotional data" refers to information about the emotional state of users, and is collected using sensors and analytical technologies.

[0729] This invention is a system for personalizing the user's purchasing experience and includes a server, terminals, and communication means to link them. First, the server collects the user's past purchase history and preferences and stores them in a database. For example, necessary information can be extracted from the database using SQL. The collected data is input into a machine learning algorithm using Python or similar to generate a model for product recommendations optimized for the user. Clustering and regression analysis may be performed using the scikit-learn library.

[0730] The terminal displays product suggestions received from the server to the user in real time. The terminal also incorporates an emotion engine that uses a camera and microphone to detect the user's facial expressions and voice, analyzing their emotional state. Facial recognition can be implemented using TensorFlow or OpenCV. This emotion data is sent to the server and used for immediate adjustments to product suggestions.

[0731] Users can view product information using augmented reality technology via their devices. During this process, users can select products they are interested in, and their feedback is collected by the device. This feedback data is sent to a server and used to improve the product recommendation algorithm.

[0732] For example, suppose a user visits a supermarket and uses a terminal to view product information. The emotion engine detects the user's interest and prioritizes displaying detailed information about specific products. If the system determines that the user is tired, it suggests promotions for products with relaxing effects. A specific prompt might be: "If the emotion engine detects that a user shopping at a supermarket is tired, please explain, with product examples, how product suggestions should be presented."

[0733] In this way, the system optimizes the purchasing experience by taking into account the user's real-time emotions and providing personalized suggestions.

[0734] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0735] Step 1:

[0736] The server collects the user's past purchase history and preferences from a database. The user ID is provided as input, and the relevant information is extracted from the database using an SQL query. The output is a dataset of the user's past purchase history and preference information. This dataset is preprocessed for analysis, including standardization and handling of missing values.

[0737] Step 2:

[0738] The server trains a machine learning algorithm using the collected dataset to generate a personalized product recommendation model. A pre-processed dataset is used as input. The scikit-learn library in Python is used for data analysis and algorithm training, including clustering and regression analysis. The output is a product recommendation model optimized for the user, preparing the system for product recommendations based on user preferences.

[0739] Step 3:

[0740] The terminal presents product suggestions to the user based on those received from the server. Product information is displayed on the terminal as output. Optimized product suggestions from the server are included as input. The terminal uses augmented reality technology to provide product information to the user visually. When the user shows interest in a product, the camera and AR functions are used to display detailed product information.

[0741] Step 4:

[0742] The device analyzes the user's emotions in real time. Data from the device's built-in camera and microphone is used as input. The device processes this data using an emotion analysis engine, employing TensorFlow and OpenCV for facial recognition. The output is data indicating the user's emotional state. This data is used to adjust the content and priority of product recommendations.

[0743] Step 5:

[0744] The server dynamically adjusts product recommendations based on real-time sentiment data. It uses sentiment state data as input. The server updates its product recommendation model and selects products appropriate to the user's current situation. The output is the updated product recommendations, which are then resent to the terminal. This creates a shopping experience that harmonizes with the user's emotions.

[0745] Step 6:

[0746] Users select products through their devices, and their behavioral data is collected in real time. User actions are captured as input. The device sends the selected product information to a server, which is used as feedback data to improve the algorithm. The output is feedback data, contributing to improved accuracy in product recommendations.

[0747] (Application Example 2)

[0748] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0749] Modern retail stores are required to provide a shopping experience optimized for consumers' diverse preferences and dynamic emotional changes. However, conventional product recommendation systems lack the ability to suggest products in real time while considering consumer emotions, making it difficult to provide a truly satisfying service to individual consumers. Therefore, a new system is needed to increase consumer purchasing intent and improve the in-store experience.

[0750] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0751] In this invention, the server includes means for analyzing the user's past purchase history and preferences using an information processing device and generating an algorithm for suggesting optimal products; means for analyzing the user's emotional state using an emotion recognition engine and dynamically adjusting the product suggestions; and means for presenting the generated product suggestions to the user using augmented reality technology via a communication device. This enables personalized product suggestions that respond to the consumer's emotions, thereby improving the purchasing experience at retail stores.

[0752] An "information processing device" is a device that performs analysis and generates algorithms based on user data, and also enables data communication with external parties.

[0753] An "algorithm" is a series of calculation procedures designed to provide optimal product recommendations by taking into account the user's purchase history and preferences.

[0754] An "emotion recognition engine" is a system that analyzes a user's facial expressions and voice to evaluate their emotional state in real time.

[0755] A "communication device" is a device that sends and receives data between a server and other devices.

[0756] Augmented reality technology is a technology that overlays digital information onto the real world, enabling users to experience three-dimensional and interactive information presentations.

[0757] "Product recommendations" refer to information about products that are presented to users based on analyzed data and are recommended for purchase.

[0758] A "consumer" is a customer who purchases goods at a retail store.

[0759] In order to implement this invention, it is necessary to build a system by combining three main elements: a server, a communication terminal, and an emotion recognition engine.

[0760] The server is responsible for analyzing users' past purchase history and preferences. Specifically, it retrieves purchase history data from the database and uses machine learning algorithms to generate product recommendation algorithms optimized for each user. Possible software options include machine learning frameworks such as TensorFlow and PyTorch.

[0761] A communication terminal is a device that presents proposed product information to the user through augmented reality technology. Examples include smart glasses and smartphones. These devices receive information from a server and simultaneously overlay digital information onto the user's real-world field of view. Specific hardware examples include devices such as the Oculus Quest.

[0762] The emotion recognition engine analyzes the user's facial expressions and voice to evaluate their emotional state in real time. This allows the server to dynamically adjust product recommendations based on this emotional data. Technologies used include services such as Microsoft Azure Facial Recognition API and Google Cloud Vision API.

[0763] Users use communication terminals within retail stores to view product information presented using augmented reality technology. The suggestions are tailored to the user's emotions, improving shopping satisfaction. For example, when a user is in the book section, smart glasses display information on related books and offer new suggestions based on their emotional response.

[0764] An example of a prompt sentence to input into a generative AI model would be, "When a user is looking at an interesting product, what are some effective ways to suggest related product information?"

[0765] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0766] Step 1:

[0767] The server retrieves user purchase history and preference data from a database. This data is then analyzed using a machine learning algorithm to generate a product recommendation algorithm optimized for each user. The output is a customized product recommendation algorithm tailored to each individual user.

[0768] Step 2:

[0769] The server retrieves the latest product information from an online database and incorporates it into a product suggestion algorithm generated for each user. During this process, the algorithm is optimized to include product inventory status and promotional information. The output of this process is updated product suggestion information.

[0770] Step 3:

[0771] The communication terminal receives product suggestion information transmitted from the server. The terminal uses this as input and presents it to the user as appropriate visual content using augmented reality technology. As output, product information is displayed integrated into the user's field of view.

[0772] Step 4:

[0773] The emotion recognition engine acquires the user's facial expressions and voice in real time from the camera and microphone, and uses this data as input to analyze their emotional state. This allows the system to output emotional information such as whether the user is interested or tired.

[0774] Step 5:

[0775] The server dynamically updates product recommendation information based on emotional information from the emotion recognition engine. This process involves inference by a generative AI model, reconstructing product recommendations based on prompt text. The output is the most up-to-date product recommendation tailored to the user's emotions.

[0776] Step 6:

[0777] Users access related products and additional information based on the product information displayed via their communication terminal. As output, feedback regarding product selection is sent to the server and incorporated into future suggestions.

[0778] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0779] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0780] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0781] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0782] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0783] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0784] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0785] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0786] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0787] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0788] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0789] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0790] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0791] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0792] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0793] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0794] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0795] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0796] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0797] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0798] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0799] The following is further disclosed regarding the embodiments described above.

[0800] (Claim 1)

[0801] A means for generating an algorithm to suggest the optimal product by analyzing the user's past purchase history and preferences using an information processing device,

[0802] Based on the aforementioned algorithm, a means for generating product suggestions tailored to the user's current location and situation,

[0803] A means of presenting generated product suggestions to users on mobile devices using augmented reality technology,

[0804] A means for acquiring the user's choice behavior and updating the algorithm based on it,

[0805] A system that includes this.

[0806] (Claim 2)

[0807] The system according to claim 1, wherein the information processing device is equipped with means for obtaining the latest product information from a retail store and reflecting it in the product proposal.

[0808] (Claim 3)

[0809] The system according to claim 1, wherein the mobile terminal is equipped with means for displaying additional information related to products that the user has shown interest in in real time.

[0810] "Example 1"

[0811] (Claim 1)

[0812] A means for collecting users' past purchase history and preference information using an information processing device, and generating an optimal product suggestion algorithm through data analysis,

[0813] A means for generating personalized product suggestions for the user based on the aforementioned algorithm and transmitting them to a mobile device,

[0814] A means of presenting received product suggestion information to the user in real time using augmented reality technology on a mobile device, visually displaying product characteristics and discount information,

[0815] A means for acquiring user selection and evaluation information and improving the proposed algorithm based on that information,

[0816] A means for obtaining the latest product inventory and promotion information from retailers and reflecting it in the algorithm,

[0817] A system that includes this.

[0818] (Claim 2)

[0819] The system according to claim 1, comprising means for a mobile terminal to display additional information related to products that the user has shown interest in, in real time using augmented reality technology.

[0820] (Claim 3)

[0821] The system according to claim 1, wherein the information processing device comprises means for generating a product suggestion algorithm that takes prompt sentences as input using a generation AI model.

[0822] "Application Example 1"

[0823] (Claim 1)

[0824] The information processing device includes means for analyzing the user's past purchase history and preferences and generating calculation rules for suggesting the optimal product,

[0825] A means for generating product suggestions based on the user's current location and situation, based on the aforementioned calculation rules,

[0826] A means of presenting generated product suggestions to users on mobile devices using augmented reality technology,

[0827] A method for recognizing products using a camera and displaying them as superimposed visual information,

[0828] A means for obtaining the user's choice behavior and updating the calculation rules based thereon,

[0829] A system that includes this.

[0830] (Claim 2)

[0831] The system according to claim 1, comprising an information processing device that acquires the latest product information from a retail store and reflects it in the product proposal.

[0832] (Claim 3)

[0833] The system according to claim 1, wherein the mobile terminal has means for displaying additional information in real time related to an item that the user has shown interest in.

[0834] "Example 2 of combining an emotion engine"

[0835] (Claim 1)

[0836] A means for generating an algorithm using machine learning to provide personalized product recommendations by analyzing the user's past purchase history and preferences using an information processing device,

[0837] Based on the aforementioned algorithm, a means for analyzing the user's emotional state in real time and adjusting product suggestions accordingly,

[0838] A means of presenting generated product suggestions to users on mobile devices using augmented reality technology,

[0839] A means for collecting user emotion data via a mobile device and updating the algorithm based on that data,

[0840] A system that includes this.

[0841] (Claim 2)

[0842] The system according to claim 1, comprising an information processing device that includes means for reflecting the latest product information obtained from a retail facility into the product proposal.

[0843] (Claim 3)

[0844] The system according to claim 1, wherein the mobile terminal is equipped with means for providing additional information in real time related to products that the user has shown interest in.

[0845] "Application example 2 when combining with an emotional engine"

[0846] (Claim 1)

[0847] A means for generating an algorithm to suggest the optimal product by analyzing the user's past purchase history and preferences using an information processing device,

[0848] Based on the aforementioned algorithm, a means for generating product suggestions tailored to the user's current location and situation,

[0849] A means of presenting generated product suggestions to users using augmented reality technology in a communication device,

[0850] A means for acquiring the user's choice behavior and updating the algorithm based on it,

[0851] A means of analyzing the user's emotional state using an emotion recognition engine and dynamically adjusting product recommendations,

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The system according to claim 1, wherein the information processing device is equipped with means for obtaining the latest product information from a sales office and reflecting it in the product proposal.

[0855] (Claim 3)

[0856] The system according to claim 1, wherein the communication device has means for displaying additional information related to products that the user has shown interest in in real time and for optimizing the suggested content based on sentiment analysis data. [Explanation of symbols]

[0857] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. The information processing device includes means for analyzing the user's past purchase history and preferences and generating calculation rules for suggesting the optimal product, A means for generating product suggestions based on the user's current location and situation, based on the aforementioned calculation rules, A means of presenting generated product suggestions to users on mobile devices using augmented reality technology, A method for recognizing products using a camera and displaying them as superimposed visual information, A means for obtaining the user's choice behavior and updating the calculation rules based thereon, A system that includes this.

2. The system according to claim 1, comprising an information processing device that acquires the latest product information from retail stores and reflects it in the product proposal.

3. The system according to claim 1, wherein the mobile terminal has means for displaying additional information in real time related to an item that the user has shown interest in.

Images