system

Abstract

We provide the system. [Solution] An input method for receiving commercial transaction requests from users, An information generation means that proposes product information based on a request obtained by the input means, A presentation means for presenting the aforementioned product information to the user and accepting their selection, An analytical tool that creates an order list based on user selections and makes additional suggestions, An ordering method in which an online order is completed after final confirmation of the aforementioned order list, Information processing means that converts input information into text using speech recognition, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, although consumer activities using the Internet have become widespread, online shopping still poses a major barrier for users who are not familiar with digital technologies, such as the elderly. Therefore, there is a need to provide a support system that enables users who are not good at online shopping to easily select and order products.

Means for Solving the Problems

[0005] This invention provides a system that receives shopping requests from users, generates and presents relevant product information based on those requests, and enables users to easily make selections. Specifically, it obtains user requests using an input means, proposes product information using an information generation means, and presents the proposals to the user using a presentation means. Furthermore, it creates and optimizes an order list based on the user's selection using an analysis means, and allows the user to complete an online order using an ordering means. In this way, it reduces the burden on users in online shopping and provides a convenient purchasing experience.

[0006] A "user" is an individual or entity that uses the system to search for, select, and order products.

[0007] A "purchase request" is a request or need from a user regarding goods or services they wish to purchase.

[0008] An "input method" refers to an interface or device used to receive a user's shopping request.

[0009] "Information generation means" refers to a function or component that creates relevant product information based on a user's shopping request.

[0010] "Product information" refers to detailed data about a product that a user is considering purchasing.

[0011] "Presentation means" refers to a device or method for presenting generated product information to a user visually or audibly.

[0012] "Selection" refers to the act of a user deciding on a specific item from the product information presented to them.

[0013] "Analysis tools" refer to the function that performs the process of creating and optimizing order lists, taking into account the user's selection history and needs.

[0014] An "order list" is a compilation of the products and services selected by the user.

[0015] The "ordering means" is a function or process for performing a purchase procedure via an online system based on an order list determined by a user.

Brief Description of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a RAM (Random Access Memory) with a reference number is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a storage with a reference number is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The system according to the present invention is designed to make online shopping easier for users. Its embodiments are described in detail below.

[0038] Overall structure

[0039] This system primarily operates on a user terminal, a server, and the online supermarket's database. Users access the system through their terminals, and the server handles the central processing, enabling information generation, analysis, and ordering.

[0040] User Interface

[0041] The terminal provides a means for users to enter shopping requests. Specifically, users can specify the conditions necessary for shopping by entering text, using voice recognition, or selecting from a designated menu. Users can enter specific requests such as, "I want to buy ingredients for a hot pot dish."

[0042] Information generation and presentation

[0043] The server generates relevant product information based on user requests obtained through input methods. This includes a process of creating a product list related to user needs. It compares this information with a database of ingredients and seasonings to collect information on the products the user is looking for. The generated information is sent to the terminal and displayed visually to the user.

[0044] User selection and order list creation

[0045] The user reviews the presented product information and selects the product they wish to purchase. The terminal sends this selection information to the server. The server automatically generates an order list based on the user's selection and compiles the total number of products the user has chosen. It can also suggest other similar or complementary products, which the user can also select.

[0046] Final order confirmation and order placement

[0047] The server verifies the final order list, and after the user approves the contents, it places the order in the online supermarket's ordering system. Once final confirmation based on the user's review is received, the system completes the order and sends an order confirmation notification to the user.

[0048] Specific example

[0049] For example, if a user enters "I want to make vegetable soup" into the terminal, the server generates a list of ingredients such as "cabbage," "carrots," "broccoli," and "consommé" and sends it to the terminal, which the user then selects. Additional suggested items such as "bacon" and "pasta" are also presented, allowing the user to easily add all the necessary ingredients to their cart and complete their order smoothly.

[0050] As a result, this system significantly reduces the effort required for users to shop online, making it accessible to a wide range of users, including the elderly.

[0051] The following describes the processing flow.

[0052] Step 1:

[0053] The user activates the online shopping assistant through the terminal. The terminal displays prompts such as "What kind of products are you looking for?" and provides input methods.

[0054] Step 2:

[0055] The user enters "I want to buy ingredients for a hot pot dish" as a shopping request into the terminal. The terminal sends this information to the server.

[0056] Step 3:

[0057] The server uses the received shopping request as its core and generates relevant product information by referring to its internal database. For example, based on "hot pot dish," it lists candidates such as "chicken," "Chinese cabbage," and "tofu."

[0058] Step 4:

[0059] The server sends the generated product list to the terminal. The terminal visually presents the information to the user, displaying the options for each product.

[0060] Step 5:

[0061] The user selects the items they wish to purchase from the displayed product list. Once the selection is complete, the device sends the data of the selected items to the server.

[0062] Step 6:

[0063] The server creates an order list based on the user's selection. At the same time, it generates suggestions for complementary or synergistic products to the selected items and sends them to the terminal.

[0064] Step 7:

[0065] The terminal presents the user with supplemental products received from the server and prompts them to make additional selections. If the user selects additional products, that information is also sent to the server.

[0066] Step 8:

[0067] Once the final order details are confirmed, the server generates an order confirmation message for the user and sends it to their device. The user reviews the order details and approves it.

[0068] Step 9:

[0069] The server places an order with the online supermarket's ordering system based on the order list confirmed by the user. This completes the user's online order.

[0070] (Example 1)

[0071] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0072] In online shopping, users sometimes find it difficult to easily find and purchase the right products. This is especially true when choosing ingredients for a new dish or recipe, where selecting the appropriate products can be time-consuming and cumbersome. Furthermore, when suggestions are made without considering the user's past choices or popular products, it becomes difficult to make the best product selection.

[0073] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0074] In this invention, the server includes an input device for receiving shopping requests from users, an information generation device, a presentation device, and a selection analysis device. This allows users to easily find products that suit their needs and efficiently select and purchase products while receiving optimal suggestions that take into account past choices and popularity.

[0075] An "input device" is a means of receiving shopping requests from users.

[0076] An "information generation device" is a means for generating product information based on a request obtained by an input device.

[0077] A "presentation device" is a means of presenting generated product information to the user and accepting their selection.

[0078] A "selection analysis device" is a means of creating an order list based on user selections and suggesting similar products.

[0079] An "ordering device" is a means of completing an electronic transaction after final confirmation of the order list.

[0080] The system according to the present invention is designed to enable users to shop online efficiently. Specific embodiments thereof are shown below.

[0081] This system primarily consists of user terminals, servers, and online databases.

[0082] The user enters their shopping request using an input device on the terminal. This input device may include, for example, a keyboard or a microphone for voice recognition. The user can manually enter text or give voice instructions. The entered data is transmitted to the server via the network.

[0083] The server, acting as an information generator, processes input data using algorithms built with programming languages ​​such as Python. These algorithms cross-reference data with a database to identify the ingredients needed for specific dishes, generating product information. This product information includes the name, price, and description of each item.

[0084] The generated product information is transmitted to the terminal via a display device and visually presented on the user's display. The user can view the details of each product on the screen and select the product they wish to purchase. The selection information is returned to the server, which uses a selection analysis device to create an order list. At this time, the server suggests appropriate similar and complementary products using the user's past selection history and popular product data.

[0085] Final confirmation is performed by the user reviewing the order list displayed on the terminal. After the user approves, the server uses the ordering device to confirm the online order and transmits the necessary data to the external ordering system.

[0086] For example, if a user enters "I want to buy ingredients for vegetable soup" into the terminal, the server generates product information such as "cabbage," "carrots," "broccoli," and "consommé" and displays it on the terminal. The user can select the necessary products and, if needed, choose additional products such as "bacon" or "pasta." In this way, the system allows users to easily and efficiently select and purchase the products they want.

[0087] Examples of prompts generated using a generative AI model include "I want to make vegetable soup, please tell me what ingredients I need" and "Please tell me what ingredients you recommend for making a hot pot dish." This allows users to interact with the system using natural language and quickly obtain the information they need.

[0088] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0089] Step 1:

[0090] User input of shopping requests

[0091] The user enters their shopping request using the terminal's input device. This input can be done via text or voice. An example of input might be the prompt "I want to buy ingredients for vegetable soup." The entered data is sent to the server.

[0092] Step 2:

[0093] Product information generation by the server

[0094] The server generates product information based on the prompt text received from the user. Specifically, it searches the database based on the input data and creates a list of related products. This process uses a generative AI model to perform text processing and select the products best suited to the user's needs. For example, the list might include "cabbage," "carrots," "broccoli," and "consommé."

[0095] Step 3:

[0096] Displaying product information on the device

[0097] Product information generated from the server is sent to the terminal. The terminal receives this information and presents it to the user as a visual list. The user is configured to view product details, prices, and descriptions on the display.

[0098] Step 4:

[0099] User selection of products

[0100] The user selects the product they wish to purchase from the displayed product information. Selection can be made using a touch display or mouse. The user's selection results are then sent back to the server.

[0101] Step 5:

[0102] Server-based creation of order list

[0103] The server creates an order list based on the user's selection information. This list includes information about the selected products. Furthermore, by utilizing selection analysis tools, it is possible to analyze the user's past selection history and popular product information to suggest similar or complementary products.

[0104] Step 6:

[0105] Final confirmation by the user

[0106] The user makes a final confirmation of the order list displayed on the terminal. After the user has reviewed everything, they can approve the order, and this information is sent to the server.

[0107] Step 7:

[0108] Order processing by server

[0109] The server sends the final confirmed order list to an external e-commerce system to finalize the online order. This step involves communication with the external system using the necessary data communication protocols. Once the order is successfully finalized, the server sends a confirmation notification to the user.

[0110] (Application Example 1)

[0111] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0112] In modern society, there is a growing demand for the convenience of online shopping. However, for users, especially the elderly and those unfamiliar with technology, the complex operating procedures and lack of intuitive usability present challenges. Furthermore, users may not receive accurate suggestions when selecting products, leading to wasted time. Additionally, while voice input is becoming widespread, there is a lack of shopping systems that efficiently utilize this technology.

[0113] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0114] In this invention, the server includes an input means for receiving commercial transaction requests from users, an information processing means for converting the input information into text using speech recognition, and an information generation means for suggesting product information based on the requests obtained by the input means. This allows users to obtain product information with simple voice operations and complete their shopping efficiently.

[0115] A "user" is an entity that operates the system and makes requests for commercial transactions.

[0116] A "commercial transaction request" is a set of requirements regarding a product or service that a user is considering purchasing.

[0117] An "input method" is an interface through which a user communicates a business transaction request to the system.

[0118] "Information generation means" refers to a device or process that creates relevant product information based on acquired commercial transaction requests.

[0119] A "presentation means" is an interface that shows product information created by an information generation means to the user and accepts their selection.

[0120] "Analysis tools" refer to functions and processes that create an order list based on user selections and provide additional suggestions.

[0121] The "ordering method" refers to the processing function that allows users to complete online orders after final confirmation of the order list.

[0122] "Speech recognition" is a technology that takes speech as input and converts it into text data.

[0123] "Information processing means" refers to functions and processes that process input speech information using speech recognition technology and convert it into text.

[0124] The "order list" is a list of products and services that a user has decided to order.

[0125] The system implementing this invention consists of a user terminal, a server, a voice recognition service, and an online supermarket database. The user uses a terminal such as a smartphone to perform voice input. This voice input is converted into text by the voice recognition service and sent to the server. The server generates product information based on the acquired text information. This product information is designed to meet the user's needs using an algorithm that includes suggestions according to the type of meal. The server then presents the generated product information to the user's terminal and accepts the user's selection. The information selected by the user is sent to the server, and an order list is created using an analysis means. After the user makes a final confirmation, the information is sent to the online supermarket's ordering system by an ordering means, and the commercial transaction is completed.

[0126] A concrete example of its use is when a user voice-inputs, "I want to make Japanese-style curry for dinner tonight." The voice recognition system then converts the instruction into text, which is sent to the server. Based on this, the server generates and suggests related product information such as "curry roux," "beef," "potatoes," "carrots," and "onions." It can also suggest additional side dishes such as "miso soup" and "rice." Through this entire process, the user can efficiently and easily select the necessary products and confirm their order.

[0127] Examples of prompts include "I'd like to buy a Father's Day lunch set" or "Can you tell me some ingredients for an easy pasta dish?". These allow users to quickly receive product suggestions tailored to specific events or situations.

[0128] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0129] Step 1:

[0130] The user provides voice input via the device. The device receives the voice data and sends it to a voice recognition service. The input is voice, and the output is text data.

[0131] Step 2:

[0132] The speech recognition service converts speech data into text data. The input speech data is analyzed by a recognition algorithm, and the corresponding text is generated. The input is speech data, and the output is text data.

[0133] Step 3:

[0134] The terminal sends the text data received from the speech recognition service to the server. The server receives the text data and begins processing to generate product information. The input is text data, and the output is a product search query.

[0135] Step 4:

[0136] The server searches and retrieves relevant product information from the product database based on the acquired text data. Search operations are performed based on keywords related to the type of food. The input is a product search query, and the output is a list of product information.

[0137] Step 5:

[0138] The server sends a generated list of product information to the terminal, where it is presented to the user. The user reviews the presented products and selects the items they wish to purchase. The input is the product information list, and the output is the user's selection information.

[0139] Step 6:

[0140] The terminal sends user selection information to the server, which uses analysis tools to create an order list based on that information. The server also considers the selection history for additional suggestions. The input is user selection information, and the output is an order list.

[0141] Step 7:

[0142] The server notifies the user to perform a final check of the order list, and after confirmation, proceeds to the order placement process. The information is sent to the online supermarket's ordering system, and the transaction is completed. The input is the final confirmed order list, and the output is the order completion notification.

[0143] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0144] The system of the present invention provides comprehensive support to enable users to shop online smoothly. In particular, it includes a function that recognizes the user's emotions and provides product suggestions and selection support based on those emotions.

[0145] System Configuration

[0146] This system is built around a user terminal, a central server, and an emotion engine. The terminal is a tool for users to input shopping requests through an interface. The emotion engine analyzes the user's emotional state and sends the results to the server to optimize product suggestions.

[0147] Emotion recognition and information generation

[0148] The device acquires emotional data using its camera and microphone during user interaction. The server sends this emotional data to an emotion engine, which analyzes the user's emotional state. It also responds to subtle emotional changes to ensure that users can make purchases without stress.

[0149] Based on the emotional state analyzed by the emotion engine, the server generates product information tailored to the user's mood. This information includes options that align with the user's emotions, such as refreshing products for when they are relaxed, or nutritional supplements for when they are feeling down.

[0150] Information presentation and selection

[0151] The generated product information is sent back to the terminal and presented to the user. The user can then select the products they like. The selection is made directly on the terminal, and the system is designed to seamlessly select products that meet the user's preferences through simple touch operations or voice input.

[0152] Creating and optimizing order lists

[0153] The server creates an order list based on the user's selections. The analysis system combines user sentiment data obtained from the sentiment engine with past selection history to suggest optimal additional products. This process aims to provide suggestions that help the user maintain positive emotions.

[0154] Final confirmation and order completion

[0155] The final order list is confirmed by the user via the terminal. Once the user confirms the order, the server sends an order instruction to the online supermarket system, completing the order.

[0156] Specific example

[0157] For example, if a user is emotionally exhausted, the emotion engine recognizes this state and suggests items to lift the user's spirits, such as "relaxing herbal tea" or "popcorn for watching a movie." If the user selects an item, the system adds it to the cart and proceeds to the final confirmation stage.

[0158] This allows the system to go beyond simply supporting product selection and provide a pleasant purchasing experience that resonates with the user's emotions.

[0159] The following describes the processing flow.

[0160] Step 1:

[0161] The user powers on the device and launches the online shopping assistant application. The device activates its camera and microphone for emotion recognition and begins interacting with the user.

[0162] Step 2:

[0163] The terminal provides an interface for users to input their shopping requests and displays prompts such as, "What kind of products are you looking for?" The user then communicates their desire to the terminal via voice or text input, for example, "I want to make a hot pot dish."

[0164] Step 3:

[0165] The device acquires emotional data from the user's facial expressions and voice, and transmits this data to the server in real time. This emotional data includes the user's facial expressions, tone of voice, and word choice.

[0166] Step 4:

[0167] The server processes the received shopping request and emotion data, and uses an emotion engine to analyze the user's psychological state. Based on the analysis results, it generates product information that is appropriate for the user's emotions.

[0168] Step 5:

[0169] The server sends a list of products to the device that includes suggestions tailored to the user's emotional state. For example, if the server determines that the user needs to relax, it will suggest products containing relaxing beverages or ingredients.

[0170] Step 6:

[0171] The terminal visually displays a list of products sent from the server to the user. The user selects the items they wish to purchase from the products displayed on the screen.

[0172] Step 7:

[0173] User selection information is sent back from the terminal to the server. The server creates an order list based on the selected products and makes additional suggestions, taking sentiment data into consideration.

[0174] Step 8:

[0175] The terminal presents the user with additional suggestions from the server and prompts them to make further selections. The user then selects from the suggested items and reviews the final order list.

[0176] Step 9:

[0177] Once the final order list is confirmed, the server requests final confirmation from the user. Upon receiving confirmation, it proceeds with the order confirmation process with the online supermarket's ordering system.

[0178] Step 10:

[0179] The server creates an order completion notification and sends it to the user via the terminal. This allows the user to confirm that their order has been successfully completed.

[0180] (Example 2)

[0181] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0182] Modern online shopping systems face the challenge of failing to optimize user purchasing intent and satisfaction because they do not take into account the individual emotional state or real-time mental condition of users when suggesting products. In particular, when purchasing intent fluctuates depending on the user's emotions, traditional systems cannot respond effectively, resulting in a decrease in the quality of the purchasing experience.

[0183] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0184] In this invention, the server includes emotion recognition means for collecting user emotion data, information generation means for generating product information based on the data acquired by the input means and emotion recognition means, and analysis means for creating an order list based on the user's selection and making additional suggestions based on emotion data. This enables product suggestions that correspond to the user's emotional state, providing a more personalized purchasing experience.

[0185] "Input means" refers to a device or software for receiving shopping requests from users.

[0186] "Emotion recognition means" refers to a device or software for collecting and analyzing user emotion data.

[0187] "Information generation means" refers to a device or software that generates product information based on collected data.

[0188] "Presentation means" refers to a device or software used to present generated product information to the user.

[0189] "Analysis means" refers to a device or software for creating an order list based on user selections and making additional suggestions.

[0190] "Ordering method" refers to a device or software used to complete an online order after final confirmation.

[0191] The present invention provides a method for users to conduct online shopping efficiently and comfortably. The system consists of a user's terminal, a central server, and an emotion engine.

[0192] The terminal features a user interface that provides a means for users to input shopping requests. Through this interface, users can search for specific products or specify items they wish to purchase. The terminal is also equipped with a camera and microphone, which are used to sense the user's facial expressions and voice in real time and collect emotional data.

[0193] The server receives the emotional data and passes it to the emotion engine. The emotion engine uses a generative AI model to analyze the user's emotional state. This analysis accurately determines the user's current mood and emotions, enabling product suggestions tailored to their individual state. For example, when a user wants to relax, fragrant bath items or herbal teas might be suggested.

[0194] The generated product information is sent back to the terminal via the server and presented to the user. The user selects from the suggested products using touch controls or voice commands. Based on the selected information, the server creates an order list and makes additional suggestions that take into account emotional data and past purchase history. Through this process, the user is emotionally satisfied and has a more comfortable shopping experience.

[0195] For example, if a user says, "Suggest products that are more suitable for relaxation," the system will display relevant products in response to that request. In this case, an example of a prompt in the generative AI model might be, "Create a list of products that have a calming effect, based on the user's emotional data and activity history."

[0196] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0197] Step 1:

[0198] The user enters their shopping request through the terminal's interface. The terminal activates its camera and microphone to capture emotional data obtained from the user's facial expressions and voice. The input here consists of the user's purchase preference information and emotional data. This data is output as the system's initial input.

[0199] Step 2:

[0200] The device sends the collected emotional data to the server. The server passes the emotional data to the emotion engine, which then performs emotion analysis using a generative AI model. This analysis involves calculations to determine the user's mood and emotional state. The output is the analysis result, which indicates the user's emotional state.

[0201] Step 3:

[0202] The server generates product information based on the analyzed emotional data. It refers to the product database and performs calculations to identify products that are appropriate for the user's emotions. For example, if the user is tired, relaxation items will be selected. In this process, emotional data is taken as input and product information is output.

[0203] Step 4:

[0204] The server sends the created product information back to the terminal. The terminal presents this product information to the user. The user browses the products and selects items of interest. In this step, information is output to the user through a visual interface.

[0205] Step 5:

[0206] Based on the user's selected product information, the terminal sends data back to the server. The server adds the selected product to the order list and then performs data calculations using sentiment data and past selection history to make additional suggestions. At this stage, the order list and optimized suggested products are output.

[0207] Step 6:

[0208] The terminal presents the user with the final order list and suggested products, and the user confirms the purchase. Upon receiving the purchase confirmation instruction, the server outputs data to complete the online order and sends the order instruction to the relevant system. In this step, the completion of the online order is output.

[0209] (Application Example 2)

[0210] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0211] In online shopping, there is a challenge in that there is no system that can appropriately present product suggestions and payment information according to the user's emotional state, making it difficult for users to efficiently select and pay for products without experiencing stress.

[0212] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0213] In this invention, the server includes data acquisition means, an information generation unit, and emotion analysis means. This makes it possible to provide optimal product suggestions and payment information tailored to the user's emotions.

[0214] "Data acquisition means" refers to devices or methods for receiving information from users.

[0215] An "information generation unit" is a device or method that generates product information and service details based on acquired data.

[0216] "Emotional analysis tools" are devices or methods that analyze a user's emotional data and infer their emotional state.

[0217] A "payment information generation means" is a device or method that suggests appropriate payment methods and campaigns based on the user's emotional state.

[0218] An "information presentation means" is a device or method that displays generated information to a user and accepts their selection.

[0219] "Data analysis means" refers to devices or methods that analyze a user's selection history and related information to generate additional suggestions.

[0220] "Processing management means" refers to devices or methods that complete online order procedures according to the user's selection.

[0221] The system for realizing this application is built around a user terminal, a central server, and an emotion engine. The user terminal uses a camera and microphone as data acquisition tools and functions as an interface to understand the user's emotional state. The emotional data acquired by the terminal is sent to the server and analyzed by emotion analysis tools. For the analysis, emotion analysis APIs (e.g., generative AI models such as Google® Cloud Vision API and IBM Watson® Tone Analyzer) are used. Based on this analysis, the server uses an information generation unit to generate product suggestions and payment information that are tailored to the user's emotions.

[0222] The generated product and payment information is transmitted to the user's terminal via an information display device, where the user can review and select their items. The user's selection is analyzed by the server's data analysis device, and additional suggestions are made as needed. Once the order is confirmed, the processing management device completes the online order. This system is designed to ensure a pleasant shopping experience for the user.

[0223] For example, if the system analyzes that a user is feeling stressed, the server will suggest "discount coupons" or "cashback campaigns" based on the emotion analysis to alleviate the user's feelings. Furthermore, it utilizes a generative AI model to predict what kind of campaigns a user might be interested in using prompt messages. By using prompt messages such as, "What kind of campaigns or payment options would be best to suggest when a user is currently experiencing stress regarding their purchase?", more personalized suggestions become possible.

[0224] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0225] Step 1:

[0226] The user's device uses a camera and microphone to capture the user's facial expressions and voice data. This data is sent to the server as emotion data. The input is the user's facial image and voice data, and the output is the emotion data sent to the server.

[0227] Step 2:

[0228] The server analyzes the received emotional data through emotion analysis tools. Using generative AI models such as the Google Cloud Vision API and IBM Watson Tone Analyzer, it classifies the user's emotional state into categories such as "relaxed" and "stressed." The input is the user's emotional data, and the output is the analyzed emotional state.

[0229] Step 3:

[0230] The server uses an information generation unit to generate appropriate product suggestions and payment information based on the analyzed emotional state. Using a generation AI model, it leverages prompts such as, "What campaigns or payment options are best to suggest when the user is currently experiencing stress regarding purchasing?" to derive appropriate information. The input is the analyzed emotional state, and the output is the generated product suggestions and payment information.

[0231] Step 4:

[0232] The server sends the generated product and payment information to the user's terminal. The terminal displays this information to the user through an information display device, allowing the user to review it and select products and payment methods. The input is the generated product and payment information, and the output is the user's selection.

[0233] Step 5:

[0234] The server analyzes user selections using data analysis tools and provides additional suggestions as needed. This analysis uses algorithms that consider the user's selection history and trending product information to provide optimal suggestions. The input is the user's selections and historical data, and the output is optimized additional suggestions.

[0235] Step 6:

[0236] Once the user selects and confirms the items they wish to purchase, the server uses processing and management tools to complete the online order. The input is the final confirmed order list, and the output is the completed order information.

[0237] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0238] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0239] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0240] [Second Embodiment]

[0241] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0242] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0243] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0244] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0245] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0246] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0247] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0248] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0249] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0250] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0251] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0252] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0253] The system according to the present invention is designed to make online shopping easier for users. Its embodiments are described in detail below.

[0254] Overall structure

[0255] This system primarily operates on a user terminal, a server, and the online supermarket's database. Users access the system through their terminals, and the server handles the central processing, enabling information generation, analysis, and ordering.

[0256] User Interface

[0257] The terminal provides a means for users to enter shopping requests. Specifically, users can specify the conditions necessary for shopping by entering text, using voice recognition, or selecting from a designated menu. Users can enter specific requests such as, "I want to buy ingredients for a hot pot dish."

[0258] Information generation and presentation

[0259] The server generates relevant product information based on user requests obtained through input methods. This includes a process of creating a product list related to user needs. It compares this information with a database of ingredients and seasonings to collect information on the products the user is looking for. The generated information is sent to the terminal and displayed visually to the user.

[0260] User selection and order list creation

[0261] The user reviews the presented product information and selects the product they wish to purchase. The terminal sends this selection information to the server. The server automatically generates an order list based on the user's selection and compiles the total number of products the user has chosen. It can also suggest other similar or complementary products, which the user can also select.

[0262] Final order confirmation and order placement

[0263] The server verifies the final order list, and after the user approves the contents, it places the order in the online supermarket's ordering system. Once final confirmation based on the user's review is received, the system completes the order and sends an order confirmation notification to the user.

[0264] Specific example

[0265] For example, if a user enters "I want to make vegetable soup" into the terminal, the server generates a list of ingredients such as "cabbage," "carrots," "broccoli," and "consommé" and sends it to the terminal, which the user then selects. Additional suggested items such as "bacon" and "pasta" are also presented, allowing the user to easily add all the necessary ingredients to their cart and complete their order smoothly.

[0266] As a result, this system significantly reduces the effort required for users to shop online, making it accessible to a wide range of users, including the elderly.

[0267] The following describes the processing flow.

[0268] Step 1:

[0269] The user activates the online shopping assistant through the terminal. The terminal displays prompts to the user such as "What kind of products are you looking for?" and provides input methods.

[0270] Step 2:

[0271] The user enters "I want to buy ingredients for a hot pot dish" as a shopping request into the terminal. The terminal sends this information to the server.

[0272] Step 3:

[0273] The server uses the received shopping request as its core and generates relevant product information by referring to its internal database. For example, based on "hot pot dish," it lists candidates such as "chicken," "Chinese cabbage," and "tofu."

[0274] Step 4:

[0275] The server sends the generated product list to the terminal. The terminal visually presents the information to the user, displaying the options for each product.

[0276] Step 5:

[0277] The user selects an item to purchase from the displayed product list. When the selection is complete, the terminal sends the data of the selected product to the server.

[0278] Step 6:

[0279] The server creates an order list based on the user's selection. At the same time, it generates proposals for complementary products and products with synergistic effects of the selected product and sends them to the terminal.

[0280] Step 7:

[0281] The terminal presents the complementary products received from the server to the user and prompts for additional selections. If the user selects additional products, that information is also sent to the server.

[0282] Step 8:

[0283] [[ID=2B]] When the final order details are confirmed, the server generates an order confirmation message for the user and sends it to the terminal. The user checks the order details and gives approval.

[0284] Step 9:

[0285] The server conducts an ordering procedure for the online supermarket's order system based on the confirmed order list from the user. Thereby, the user's online order is completed.

[0286] (Example 1)

[0287] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0288] In online shopping, users sometimes find it difficult to easily find and purchase the right products. This is especially true when choosing ingredients for a new dish or recipe, where selecting the appropriate products can be time-consuming and cumbersome. Furthermore, when suggestions are made without considering the user's past choices or popular products, it becomes difficult to make the best product selection.

[0289] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0290] In this invention, the server includes an input device for receiving shopping requests from users, an information generation device, a presentation device, and a selection analysis device. This allows users to easily find products that suit their needs and efficiently select and purchase products while receiving optimal suggestions that take into account past choices and popularity.

[0291] An "input device" is a means of receiving shopping requests from users.

[0292] An "information generation device" is a means for generating product information based on a request obtained by an input device.

[0293] A "presentation device" is a means of presenting generated product information to the user and accepting their selection.

[0294] A "selection analysis device" is a means of creating an order list based on user selections and suggesting similar products.

[0295] An "ordering device" is a means of completing an electronic transaction after final confirmation of the order list.

[0296] The system according to the present invention is designed to enable users to shop online efficiently. Specific embodiments are described below.

[0297] This system primarily consists of user terminals, servers, and online databases.

[0298] The user enters their shopping request using an input device on the terminal. This input device may include, for example, a keyboard or a microphone for voice recognition. The user can manually enter text or give voice instructions. The entered data is transmitted to the server via the network.

[0299] The server, acting as an information generator, processes input data using algorithms built with programming languages ​​such as Python. These algorithms cross-reference data with a database to identify the ingredients needed for specific dishes, generating product information. This product information includes the name, price, and description of each item.

[0300] The generated product information is transmitted to the terminal via a display device and visually presented on the user's display. The user can view the details of each product on the screen and select the product they wish to purchase. The selection information is returned to the server, which uses a selection analysis device to create an order list. At this time, the server suggests appropriate similar and complementary products using the user's past selection history and popular product data.

[0301] Final confirmation is performed by the user reviewing the order list displayed on the terminal. After the user approves, the server uses the ordering device to confirm the online order and transmits the necessary data to the external ordering system.

[0302] For example, if a user enters "I want to buy ingredients for vegetable soup" into the terminal, the server generates product information such as "cabbage," "carrots," "broccoli," and "consommé" and displays it on the terminal. The user can select the necessary products and, if needed, choose additional products such as "bacon" or "pasta." In this way, the system allows users to easily and efficiently select and purchase the products they want.

[0303] Examples of prompt sentences using a generative AI model include "I want to make vegetable soup. Please tell me the necessary ingredients." and "Please tell me recommended ingredients for making pot dishes." This enables the user to operate the system in natural language and quickly obtain the information they seek.

[0304] The flow of the specific process in Example 1 will be described using FIG. 11.

[0305] Step 1:

[0306] Input of shopping request by the user

[0307] The user inputs a shopping request using the input device of the terminal. This input is performed by text input or voice input. As an example of the input, a prompt sentence such as "I want to purchase ingredients for vegetable soup" can be considered. The input data is sent to the server.

[0308] Step 2:

[0309] Generation of product information by the server

[0310] The server generates product information based on the prompt sentence received from the user. Specifically, it searches the database based on the input data and creates a list of related products. In this process, text processing using a generative AI model is executed to select the products that best meet the user's needs. For example, "cabbage", "carrot", "broccoli", and "consommé" are included in the list.

[0311] Step 3:

[0312] Presentation of product information to the terminal

[0313] The product information generated by the server is sent to the terminal. The terminal receives this and presents it to the user as a visual list. The user is set to be able to check the details, prices, and descriptions of the products on the display.

[0314] Step 4:

[0315] User selection of products

[0316] The user selects the product they wish to purchase from the displayed product information. Selection can be made using a touch display or mouse. The user's selection results are then sent back to the server.

[0317] Step 5:

[0318] Server-based creation of order list

[0319] The server creates an order list based on the user's selection information. This list includes information about the selected products. Furthermore, by utilizing selection analysis tools, it is possible to analyze the user's past selection history and popular product information to suggest similar or complementary products.

[0320] Step 6:

[0321] Final confirmation by the user

[0322] The user makes a final confirmation of the order list displayed on the terminal. After the user has reviewed everything, they can approve the order, and this information is sent to the server.

[0323] Step 7:

[0324] Order processing by server

[0325] The server sends the final confirmed order list to an external e-commerce system to finalize the online order. This step involves communicating with the external system using the necessary data communication protocols. Once the order is successfully finalized, the server sends a confirmation notification to the user.

[0326] (Application Example 1)

[0327] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0328] In modern society, there is a growing demand for the convenience of online shopping. However, for users, especially the elderly and those unfamiliar with technology, the complex operating procedures and lack of intuitive usability present challenges. Furthermore, users may not receive accurate suggestions when selecting products, leading to wasted time. Additionally, while voice input is becoming widespread, there is a lack of shopping systems that efficiently utilize this technology.

[0329] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0330] In this invention, the server includes an input means for receiving commercial transaction requests from users, an information processing means for converting the input information into text using speech recognition, and an information generation means for suggesting product information based on the requests obtained by the input means. This allows users to obtain product information with simple voice operations and complete their shopping efficiently.

[0331] A "user" is an entity that operates the system and makes requests for commercial transactions.

[0332] A "commercial transaction request" is a set of requirements regarding a product or service that a user is considering purchasing.

[0333] An "input method" is an interface through which a user communicates a business transaction request to the system.

[0334] "Information generation means" refers to a device or process that creates relevant product information based on acquired commercial transaction requests.

[0335] A "presentation means" is an interface that shows product information created by an information generation means to the user and accepts their selection.

[0336] "Analysis tools" refer to functions and processes that create an order list based on user selections and provide additional suggestions.

[0337] The "ordering method" refers to the processing function that allows users to complete online orders after final confirmation of the order list.

[0338] "Speech recognition" is a technology that takes speech as input and converts it into text data.

[0339] "Information processing means" refers to functions and processes that process input speech information using speech recognition technology and convert it into text.

[0340] The "order list" is a list of products and services that a user has decided to order.

[0341] The system implementing this invention consists of a user terminal, a server, a voice recognition service, and an online supermarket database. The user uses a terminal such as a smartphone to perform voice input. This voice input is converted into text by the voice recognition service and sent to the server. The server generates product information based on the acquired text information. This product information is designed to meet the user's needs using an algorithm that includes suggestions according to the type of meal. The server then presents the generated product information to the user's terminal and accepts the user's selection. The information selected by the user is sent to the server, and an order list is created using an analysis means. After the user makes a final confirmation, the information is sent to the online supermarket's ordering system by an ordering means, and the commercial transaction is completed.

[0342] A concrete example of its use is when a user voice-inputs, "I want to make Japanese-style curry for dinner tonight." The voice recognition system then converts the instruction into text, which is sent to the server. Based on this, the server generates and suggests related product information such as "curry roux," "beef," "potatoes," "carrots," and "onions." It can also suggest additional side dishes such as "miso soup" and "rice." Through this entire process, the user can efficiently and easily select the necessary products and confirm their order.

[0343] Examples of prompts include "I'd like to buy a Father's Day lunch set" or "Can you tell me some ingredients for an easy pasta dish?". These allow users to quickly receive product suggestions tailored to specific events or situations.

[0344] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0345] Step 1:

[0346] The user provides voice input via the device. The device receives the voice data and sends it to a voice recognition service. The input is voice, and the output is text data.

[0347] Step 2:

[0348] The speech recognition service converts speech data into text data. The input speech data is analyzed by a recognition algorithm, and the corresponding text is generated. The input is speech data, and the output is text data.

[0349] Step 3:

[0350] The terminal sends the text data received from the speech recognition service to the server. The server receives the text data and begins processing to generate product information. The input is text data, and the output is a product search query.

[0351] Step 4:

[0352] The server searches and retrieves relevant product information from the product database based on the acquired text data. Search operations are performed based on keywords related to the type of food. The input is a product search query, and the output is a list of product information.

[0353] Step 5:

[0354] The server sends a generated list of product information to the terminal, where it is presented to the user. The user reviews the presented products and selects the items they wish to purchase. The input is the product information list, and the output is the user's selection information.

[0355] Step 6:

[0356] The terminal sends user selection information to the server, which uses analysis tools to create an order list based on that information. The server also considers the selection history for additional suggestions. The input is the user's selection information, and the output is the order list.

[0357] Step 7:

[0358] The server notifies the user to perform a final check of the order list, and after confirmation, proceeds to the order placement process. The information is sent to the online supermarket's ordering system, and the transaction is completed. The input is the final confirmed order list, and the output is the order completion notification.

[0359] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0360] The system of the present invention provides comprehensive support to enable users to shop online smoothly. In particular, it includes a function that recognizes the user's emotions and provides product suggestions and selection support based on those emotions.

[0361] System Configuration

[0362] This system is built around a user terminal, a central server, and an emotion engine. The terminal is a tool for users to input shopping requests through an interface. The emotion engine analyzes the user's emotional state and sends the results to the server to optimize product suggestions.

[0363] Emotion recognition and information generation

[0364] The device acquires emotional data using its camera and microphone during user interaction. The server sends this emotional data to an emotion engine, which analyzes the user's emotional state. It also responds to subtle emotional changes to ensure that users can make purchases without stress.

[0365] Based on the emotional state analyzed by the emotion engine, the server generates product information tailored to the user's mood. This information includes options that align with the user's emotions, such as refreshing products for when they are relaxed, or nutritional supplements for when they are feeling down.

[0366] Information presentation and selection

[0367] The generated product information is sent back to the terminal and presented to the user. The user can then select the products they like. The selection is made directly on the terminal, and the system is designed to seamlessly select products that meet the user's preferences through simple touch operations or voice input.

[0368] Creating and optimizing order lists

[0369] The server creates an order list based on the user's selections. The analysis system combines user sentiment data obtained from the sentiment engine with past selection history to suggest optimal additional products. This process aims to provide suggestions that help the user maintain positive emotions.

[0370] Final confirmation and order completion

[0371] The final order list is confirmed by the user via the terminal. Once the user confirms the order, the server sends an order instruction to the online supermarket system, completing the order.

[0372] Specific example

[0373] For example, if a user is emotionally exhausted, the emotion engine recognizes this state and suggests items to lift the user's spirits, such as "relaxing herbal tea" or "popcorn for watching a movie." If the user selects an item, the system adds it to the cart and proceeds to the final confirmation stage.

[0374] This allows the system to go beyond simply supporting product selection and provide a pleasant purchasing experience that resonates with the user's emotions.

[0375] The following describes the processing flow.

[0376] Step 1:

[0377] The user powers on the device and launches the online shopping assistant application. The device activates its camera and microphone for emotion recognition and begins interacting with the user.

[0378] Step 2:

[0379] The terminal provides an interface for users to input their shopping requests and displays prompts such as, "What kind of products are you looking for?" The user then communicates their desire to the terminal via voice or text input, for example, "I want to make a hot pot dish."

[0380] Step 3:

[0381] The device acquires emotional data from the user's facial expressions and voice, and transmits this data to the server in real time. This emotional data includes the user's facial expressions, tone of voice, and word choice.

[0382] Step 4:

[0383] The server processes the received shopping request and emotion data, and uses an emotion engine to analyze the user's psychological state. Based on the analysis results, it generates product information that is appropriate for the user's emotions.

[0384] Step 5:

[0385] The server sends a list of products to the device that includes suggestions tailored to the user's emotional state. For example, if the server determines that the user needs to relax, it will suggest products containing relaxing beverages or ingredients.

[0386] Step 6:

[0387] The terminal visually displays a list of products sent from the server to the user. The user selects the items they wish to purchase from the products displayed on the screen.

[0388] Step 7:

[0389] User selection information is sent back from the terminal to the server. The server creates an order list based on the selected products and makes additional suggestions, taking sentiment data into consideration.

[0390] Step 8:

[0391] The terminal presents the user with additional suggestions from the server and prompts them to make further selections. The user then selects from the suggested items and reviews the final order list.

[0392] Step 9:

[0393] Once the final order list is confirmed, the server requests final confirmation from the user. Upon receiving confirmation, it proceeds with the order confirmation process with the online supermarket's ordering system.

[0394] Step 10:

[0395] The server creates an order completion notification and sends it to the user via the terminal. This allows the user to confirm that their order has been successfully completed.

[0396] (Example 2)

[0397] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0398] Modern online shopping systems face the challenge of failing to optimize user purchasing intent and satisfaction because they do not take into account the individual emotional state or real-time mental condition of users when suggesting products. In particular, when purchasing intent fluctuates depending on the user's emotions, traditional systems cannot respond effectively, resulting in a decrease in the quality of the purchasing experience.

[0399] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0400] In this invention, the server includes emotion recognition means for collecting user emotion data, information generation means for generating product information based on the data acquired by the input means and emotion recognition means, and analysis means for creating an order list based on the user's selection and making additional suggestions based on emotion data. This enables product suggestions that correspond to the user's emotional state, providing a more personalized purchasing experience.

[0401] "Input means" refers to a device or software for receiving shopping requests from users.

[0402] "Emotion recognition means" refers to a device or software for collecting and analyzing user emotion data.

[0403] "Information generation means" refers to a device or software that generates product information based on collected data.

[0404] "Presentation means" refers to a device or software used to present generated product information to the user.

[0405] "Analysis means" refers to a device or software for creating an order list based on user selections and making additional suggestions.

[0406] "Ordering method" refers to a device or software used to complete an online order after final confirmation.

[0407] The present invention provides a method for users to conduct online shopping efficiently and comfortably. The system consists of a user's terminal, a central server, and an emotion engine.

[0408] The terminal features a user interface that provides a means for users to input shopping requests. Through this interface, users can search for specific products or specify items they wish to purchase. The terminal is also equipped with a camera and microphone, which are used to sense the user's facial expressions and voice in real time and collect emotional data.

[0409] The server receives the emotional data and passes it to the emotion engine. The emotion engine uses a generative AI model to analyze the user's emotional state. This analysis accurately determines the user's current mood and emotions, enabling product suggestions tailored to their individual state. For example, when a user wants to relax, fragrant bath items or herbal teas might be suggested.

[0410] The generated product information is sent back to the terminal via the server and presented to the user. The user selects from the suggested products using touch controls or voice commands. Based on the selected information, the server creates an order list and makes additional suggestions that take into account emotional data and past purchase history. Through this process, the user is emotionally satisfied and has a more comfortable shopping experience.

[0411] For example, if a user says, "Suggest products that are more suitable for relaxation," the system will display relevant products in response to that request. In this case, an example of a prompt in the generative AI model might be, "Create a list of products that have a calming effect, based on the user's emotional data and activity history."

[0412] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0413] Step 1:

[0414] The user enters their shopping request through the terminal's interface. The terminal activates its camera and microphone to capture emotional data obtained from the user's facial expressions and voice. The input here consists of the user's purchase preference information and emotional data. This data is output as the system's initial input.

[0415] Step 2:

[0416] The device sends the collected emotional data to the server. The server passes the emotional data to the emotion engine, which then performs emotion analysis using a generative AI model. This analysis involves calculations to determine the user's mood and emotional state. The output is the analysis result, which indicates the user's emotional state.

[0417] Step 3:

[0418] The server generates product information based on the analyzed emotional data. It refers to the product database and performs calculations to identify products that are appropriate for the user's emotions. For example, if the user is tired, relaxation items will be selected. In this process, emotional data is taken as input and product information is output.

[0419] Step 4:

[0420] The server sends the created product information back to the terminal. The terminal presents this product information to the user. The user browses the products and selects items of interest. In this step, information is output to the user through a visual interface.

[0421] Step 5:

[0422] Based on the user's selected product information, the terminal sends data back to the server. The server adds the selected product to the order list and then performs data calculations using sentiment data and past selection history to make additional suggestions. At this stage, the order list and optimized suggested products are output.

[0423] Step 6:

[0424] The terminal presents the user with the final order list and suggested products, and the user confirms the purchase. Upon receiving the purchase confirmation instruction, the server outputs data to complete the online order and sends the order instruction to the relevant system. In this step, the completion of the online order is output.

[0425] (Application Example 2)

[0426] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0427] In online shopping, there is a challenge in that there is no system that can appropriately present product suggestions and payment information according to the user's emotional state, making it difficult for users to efficiently select and pay for products without experiencing stress.

[0428] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0429] In this invention, the server includes data acquisition means, an information generation unit, and emotion analysis means. This makes it possible to provide optimal product suggestions and payment information tailored to the user's emotions.

[0430] "Data acquisition means" refers to devices or methods for receiving information from users.

[0431] An "information generation unit" is a device or method that generates product information and service details based on acquired data.

[0432] "Emotional analysis tools" are devices or methods that analyze a user's emotional data and infer their emotional state.

[0433] A "payment information generation means" is a device or method that suggests appropriate payment methods and campaigns based on the user's emotional state.

[0434] An "information presentation means" is a device or method that displays generated information to a user and accepts their selection.

[0435] "Data analysis means" refers to devices or methods that analyze a user's selection history and related information to generate additional suggestions.

[0436] "Processing management means" refers to devices or methods that complete online order procedures according to the user's selection.

[0437] The system for realizing this application is built around a user terminal, a central server, and an emotion engine. The user terminal uses a camera and microphone as data acquisition tools and functions as an interface to understand the user's emotional state. The emotional data acquired by the terminal is sent to the server and analyzed by emotion analysis tools. Emotion analysis APIs (such as Google Cloud Vision API or IBM Watson Tone Analyzer, which are generative AI models) are used for the analysis. Based on this analysis, the server uses an information generation unit to generate product suggestions and payment information that are tailored to the user's emotions.

[0438] The generated product and payment information is transmitted to the user's terminal via an information display device, where the user can review and select their items. The user's selection is analyzed by the server's data analysis device, and additional suggestions are made as needed. Once the order is confirmed, the processing management device completes the online order. This system is designed to ensure a pleasant shopping experience for the user.

[0439] For example, if the system analyzes that a user is feeling stressed, the server will suggest "discount coupons" or "cashback campaigns" based on the emotion analysis to alleviate the user's feelings. Furthermore, it utilizes a generative AI model to predict what kind of campaigns a user might be interested in using prompt messages. By using prompt messages such as, "What kind of campaigns or payment options would be best to suggest when a user is currently experiencing stress regarding their purchase?", more personalized suggestions become possible.

[0440] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0441] Step 1:

[0442] The user's device uses a camera and microphone to capture the user's facial expressions and voice data. This data is sent to the server as emotion data. The input is the user's facial image and voice data, and the output is the emotion data sent to the server.

[0443] Step 2:

[0444] The server analyzes the received emotional data through emotion analysis tools. Using generative AI models such as the Google Cloud Vision API and IBM Watson Tone Analyzer, it classifies the user's emotional state into categories such as "relaxed" and "stressed." The input is the user's emotional data, and the output is the analyzed emotional state.

[0445] Step 3:

[0446] The server uses an information generation unit to generate appropriate product suggestions and payment information based on the analyzed emotional state. Using a generation AI model, it leverages prompts such as, "What campaigns or payment options are best to suggest when the user is currently experiencing stress regarding purchasing?" to derive appropriate information. The input is the analyzed emotional state, and the output is the generated product suggestions and payment information.

[0447] Step 4:

[0448] The server sends the generated product and payment information to the user's terminal. The terminal displays this information to the user through an information display device, allowing the user to review it and select products and payment methods. The input is the generated product and payment information, and the output is the user's selection.

[0449] Step 5:

[0450] The server analyzes user selections using data analysis tools and provides additional suggestions as needed. This analysis uses algorithms that consider the user's selection history and trending product information to provide optimal suggestions. The input is the user's selections and historical data, and the output is optimized additional suggestions.

[0451] Step 6:

[0452] Once the user selects and confirms the items they wish to purchase, the server uses processing and management tools to complete the online order. The input is the final confirmed order list, and the output is the completed order information.

[0453] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0454] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0455] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0456] [Third Embodiment]

[0457] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0458] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0459] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0460] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0461] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0463] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0464] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0465] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0466] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0467] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0468] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0469] The system according to the present invention is designed to make online shopping easier for users. Its embodiments are described in detail below.

[0470] Overall structure

[0471] This system primarily operates on a user terminal, a server, and the online supermarket's database. Users access the system through their terminals, and the server handles the central processing, enabling information generation, analysis, and ordering.

[0472] User Interface

[0473] The terminal provides a means for users to enter shopping requests. Specifically, users can specify the conditions necessary for shopping by entering text, using voice recognition, or selecting from a designated menu. Users can enter specific requests such as, "I want to buy ingredients for a hot pot dish."

[0474] Information generation and presentation

[0475] The server generates relevant product information based on user requests obtained through input methods. This includes a process of creating a product list related to user needs. It compares this information with a database of ingredients and seasonings to collect information on the products the user is looking for. The generated information is sent to the terminal and displayed visually to the user.

[0476] User selection and order list creation

[0477] The user reviews the presented product information and selects the product they wish to purchase. The terminal sends this selection information to the server. The server automatically generates an order list based on the user's selection and compiles the total number of products the user has chosen. It can also suggest other similar or complementary products, which the user can also select.

[0478] Final order confirmation and order placement

[0479] The server verifies the final order list, and after the user approves the contents, it places the order in the online supermarket's ordering system. Once final confirmation based on the user's review is received, the system completes the order and sends an order confirmation notification to the user.

[0480] Specific example

[0481] For example, if a user enters "I want to make vegetable soup" into the terminal, the server generates a list of ingredients such as "cabbage," "carrots," "broccoli," and "consommé" and sends it to the terminal, which the user then selects. Additional suggested items such as "bacon" and "pasta" are also presented, allowing the user to easily add all the necessary ingredients to their cart and complete their order smoothly.

[0482] As a result, this system significantly reduces the effort required for users to shop online, making it accessible to a wide range of users, including the elderly.

[0483] The following describes the processing flow.

[0484] Step 1:

[0485] The user activates the online shopping assistant through the terminal. The terminal displays prompts to the user such as "What kind of products are you looking for?" and provides input methods.

[0486] Step 2:

[0487] The user enters "I want to buy ingredients for a hot pot dish" as a shopping request into the terminal. The terminal sends this information to the server.

[0488] Step 3:

[0489] The server uses the received shopping request as its core and generates relevant product information by referring to its internal database. For example, based on "hot pot dish," it lists candidates such as "chicken," "Chinese cabbage," and "tofu."

[0490] Step 4:

[0491] The server sends the generated product list to the terminal. The terminal visually presents the information to the user, displaying the options for each product.

[0492] Step 5:

[0493] The user selects the items they wish to purchase from the displayed product list. Once the selection is complete, the device sends the data of the selected items to the server.

[0494] Step 6:

[0495] The server creates an order list based on the user's selection. At the same time, it generates suggestions for complementary or synergistic products to the selected items and sends them to the terminal.

[0496] Step 7:

[0497] The terminal presents the user with supplemental products received from the server and prompts them to make additional selections. If the user selects additional products, that information is also sent to the server.

[0498] Step 8:

[0499] Once the final order details are confirmed, the server generates an order confirmation message for the user and sends it to their device. The user reviews the order details and approves it.

[0500] Step 9:

[0501] The server places an order with the online supermarket's ordering system based on the order list confirmed by the user. This completes the user's online order.

[0502] (Example 1)

[0503] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0504] In online shopping, users sometimes find it difficult to easily find and purchase the right products. This is especially true when choosing ingredients for a new dish or recipe, where selecting the appropriate products can be time-consuming and cumbersome. Furthermore, when suggestions are made without considering the user's past choices or popular products, it becomes difficult to make the best product selection.

[0505] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0506] In this invention, the server includes an input device for receiving shopping requests from users, an information generation device, a presentation device, and a selection analysis device. This allows users to easily find products that suit their needs and efficiently select and purchase products while receiving optimal suggestions that take into account past choices and popularity.

[0507] An "input device" is a means of receiving shopping requests from users.

[0508] An "information generation device" is a means for generating product information based on a request obtained by an input device.

[0509] A "presentation device" is a means of presenting generated product information to the user and accepting their selection.

[0510] A "selection analysis device" is a means of creating an order list based on user selections and suggesting similar products.

[0511] An "ordering device" is a means of completing an electronic transaction after final confirmation of the order list.

[0512] The system according to the present invention is designed to enable users to shop online efficiently. Specific embodiments are described below.

[0513] This system primarily consists of user terminals, servers, and online databases.

[0514] The user enters their shopping request using an input device on the terminal. This input device may include, for example, a keyboard or a microphone for voice recognition. The user can manually enter text or give voice instructions. The entered data is transmitted to the server via the network.

[0515] The server, acting as an information generator, processes input data using algorithms built with programming languages ​​such as Python. These algorithms cross-reference data with a database to identify the ingredients needed for specific dishes, generating product information. This product information includes the name, price, and description of each item.

[0516] The generated product information is transmitted to the terminal via a display device and visually presented on the user's display. The user can view the details of each product on the screen and select the product they wish to purchase. The selection information is returned to the server, which uses a selection analysis device to create an order list. At this time, the server suggests appropriate similar and complementary products using the user's past selection history and popular product data.

[0517] Final confirmation is performed by the user reviewing the order list displayed on the terminal. After the user approves, the server uses the ordering device to confirm the online order and transmits the necessary data to the external ordering system.

[0518] For example, if a user enters "I want to buy ingredients for vegetable soup" into the terminal, the server generates product information such as "cabbage," "carrots," "broccoli," and "consommé" and displays it on the terminal. The user can select the necessary products and, if needed, choose additional products such as "bacon" or "pasta." In this way, the system allows users to easily and efficiently select and purchase the products they want.

[0519] Examples of prompts generated using a generative AI model include "I want to make vegetable soup, please tell me what ingredients I need" and "Please tell me what ingredients you recommend for making a hot pot dish." This allows users to interact with the system using natural language and quickly obtain the information they need.

[0520] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0521] Step 1:

[0522] User input of shopping requests

[0523] The user enters their shopping request using the terminal's input device. This input can be done via text or voice. An example of input might be the prompt "I want to buy ingredients for vegetable soup." The entered data is sent to the server.

[0524] Step 2:

[0525] Product information generation by the server

[0526] The server generates product information based on the prompt text received from the user. Specifically, it searches the database based on the input data and creates a list of related products. This process uses a generative AI model to perform text processing and select the products best suited to the user's needs. For example, the list might include "cabbage," "carrots," "broccoli," and "consommé."

[0527] Step 3:

[0528] Displaying product information on the device

[0529] Product information generated from the server is sent to the terminal. The terminal receives this information and presents it to the user as a visual list. The user is configured to view product details, prices, and descriptions on the display.

[0530] Step 4:

[0531] User selection of products

[0532] The user selects the product they wish to purchase from the displayed product information. Selection can be made using a touch display or mouse. The user's selection results are then sent back to the server.

[0533] Step 5:

[0534] Server-based creation of order list

[0535] The server creates an order list based on the user's selection information. This list includes information about the selected products. Furthermore, by utilizing selection analysis tools, it is possible to analyze the user's past selection history and popular product information to suggest similar or complementary products.

[0536] Step 6:

[0537] Final confirmation by the user

[0538] The user makes a final confirmation of the order list displayed on the terminal. After the user has reviewed everything, they can approve the order, and this information is sent to the server.

[0539] Step 7:

[0540] Order processing by server

[0541] The server sends the final confirmed order list to an external e-commerce system to finalize the online order. This step involves communicating with the external system using the necessary data communication protocols. Once the order is successfully finalized, the server sends a confirmation notification to the user.

[0542] (Application Example 1)

[0543] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0544] In modern society, there is a growing demand for the convenience of online shopping. However, for users, especially the elderly and those unfamiliar with technology, the complex operating procedures and lack of intuitive usability present challenges. Furthermore, users may not receive accurate suggestions when selecting products, leading to wasted time. Additionally, while voice input is becoming widespread, there is a lack of shopping systems that efficiently utilize this technology.

[0545] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0546] In this invention, the server includes an input means for receiving commercial transaction requests from users, an information processing means for converting the input information into text using speech recognition, and an information generation means for suggesting product information based on the requests obtained by the input means. This allows users to obtain product information with simple voice operations and complete their shopping efficiently.

[0547] A "user" is an entity that operates the system and makes requests for commercial transactions.

[0548] A "commercial transaction request" is a set of requirements regarding a product or service that a user is considering purchasing.

[0549] An "input method" is an interface through which a user communicates a business transaction request to the system.

[0550] "Information generation means" refers to a device or process that creates relevant product information based on acquired commercial transaction requests.

[0551] A "presentation means" is an interface that shows product information created by an information generation means to the user and accepts their selection.

[0552] "Analysis tools" refer to functions and processes that create an order list based on user selections and provide additional suggestions.

[0553] The "ordering method" refers to the processing function that allows users to complete online orders after final confirmation of the order list.

[0554] "Speech recognition" is a technology that takes speech as input and converts it into text data.

[0555] "Information processing means" refers to functions and processes that process input speech information using speech recognition technology and convert it into text.

[0556] The "order list" is a list of products and services that a user has decided to order.

[0557] The system implementing this invention consists of a user terminal, a server, a voice recognition service, and an online supermarket database. The user uses a terminal such as a smartphone to perform voice input. This voice input is converted into text by the voice recognition service and sent to the server. The server generates product information based on the acquired text information. This product information is designed to meet the user's needs using an algorithm that includes suggestions according to the type of meal. The server then presents the generated product information to the user's terminal and accepts the user's selection. The information selected by the user is sent to the server, and an order list is created using an analysis means. After the user makes a final confirmation, the information is sent to the online supermarket's ordering system by an ordering means, and the commercial transaction is completed.

[0558] A concrete example of its use is when a user voice-inputs, "I want to make Japanese-style curry for dinner tonight." The voice recognition system then converts the instruction into text, which is sent to the server. Based on this, the server generates and suggests related product information such as "curry roux," "beef," "potatoes," "carrots," and "onions." It can also suggest additional side dishes such as "miso soup" and "rice." Through this entire process, the user can efficiently and easily select the necessary products and confirm their order.

[0559] Examples of prompts include "I'd like to buy a Father's Day lunch set" or "Can you tell me some ingredients for an easy pasta dish?". These allow users to quickly receive product suggestions tailored to specific events or situations.

[0560] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0561] Step 1:

[0562] The user provides voice input via the device. The device receives the voice data and sends it to a voice recognition service. The input is voice, and the output is text data.

[0563] Step 2:

[0564] The speech recognition service converts speech data into text data. The input speech data is analyzed by a recognition algorithm, and the corresponding text is generated. The input is speech data, and the output is text data.

[0565] Step 3:

[0566] The terminal sends the text data received from the speech recognition service to the server. The server receives the text data and begins processing to generate product information. The input is text data, and the output is a product search query.

[0567] Step 4:

[0568] The server searches and retrieves relevant product information from the product database based on the acquired text data. Search operations are performed based on keywords related to the type of food. The input is a product search query, and the output is a list of product information.

[0569] Step 5:

[0570] The server sends a generated list of product information to the terminal, where it is presented to the user. The user reviews the presented products and selects the items they wish to purchase. The input is the product information list, and the output is the user's selection information.

[0571] Step 6:

[0572] The terminal sends user selection information to the server, which uses analysis tools to create an order list based on that information. The server also considers the selection history for additional suggestions. The input is the user's selection information, and the output is the order list.

[0573] Step 7:

[0574] The server notifies the user to perform a final check of the order list, and after confirmation, proceeds to the order placement process. The information is sent to the online supermarket's ordering system, and the transaction is completed. The input is the final confirmed order list, and the output is the order completion notification.

[0575] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0576] The system of the present invention provides comprehensive support to enable users to shop online smoothly. In particular, it includes a function that recognizes the user's emotions and provides product suggestions and selection support based on those emotions.

[0577] System Configuration

[0578] This system is built around a user terminal, a central server, and an emotion engine. The terminal is a tool for users to input shopping requests through an interface. The emotion engine analyzes the user's emotional state and sends the results to the server to optimize product suggestions.

[0579] Emotion recognition and information generation

[0580] The device acquires emotional data using its camera and microphone during user interaction. The server sends this emotional data to an emotion engine, which analyzes the user's emotional state. It also responds to subtle emotional changes to ensure that users can make purchases without stress.

[0581] Based on the emotional state analyzed by the emotion engine, the server generates product information tailored to the user's mood. This information includes options that align with the user's emotions, such as refreshing products for when they are relaxed, or nutritional supplements for when they are feeling down.

[0582] Information presentation and selection

[0583] The generated product information is sent back to the terminal and presented to the user. The user can then select the products they like. The selection is made directly on the terminal, and the system is designed to seamlessly select products that meet the user's preferences through simple touch operations or voice input.

[0584] Creating and optimizing order lists

[0585] The server creates an order list based on the user's selections. The analysis system combines user sentiment data obtained from the sentiment engine with past selection history to suggest optimal additional products. This process aims to provide suggestions that help the user maintain positive emotions.

[0586] Final confirmation and order completion

[0587] The final order list is confirmed by the user via the terminal. Once the user confirms the order, the server sends an order instruction to the online supermarket system, completing the order.

[0588] Specific example

[0589] For example, if a user is emotionally exhausted, the emotion engine recognizes this state and suggests items to lift the user's spirits, such as "relaxing herbal tea" or "popcorn for watching a movie." If the user selects an item, the system adds it to the cart and proceeds to the final confirmation stage.

[0590] This allows the system to go beyond simply supporting product selection and provide a pleasant purchasing experience that resonates with the user's emotions.

[0591] The following describes the processing flow.

[0592] Step 1:

[0593] The user powers on the device and launches the online shopping assistant application. The device activates its camera and microphone for emotion recognition and begins interacting with the user.

[0594] Step 2:

[0595] The terminal provides an interface for users to input their shopping requests and displays prompts such as, "What kind of products are you looking for?" The user then communicates their desire to the terminal via voice or text input, for example, "I want to make a hot pot dish."

[0596] Step 3:

[0597] The device acquires emotional data from the user's facial expressions and voice, and transmits this data to the server in real time. This emotional data includes the user's facial expressions, tone of voice, and word choice.

[0598] Step 4:

[0599] The server processes the received shopping request and emotion data, and uses an emotion engine to analyze the user's psychological state. Based on the analysis results, it generates product information that is appropriate for the user's emotions.

[0600] Step 5:

[0601] The server sends a list of products to the device that includes suggestions tailored to the user's emotional state. For example, if the server determines that the user needs to relax, it will suggest products containing relaxing beverages or ingredients.

[0602] Step 6:

[0603] The terminal visually displays a list of products sent from the server to the user. The user selects the items they wish to purchase from the products displayed on the screen.

[0604] Step 7:

[0605] User selection information is sent back from the terminal to the server. The server creates an order list based on the selected products and makes additional suggestions, taking sentiment data into consideration.

[0606] Step 8:

[0607] The terminal presents the user with additional suggestions from the server and prompts them to make further selections. The user then selects from the suggested items and reviews the final order list.

[0608] Step 9:

[0609] Once the final order list is confirmed, the server requests final confirmation from the user. Upon receiving confirmation, it proceeds with the order confirmation process with the online supermarket's ordering system.

[0610] Step 10:

[0611] The server creates an order completion notification and sends it to the user via the terminal. This allows the user to confirm that their order has been successfully completed.

[0612] (Example 2)

[0613] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0614] Modern online shopping systems face the challenge of failing to optimize user purchasing intent and satisfaction because they do not take into account the individual emotional state or real-time mental condition of users when suggesting products. In particular, when purchasing intent fluctuates depending on the user's emotions, traditional systems cannot respond effectively, resulting in a decrease in the quality of the purchasing experience.

[0615] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0616] In this invention, the server includes emotion recognition means for collecting user emotion data, information generation means for generating product information based on the data acquired by the input means and emotion recognition means, and analysis means for creating an order list based on the user's selection and making additional suggestions based on emotion data. This enables product suggestions that correspond to the user's emotional state, providing a more personalized purchasing experience.

[0617] "Input means" refers to a device or software for receiving shopping requests from users.

[0618] "Emotion recognition means" refers to a device or software for collecting and analyzing user emotion data.

[0619] "Information generation means" refers to a device or software that generates product information based on collected data.

[0620] "Presentation means" refers to a device or software used to present generated product information to the user.

[0621] "Analysis means" refers to a device or software for creating an order list based on user selections and making additional suggestions.

[0622] "Ordering method" refers to a device or software used to complete an online order after final confirmation.

[0623] The present invention provides a method for users to conduct online shopping efficiently and comfortably. The system consists of a user's terminal, a central server, and an emotion engine.

[0624] The terminal features a user interface that provides a means for users to input shopping requests. Through this interface, users can search for specific products or specify items they wish to purchase. The terminal is also equipped with a camera and microphone, which are used to sense the user's facial expressions and voice in real time and collect emotional data.

[0625] The server receives the emotional data and passes it to the emotion engine. The emotion engine uses a generative AI model to analyze the user's emotional state. This analysis accurately determines the user's current mood and emotions, enabling product suggestions tailored to their individual state. For example, when a user wants to relax, fragrant bath items or herbal teas might be suggested.

[0626] The generated product information is sent back to the terminal via the server and presented to the user. The user selects from the suggested products using touch controls or voice commands. Based on the selected information, the server creates an order list and makes additional suggestions that take into account emotional data and past purchase history. Through this process, the user is emotionally satisfied and has a more comfortable shopping experience.

[0627] For example, if a user says, "Suggest products that are more suitable for relaxation," the system will display relevant products in response to that request. In this case, an example of a prompt in the generative AI model might be, "Create a list of products that have a calming effect, based on the user's emotional data and activity history."

[0628] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0629] Step 1:

[0630] The user enters their shopping request through the terminal's interface. The terminal activates its camera and microphone to capture emotional data obtained from the user's facial expressions and voice. The input here consists of the user's purchase preference information and emotional data. This data is output as the system's initial input.

[0631] Step 2:

[0632] The device sends the collected emotional data to the server. The server passes the emotional data to the emotion engine, which then performs emotion analysis using a generative AI model. This analysis involves calculations to determine the user's mood and emotional state. The output is the analysis result, which indicates the user's emotional state.

[0633] Step 3:

[0634] The server generates product information based on the analyzed emotional data. It refers to the product database and performs calculations to identify products that are appropriate for the user's emotions. For example, if the user is tired, relaxation items will be selected. In this process, emotional data is taken as input and product information is output.

[0635] Step 4:

[0636] The server sends the created product information back to the terminal. The terminal presents this product information to the user. The user browses the products and selects items of interest. In this step, information is output to the user through a visual interface.

[0637] Step 5:

[0638] Based on the user's selected product information, the terminal sends data back to the server. The server adds the selected product to the order list and then performs data calculations using sentiment data and past selection history to make additional suggestions. At this stage, the order list and optimized suggested products are output.

[0639] Step 6:

[0640] The terminal presents the user with the final order list and suggested products, and the user confirms the purchase. Upon receiving the purchase confirmation instruction, the server outputs data to complete the online order and sends the order instruction to the relevant system. In this step, the completion of the online order is output.

[0641] (Application Example 2)

[0642] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0643] In online shopping, there is a challenge in that there is no system that can appropriately present product suggestions and payment information according to the user's emotional state, making it difficult for users to efficiently select and pay for products without experiencing stress.

[0644] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0645] In this invention, the server includes data acquisition means, an information generation unit, and emotion analysis means. This makes it possible to provide optimal product suggestions and payment information tailored to the user's emotions.

[0646] "Data acquisition means" refers to devices or methods for receiving information from users.

[0647] An "information generation unit" is a device or method that generates product information and service details based on acquired data.

[0648] "Emotional analysis tools" are devices or methods that analyze a user's emotional data and infer their emotional state.

[0649] A "payment information generation means" is a device or method that suggests appropriate payment methods and campaigns based on the user's emotional state.

[0650] An "information presentation means" is a device or method that displays generated information to a user and accepts their selection.

[0651] "Data analysis means" refers to devices or methods that analyze a user's selection history and related information to generate additional suggestions.

[0652] "Processing management means" refers to devices or methods that complete online order procedures according to the user's selection.

[0653] The system for realizing this application is built around a user terminal, a central server, and an emotion engine. The user terminal uses a camera and microphone as data acquisition tools and functions as an interface to understand the user's emotional state. The emotional data acquired by the terminal is sent to the server and analyzed by emotion analysis tools. Emotion analysis APIs (such as Google Cloud Vision API or IBM Watson Tone Analyzer, which are generative AI models) are used for the analysis. Based on this analysis, the server uses an information generation unit to generate product suggestions and payment information that are tailored to the user's emotions.

[0654] The generated product and payment information is transmitted to the user's terminal via an information display device, where the user can review and select their items. The user's selection is analyzed by the server's data analysis device, and additional suggestions are made as needed. Once the order is confirmed, the processing management device completes the online order. This system is designed to ensure a pleasant shopping experience for the user.

[0655] For example, if the system analyzes that a user is feeling stressed, the server will suggest "discount coupons" or "cashback campaigns" based on the emotion analysis to alleviate the user's feelings. Furthermore, it utilizes a generative AI model to predict what kind of campaigns a user might be interested in using prompt messages. By using prompt messages such as, "What kind of campaigns or payment options would be best to suggest when a user is currently experiencing stress regarding their purchase?", more personalized suggestions become possible.

[0656] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0657] Step 1:

[0658] The user's device uses a camera and microphone to capture the user's facial expressions and voice data. This data is sent to the server as emotion data. The input is the user's facial image and voice data, and the output is the emotion data sent to the server.

[0659] Step 2:

[0660] The server analyzes the received emotional data through emotion analysis tools. Using generative AI models such as the Google Cloud Vision API and IBM Watson Tone Analyzer, it classifies the user's emotional state into categories such as "relaxed" and "stressed." The input is the user's emotional data, and the output is the analyzed emotional state.

[0661] Step 3:

[0662] The server uses an information generation unit to generate appropriate product suggestions and payment information based on the analyzed emotional state. Using a generation AI model, it leverages prompts such as, "What campaigns or payment options are best to suggest when the user is currently experiencing stress regarding purchasing?" to derive appropriate information. The input is the analyzed emotional state, and the output is the generated product suggestions and payment information.

[0663] Step 4:

[0664] The server sends the generated product and payment information to the user's terminal. The terminal displays this information to the user through an information display device, allowing the user to review it and select products and payment methods. The input is the generated product and payment information, and the output is the user's selection.

[0665] Step 5:

[0666] The server analyzes user selections using data analysis tools and provides additional suggestions as needed. This analysis uses algorithms that consider the user's selection history and trending product information to provide optimal suggestions. The input is the user's selections and historical data, and the output is optimized additional suggestions.

[0667] Step 6:

[0668] Once the user selects and confirms the items they wish to purchase, the server uses processing and management tools to complete the online order. The input is the final confirmed order list, and the output is the completed order information.

[0669] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0670] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0671] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0672] [Fourth Embodiment]

[0673] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0674] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0675] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0676] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0677] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0678] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0679] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0680] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0681] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0682] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0683] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0684] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0685] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0686] The system according to the present invention is designed to make online shopping easier for users. Its embodiments are described in detail below.

[0687] Overall structure

[0688] This system primarily operates on a user terminal, a server, and the online supermarket's database. Users access the system through their terminals, and the server handles the central processing, enabling information generation, analysis, and ordering.

[0689] User Interface

[0690] The terminal provides a means for users to enter shopping requests. Specifically, users can specify the conditions necessary for shopping by entering text, using voice recognition, or selecting from a designated menu. Users can enter specific requests such as, "I want to buy ingredients for a hot pot dish."

[0691] Information generation and presentation

[0692] The server generates relevant product information based on user requests obtained through input methods. This includes a process of creating a product list related to user needs. It compares this information with a database of ingredients and seasonings to collect information on the products the user is looking for. The generated information is sent to the terminal and displayed visually to the user.

[0693] User selection and order list creation

[0694] The user reviews the presented product information and selects the product they wish to purchase. The terminal sends this selection information to the server. The server automatically generates an order list based on the user's selection and compiles the total number of products the user has chosen. It can also suggest other similar or complementary products, which the user can also select.

[0695] Final order confirmation and order placement

[0696] The server verifies the final order list, and after the user approves the contents, it places the order in the online supermarket's ordering system. Once final confirmation based on the user's review is received, the system completes the order and sends an order confirmation notification to the user.

[0697] Specific example

[0698] For example, if a user enters "I want to make vegetable soup" into the terminal, the server generates a list of ingredients such as "cabbage," "carrots," "broccoli," and "consommé" and sends it to the terminal, which the user then selects. Additional suggested items such as "bacon" and "pasta" are also presented, allowing the user to easily add all the necessary ingredients to their cart and complete their order smoothly.

[0699] As a result, this system significantly reduces the effort required for users to shop online, making it accessible to a wide range of users, including the elderly.

[0700] The following describes the processing flow.

[0701] Step 1:

[0702] The user activates the online shopping assistant through the terminal. The terminal displays prompts to the user such as "What kind of products are you looking for?" and provides input methods.

[0703] Step 2:

[0704] The user enters "I want to buy ingredients for a hot pot dish" as a shopping request into the terminal. The terminal sends this information to the server.

[0705] Step 3:

[0706] The server uses the received shopping request as its core and generates relevant product information by referring to its internal database. For example, based on "hot pot dish," it lists candidates such as "chicken," "Chinese cabbage," and "tofu."

[0707] Step 4:

[0708] The server sends the generated product list to the terminal. The terminal visually presents the information to the user, displaying the options for each product.

[0709] Step 5:

[0710] The user selects the items they wish to purchase from the displayed product list. Once the selection is complete, the device sends the data of the selected items to the server.

[0711] Step 6:

[0712] The server creates an order list based on the user's selection. At the same time, it generates suggestions for complementary or synergistic products to the selected items and sends them to the terminal.

[0713] Step 7:

[0714] The terminal presents the user with supplemental products received from the server and prompts them to make additional selections. If the user selects additional products, that information is also sent to the server.

[0715] Step 8:

[0716] Once the final order details are confirmed, the server generates an order confirmation message for the user and sends it to their device. The user reviews the order details and approves it.

[0717] Step 9:

[0718] The server places an order with the online supermarket's ordering system based on the order list confirmed by the user. This completes the user's online order.

[0719] (Example 1)

[0720] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0721] In online shopping, users sometimes find it difficult to easily find and purchase the right products. This is especially true when choosing ingredients for a new dish or recipe, where selecting the appropriate products can be time-consuming and cumbersome. Furthermore, when suggestions are made without considering the user's past choices or popular products, it becomes difficult to make the best product selection.

[0722] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0723] In this invention, the server includes an input device for receiving shopping requests from users, an information generation device, a presentation device, and a selection analysis device. This allows users to easily find products that suit their needs and efficiently select and purchase products while receiving optimal suggestions that take into account past choices and popularity.

[0724] An "input device" is a means of receiving shopping requests from users.

[0725] An "information generation device" is a means for generating product information based on a request obtained by an input device.

[0726] A "presentation device" is a means of presenting generated product information to the user and accepting their selection.

[0727] A "selection analysis device" is a means of creating an order list based on user selections and suggesting similar products.

[0728] An "ordering device" is a means of completing an electronic transaction after final confirmation of the order list.

[0729] The system according to the present invention is designed to enable users to shop online efficiently. Specific embodiments are described below.

[0730] This system primarily consists of user terminals, servers, and online databases.

[0731] The user enters their shopping request using an input device on the terminal. This input device may include, for example, a keyboard or a microphone for voice recognition. The user can manually enter text or give voice instructions. The entered data is transmitted to the server via the network.

[0732] The server, acting as an information generator, processes input data using algorithms built with programming languages ​​such as Python. These algorithms cross-reference data with a database to identify the ingredients needed for specific dishes, generating product information. This product information includes the name, price, and description of each item.

[0733] The generated product information is transmitted to the terminal via a display device and visually presented on the user's display. The user can view the details of each product on the screen and select the product they wish to purchase. The selection information is returned to the server, which uses a selection analysis device to create an order list. At this time, the server suggests appropriate similar and complementary products using the user's past selection history and popular product data.

[0734] Final confirmation is performed by the user reviewing the order list displayed on the terminal. After the user approves, the server uses the ordering device to confirm the online order and transmits the necessary data to the external ordering system.

[0735] For example, if a user enters "I want to buy ingredients for vegetable soup" into the terminal, the server generates product information such as "cabbage," "carrots," "broccoli," and "consommé" and displays it on the terminal. The user can select the necessary products and, if needed, choose additional products such as "bacon" or "pasta." In this way, the system allows users to easily and efficiently select and purchase the products they want.

[0736] Examples of prompts generated using a generative AI model include "I want to make vegetable soup, please tell me what ingredients I need" and "Please tell me what ingredients you recommend for making a hot pot dish." This allows users to interact with the system using natural language and quickly obtain the information they need.

[0737] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0738] Step 1:

[0739] User input of shopping requests

[0740] The user enters their shopping request using the terminal's input device. This input can be done via text or voice. An example of input might be the prompt "I want to buy ingredients for vegetable soup." The entered data is sent to the server.

[0741] Step 2:

[0742] Product information generation by the server

[0743] The server generates product information based on the prompt text received from the user. Specifically, it searches the database based on the input data and creates a list of related products. This process uses a generative AI model to perform text processing and select the products best suited to the user's needs. For example, the list might include "cabbage," "carrots," "broccoli," and "consommé."

[0744] Step 3:

[0745] Displaying product information on the device

[0746] Product information generated from the server is sent to the terminal. The terminal receives this information and presents it to the user as a visual list. The user is configured to view product details, prices, and descriptions on the display.

[0747] Step 4:

[0748] User selection of products

[0749] The user selects the product they wish to purchase from the displayed product information. Selection can be made using a touch display or mouse. The user's selection results are then sent back to the server.

[0750] Step 5:

[0751] Server-based creation of order list

[0752] The server creates an order list based on the user's selection information. This list includes information about the selected products. Furthermore, by utilizing selection analysis tools, it is possible to analyze the user's past selection history and popular product information to suggest similar or complementary products.

[0753] Step 6:

[0754] Final confirmation by the user

[0755] The user makes a final confirmation of the order list displayed on the terminal. After the user has reviewed everything, they can approve the order, and this information is sent to the server.

[0756] Step 7:

[0757] Order processing by server

[0758] The server sends the final confirmed order list to an external e-commerce system to finalize the online order. This step involves communicating with the external system using the necessary data communication protocols. Once the order is successfully finalized, the server sends a confirmation notification to the user.

[0759] (Application Example 1)

[0760] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0761] In modern society, there is a growing demand for the convenience of online shopping. However, for users, especially the elderly and those unfamiliar with technology, the complex operating procedures and lack of intuitive usability present challenges. Furthermore, users may not receive accurate suggestions when selecting products, leading to wasted time. Additionally, while voice input is becoming widespread, there is a lack of shopping systems that efficiently utilize this technology.

[0762] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0763] In this invention, the server includes an input means for receiving commercial transaction requests from users, an information processing means for converting the input information into text using speech recognition, and an information generation means for suggesting product information based on the requests obtained by the input means. This allows users to obtain product information with simple voice operations and complete their shopping efficiently.

[0764] A "user" is an entity that operates the system and makes requests for commercial transactions.

[0765] A "commercial transaction request" is a set of requirements regarding a product or service that a user is considering purchasing.

[0766] An "input method" is an interface through which a user communicates a business transaction request to the system.

[0767] "Information generation means" refers to a device or process that creates relevant product information based on acquired commercial transaction requests.

[0768] A "presentation means" is an interface that shows product information created by an information generation means to the user and accepts their selection.

[0769] "Analysis tools" refer to functions and processes that create an order list based on user selections and provide additional suggestions.

[0770] The "ordering method" refers to the processing function that allows users to complete online orders after final confirmation of the order list.

[0771] "Speech recognition" is a technology that takes speech as input and converts it into text data.

[0772] "Information processing means" refers to functions and processes that process input speech information using speech recognition technology and convert it into text.

[0773] The "order list" is a list of products and services that a user has decided to order.

[0774] The system implementing this invention consists of a user terminal, a server, a voice recognition service, and an online supermarket database. The user uses a terminal such as a smartphone to perform voice input. This voice input is converted into text by the voice recognition service and sent to the server. The server generates product information based on the acquired text information. This product information is designed to meet the user's needs using an algorithm that includes suggestions according to the type of meal. The server then presents the generated product information to the user's terminal and accepts the user's selection. The information selected by the user is sent to the server, and an order list is created using an analysis means. After the user makes a final confirmation, the information is sent to the online supermarket's ordering system by an ordering means, and the commercial transaction is completed.

[0775] A concrete example of its use is when a user voice-inputs, "I want to make Japanese-style curry for dinner tonight." The voice recognition system then converts the instruction into text, which is sent to the server. Based on this, the server generates and suggests related product information such as "curry roux," "beef," "potatoes," "carrots," and "onions." It can also suggest additional side dishes such as "miso soup" and "rice." Through this entire process, the user can efficiently and easily select the necessary products and confirm their order.

[0776] Examples of prompts include "I'd like to buy a Father's Day lunch set" or "Can you tell me some ingredients for an easy pasta dish?". These allow users to quickly receive product suggestions tailored to specific events or situations.

[0777] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0778] Step 1:

[0779] The user provides voice input via the device. The device receives the voice data and sends it to a voice recognition service. The input is voice, and the output is text data.

[0780] Step 2:

[0781] The speech recognition service converts speech data into text data. The input speech data is analyzed by a recognition algorithm, and the corresponding text is generated. The input is speech data, and the output is text data.

[0782] Step 3:

[0783] The terminal sends the text data received from the speech recognition service to the server. The server receives the text data and begins processing to generate product information. The input is text data, and the output is a product search query.

[0784] Step 4:

[0785] The server searches and retrieves relevant product information from the product database based on the acquired text data. Search operations are performed based on keywords related to the type of food. The input is a product search query, and the output is a list of product information.

[0786] Step 5:

[0787] The server sends a generated list of product information to the terminal, where it is presented to the user. The user reviews the presented products and selects the items they wish to purchase. The input is the product information list, and the output is the user's selection information.

[0788] Step 6:

[0789] The terminal sends user selection information to the server, which uses analysis tools to create an order list based on that information. The server also considers the selection history for additional suggestions. The input is the user's selection information, and the output is the order list.

[0790] Step 7:

[0791] The server notifies the user to perform a final check of the order list, and after confirmation, proceeds to the order placement process. The information is sent to the online supermarket's ordering system, and the transaction is completed. The input is the final confirmed order list, and the output is the order completion notification.

[0792] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0793] The system of the present invention provides comprehensive support to enable users to shop online smoothly. In particular, it includes a function that recognizes the user's emotions and provides product suggestions and selection support based on those emotions.

[0794] System Configuration

[0795] This system is built around a user terminal, a central server, and an emotion engine. The terminal is a tool for users to input shopping requests through an interface. The emotion engine analyzes the user's emotional state and sends the results to the server to optimize product suggestions.

[0796] Emotion recognition and information generation

[0797] The device acquires emotional data using its camera and microphone during user interaction. The server sends this emotional data to an emotion engine, which analyzes the user's emotional state. It also responds to subtle emotional changes to ensure that users can make purchases without stress.

[0798] Based on the emotional state analyzed by the emotion engine, the server generates product information tailored to the user's mood. This information includes options that align with the user's emotions, such as refreshing products for when they are relaxed, or nutritional supplements for when they are feeling down.

[0799] Information presentation and selection

[0800] The generated product information is sent back to the terminal and presented to the user. The user can then select the products they like. The selection is made directly on the terminal, and the system is designed to seamlessly select products that meet the user's preferences through simple touch operations or voice input.

[0801] Creating and optimizing order lists

[0802] The server creates an order list based on the user's selections. The analysis system combines user sentiment data obtained from the sentiment engine with past selection history to suggest optimal additional products. This process aims to provide suggestions that help the user maintain positive emotions.

[0803] Final confirmation and order completion

[0804] The final order list is confirmed by the user via the terminal. Once the user confirms the order, the server sends an order instruction to the online supermarket system, completing the order.

[0805] Specific example

[0806] For example, if a user is emotionally exhausted, the emotion engine recognizes this state and suggests items to lift the user's spirits, such as "relaxing herbal tea" or "popcorn for watching a movie." If the user selects an item, the system adds it to the cart and proceeds to the final confirmation stage.

[0807] This allows the system to go beyond simply supporting product selection and provide a pleasant purchasing experience that resonates with the user's emotions.

[0808] The following describes the processing flow.

[0809] Step 1:

[0810] The user powers on the device and launches the online shopping assistant application. The device activates its camera and microphone for emotion recognition and begins interacting with the user.

[0811] Step 2:

[0812] The terminal provides an interface for users to input their shopping requests and displays prompts such as, "What kind of products are you looking for?" The user then communicates their desire to the terminal via voice or text input, for example, "I want to make a hot pot dish."

[0813] Step 3:

[0814] The device acquires emotional data from the user's facial expressions and voice, and transmits this data to the server in real time. This emotional data includes the user's facial expressions, tone of voice, and word choice.

[0815] Step 4:

[0816] The server processes the received shopping request and emotion data, and uses an emotion engine to analyze the user's psychological state. Based on the analysis results, it generates product information that is appropriate for the user's emotions.

[0817] Step 5:

[0818] The server sends a list of products to the device that includes suggestions tailored to the user's emotional state. For example, if the server determines that the user needs to relax, it will suggest products containing relaxing beverages or ingredients.

[0819] Step 6:

[0820] The terminal visually displays a list of products sent from the server to the user. The user selects the items they wish to purchase from the products displayed on the screen.

[0821] Step 7:

[0822] User selection information is sent back from the terminal to the server. The server creates an order list based on the selected products and makes additional suggestions, taking sentiment data into consideration.

[0823] Step 8:

[0824] The terminal presents the user with additional suggestions from the server and prompts them to make further selections. The user then selects from the suggested items and reviews the final order list.

[0825] Step 9:

[0826] Once the final order list is confirmed, the server requests final confirmation from the user. Upon receiving confirmation, it proceeds with the order confirmation process with the online supermarket's ordering system.

[0827] Step 10:

[0828] The server creates an order completion notification and sends it to the user via the terminal. This allows the user to confirm that their order has been successfully completed.

[0829] (Example 2)

[0830] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0831] Modern online shopping systems face the challenge of failing to optimize user purchasing intent and satisfaction because they do not take into account the individual emotional state or real-time mental condition of users when suggesting products. In particular, when purchasing intent fluctuates depending on the user's emotions, traditional systems cannot respond effectively, resulting in a decrease in the quality of the purchasing experience.

[0832] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0833] In this invention, the server includes emotion recognition means for collecting user emotion data, information generation means for generating product information based on the data acquired by the input means and emotion recognition means, and analysis means for creating an order list based on the user's selection and making additional suggestions based on emotion data. This enables product suggestions that correspond to the user's emotional state, providing a more personalized purchasing experience.

[0834] "Input means" refers to a device or software for receiving shopping requests from users.

[0835] "Emotion recognition means" refers to a device or software for collecting and analyzing user emotion data.

[0836] "Information generation means" refers to a device or software that generates product information based on collected data.

[0837] "Presentation means" refers to a device or software used to present generated product information to the user.

[0838] "Analysis means" refers to a device or software for creating an order list based on user selections and making additional suggestions.

[0839] "Ordering method" refers to a device or software used to complete an online order after final confirmation.

[0840] The present invention provides a method for users to conduct online shopping efficiently and comfortably. The system consists of a user's terminal, a central server, and an emotion engine.

[0841] The terminal features a user interface that provides a means for users to input shopping requests. Through this interface, users can search for specific products or specify items they wish to purchase. The terminal is also equipped with a camera and microphone, which are used to sense the user's facial expressions and voice in real time and collect emotional data.

[0842] The server receives the emotional data and passes it to the emotion engine. The emotion engine uses a generative AI model to analyze the user's emotional state. This analysis accurately determines the user's current mood and emotions, enabling product suggestions tailored to their individual state. For example, when a user wants to relax, fragrant bath items or herbal teas might be suggested.

[0843] The generated product information is sent back to the terminal via the server and presented to the user. The user selects from the suggested products using touch controls or voice commands. Based on the selected information, the server creates an order list and makes additional suggestions that take into account emotional data and past purchase history. Through this process, the user is emotionally satisfied and has a more comfortable shopping experience.

[0844] For example, if a user says, "Suggest products that are more suitable for relaxation," the system will display relevant products in response to that request. In this case, an example of a prompt in the generative AI model might be, "Create a list of products that have a calming effect, based on the user's emotional data and activity history."

[0845] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0846] Step 1:

[0847] The user enters their shopping request through the terminal's interface. The terminal activates its camera and microphone to capture emotional data obtained from the user's facial expressions and voice. The input here consists of the user's purchase preference information and emotional data. This data is output as the system's initial input.

[0848] Step 2:

[0849] The device sends the collected emotional data to the server. The server passes the emotional data to the emotion engine, which then performs emotion analysis using a generative AI model. This analysis involves calculations to determine the user's mood and emotional state. The output is the analysis result, which indicates the user's emotional state.

[0850] Step 3:

[0851] The server generates product information based on the analyzed emotional data. It refers to the product database and performs calculations to identify products that are appropriate for the user's emotions. For example, if the user is tired, relaxation items will be selected. In this process, emotional data is taken as input and product information is output.

[0852] Step 4:

[0853] The server sends the created product information back to the terminal. The terminal presents this product information to the user. The user browses the products and selects items of interest. In this step, information is output to the user through a visual interface.

[0854] Step 5:

[0855] Based on the user's selected product information, the terminal sends data back to the server. The server adds the selected product to the order list and then performs data calculations using sentiment data and past selection history to make additional suggestions. At this stage, the order list and optimized suggested products are output.

[0856] Step 6:

[0857] The terminal presents the user with the final order list and suggested products, and the user confirms the purchase. Upon receiving the purchase confirmation instruction, the server outputs data to complete the online order and sends the order instruction to the relevant system. In this step, the completion of the online order is output.

[0858] (Application Example 2)

[0859] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0860] In online shopping, there is a challenge in that there is no system that can appropriately present product suggestions and payment information according to the user's emotional state, making it difficult for users to efficiently select and pay for products without experiencing stress.

[0861] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0862] In this invention, the server includes data acquisition means, an information generation unit, and emotion analysis means. This makes it possible to provide optimal product suggestions and payment information tailored to the user's emotions.

[0863] "Data acquisition means" refers to devices or methods for receiving information from users.

[0864] An "information generation unit" is a device or method that generates product information and service details based on acquired data.

[0865] "Emotional analysis tools" are devices or methods that analyze a user's emotional data and infer their emotional state.

[0866] A "payment information generation means" is a device or method that suggests appropriate payment methods and campaigns based on the user's emotional state.

[0867] An "information presentation means" is a device or method that displays generated information to a user and accepts their selection.

[0868] "Data analysis means" refers to devices or methods that analyze a user's selection history and related information to generate additional suggestions.

[0869] "Processing management means" refers to devices or methods that complete online order procedures according to the user's selection.

[0870] The system for realizing this application is built around a user terminal, a central server, and an emotion engine. The user terminal uses a camera and microphone as data acquisition tools and functions as an interface to understand the user's emotional state. The emotional data acquired by the terminal is sent to the server and analyzed by emotion analysis tools. Emotion analysis APIs (such as Google Cloud Vision API or IBM Watson Tone Analyzer, which are generative AI models) are used for the analysis. Based on this analysis, the server uses an information generation unit to generate product suggestions and payment information that are tailored to the user's emotions.

[0871] The generated product and payment information is transmitted to the user's terminal via an information display device, where the user can review and select their items. The user's selection is analyzed by the server's data analysis device, and additional suggestions are made as needed. Once the order is confirmed, the processing management device completes the online order. This system is designed to ensure a pleasant shopping experience for the user.

[0872] For example, if the system analyzes that a user is feeling stressed, the server will suggest "discount coupons" or "cashback campaigns" based on the emotion analysis to alleviate the user's feelings. Furthermore, it utilizes a generative AI model to predict what kind of campaigns a user might be interested in using prompt messages. By using prompt messages such as, "What kind of campaigns or payment options would be best to suggest when a user is currently experiencing stress regarding their purchase?", more personalized suggestions become possible.

[0873] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0874] Step 1:

[0875] The user's device uses a camera and microphone to capture the user's facial expressions and voice data. This data is sent to the server as emotion data. The input is the user's facial image and voice data, and the output is the emotion data sent to the server.

[0876] Step 2:

[0877] The server analyzes the received emotional data through emotion analysis tools. Using generative AI models such as the Google Cloud Vision API and IBM Watson Tone Analyzer, it classifies the user's emotional state into categories such as "relaxed" and "stressed." The input is the user's emotional data, and the output is the analyzed emotional state.

[0878] Step 3:

[0879] The server uses an information generation unit to generate appropriate product suggestions and payment information based on the analyzed emotional state. Using a generation AI model, it leverages prompts such as, "What campaigns or payment options are best to suggest when the user is currently experiencing stress regarding purchasing?" to derive appropriate information. The input is the analyzed emotional state, and the output is the generated product suggestions and payment information.

[0880] Step 4:

[0881] The server sends the generated product and payment information to the user's terminal. The terminal displays this information to the user through an information display device, allowing the user to review it and select products and payment methods. The input is the generated product and payment information, and the output is the user's selection.

[0882] Step 5:

[0883] The server analyzes user selections using data analysis tools and provides additional suggestions as needed. This analysis uses algorithms that consider the user's selection history and trending product information to provide optimal suggestions. The input is the user's selections and historical data, and the output is optimized additional suggestions.

[0884] Step 6:

[0885] Once the user selects and confirms the items they wish to purchase, the server uses processing and management tools to complete the online order. The input is the final confirmed order list, and the output is the completed order information.

[0886] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0887] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0888] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0889] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0890] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0891] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0892] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0893] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0894] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0895] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0896] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0897] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0898] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0899] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0900] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0901] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0902] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0903] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0904] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0905] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0906] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0907] The following is further disclosed regarding the embodiments described above.

[0908] (Claim 1)

[0909] An input method for receiving shopping requests from users,

[0910] An information generation means that proposes product information based on a request obtained by the input means,

[0911] A presentation means for presenting the aforementioned product information to the user and accepting their selection,

[0912] An analytical tool that creates an order list based on user selections and makes additional suggestions,

[0913] An ordering method for completing an online order after final confirmation of the aforementioned order list,

[0914] A system that includes this.

[0915] (Claim 2)

[0916] The system according to claim 1, which includes an algorithm that suggests appropriate ingredients based on the type of dish.

[0917] (Claim 3)

[0918] The system according to claim 1, wherein the analysis means for making the additional suggestions has a function to optimize the suggestions by taking into account the user's selection history and popular product information.

[0919] "Example 1"

[0920] (Claim 1)

[0921] An input device that accepts shopping requests from users,

[0922] An information generation device that generates product information based on a request obtained by the input device,

[0923] A presentation device that presents the aforementioned product information to the user and accepts their selection,

[0924] A selection analysis device that creates an order list based on user selections and suggests similar products,

[0925] An ordering device that completes electronic transactions after final confirmation of the aforementioned order list,

[0926] A system that includes this.

[0927] (Claim 2)

[0928] The system according to claim 1, which includes an algorithm that suggests appropriate components based on the type of cooking of the information generating device.

[0929] (Claim 3)

[0930] The system according to claim 1, wherein the selection analysis device has a function to optimize suggestions by taking into account the user's selection history and popular product information.

[0931] "Application Example 1"

[0932] (Claim 1)

[0933] An input method for receiving commercial transaction requests from users,

[0934] An information generation means that proposes product information based on a request obtained by the input means,

[0935] A presentation means for presenting the aforementioned product information to the user and accepting their selection,

[0936] An analytical tool that creates an order list based on user selections and makes additional suggestions,

[0937] An ordering method in which an online order is completed after final confirmation of the aforementioned order list,

[0938] Information processing means that converts input information into text using speech recognition,

[0939] A system that includes this.

[0940] (Claim 2)

[0941] The system according to claim 1, which includes an algorithm that suggests appropriate ingredients based on the type of meal, as an information generation means.

[0942] (Claim 3)

[0943] The system according to claim 1, wherein the analysis means for making the additional suggestions has a function to optimize the suggestions by taking into account the user's selection history and popular product information.

[0944] "Example 2 of combining an emotion engine"

[0945] (Claim 1)

[0946] An input method for receiving shopping requests from users,

[0947] A means of emotion recognition that collects user emotion data,

[0948] Information generation means that generates product information based on data acquired by the input means and emotion recognition means,

[0949] A presentation means for presenting the aforementioned product information to the user and accepting their selection,

[0950] An analytical tool that creates an order list based on user selections and makes additional suggestions based on sentiment data,

[0951] An ordering method for completing an online order after final confirmation of the aforementioned order list,

[0952] A system that includes this.

[0953] (Claim 2)

[0954] The system according to claim 1, wherein the information generation means includes an algorithm that analyzes the user's emotional state and selects a product suitable for their mood.

[0955] (Claim 3)

[0956] The system according to claim 1, wherein the analysis means has the function of making optimized additional suggestions by combining past selection history and sentiment data.

[0957] "Application example 2 when combining with an emotional engine"

[0958] (Claim 1)

[0959] A data acquisition method for receiving shopping requests from users,

[0960] An information generation unit that proposes product information based on a request obtained by the aforementioned data acquisition means,

[0961] A sentiment analysis method that acquires and analyzes sentiment data through interaction with the user,

[0962] A payment information generation means that proposes appropriate payment methods and campaigns based on the user's emotional state obtained by the emotion analysis means,

[0963] Information presentation means for presenting the aforementioned product information and payment information to the user and accepting their selection,

[0964] A data analysis tool that creates an order list based on user selections and makes additional suggestions,

[0965] A processing management means for completing online orders after final confirmation of the aforementioned order list,

[0966] A system that includes this.

[0967] (Claim 2)

[0968] The system according to claim 1, wherein the information generation unit includes an algorithm that suggests appropriate ingredients based on the type of dish.

[0969] (Claim 3)

[0970] The system according to claim 1, wherein the data analysis means has a function to optimize suggestions by taking into account the user's selection history and trending product information, and further includes a function to adjust suggestions using the user's emotional state. [Explanation of symbols]

[0971] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An input method for receiving commercial transaction requests from users, An information generation means that proposes product information based on a request obtained by the input means, A presentation means for presenting the aforementioned product information to the user and accepting their selection, An analytical tool that creates an order list based on user selections and makes additional suggestions, An ordering method in which an online order is completed after final confirmation of the aforementioned order list, Information processing means that converts input information into text using speech recognition, A system that includes this.

2. The system according to claim 1, which includes an algorithm that suggests appropriate ingredients based on the type of meal, as an information generation means.

3. The system according to claim 1, wherein the analysis means for making the additional suggestions has a function to optimize the suggestions by taking into account the user's selection history and popular product information.

Images