System

Abstract

To provide a system.SOLUTION: A system is to understand a user's preference and needs and propose the optimal commodity, and comprises: means that understands a user's needs, age, and gender through an interaction with the user; means that proposes clothes and the like in accordance with the details of a commodity that the user wants to purchase or a specific event; and means that acquires the user's past purchase history or reaction to the commodity, and improves the accuracy of proposing commodities.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to solve the problems of conventional e-commerce applications by proposing optimal products that match the user's preferences and needs. [Means for solving the problem]

[0005] The present invention provides the following means for accurately grasping the needs and desires of users and proposing appropriate products.

[0006] 1. Means of collecting information through user interaction:

[0007] Through dialogue with the user, information such as the user's needs, age, gender, etc. is obtained, allowing the user's preferences and needs to be accurately understood.

[0008] 2. Proposals tailored to product details or specific events:

[0009] It provides details about the products the user wants to purchase, as well as clothing suggestions for specific events (e.g. weddings and business meetings), allowing it to suggest the best products for the user.

[0010] 3. Leverage past purchase history and product responses:

[0011] The system obtains and analyzes users' past purchase history and reactions (reviews) to products from online shopping sites, etc. This allows the system to understand trends in users' preferences and needs, enabling it to suggest products with greater precision.

[0012] By using the above means, the present invention accurately grasps user needs and proposes optimal products, thereby solving the problems of conventional e-commerce applications.

[0013] "User preferences and needs" refers to what a user wants, is interested in, or needs.

[0014] The "best product" refers to the product that best suits the user's preferences and needs.

[0015] "Suggest" refers to presenting specific items or options to the user.

[0016] "Collecting information through dialogue" refers to collecting user requests and information through conversation and communication with the user.

[0017] "Suggestions tailored to product details or specific events" refers to taking into consideration the details of the product the user wants to purchase or factors related to a specific event, and making suggestions about products that match those.

[0018] "Utilizing past purchase history and reactions to products" refers to using information such as a user's past purchase history and reviews and ratings of products by other users to make product suggestions and improvements. [Brief description of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] 4 is a sequence diagram showing a process flow of the data processing system according to the first embodiment. FIG. [Figure 12] 11 is a sequence diagram showing a process flow of the data processing system in application example 1. FIG. [Figure 13] FIG. 11 is a sequence diagram showing the flow of processing of the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in application example 2 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0023] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] An embodiment for implementing the present invention includes the following elements.

[0041] 1. Server:

[0042] It provides an interface for interacting with the user.

[0043] Collect and process user responses and requests.

[0044] Search for and suggest appropriate products from online shopping sites, etc.

[0045] 2. Terminal:

[0046] It provides an interface for interacting with the user.

[0047] It is responsible for communicating with the server and sending the user's responses and requests to the server.

[0048] Receives suggestions and information from the server and displays them to the user.

[0049] 3. User:

[0050] It interacts with servers and terminals through an interface.

[0051] Provide details about a product you want to purchase or information related to a particular event.

[0052] Browse and select suggested products.

[0053] 4. Program:

[0054] Controls the transfer and processing of data between the server and the terminal.

[0055] Analyze the user's responses and requests to search for and suggest appropriate products.

[0056] Acquire and analyze data from online shopping sites, etc.

[0057] (Specific examples)

[0058] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[0059] 1. The server initiates a dialogue with the user via the terminal.

[0060] 2. The user provides information such as wedding theme, dress preferences, budget, etc.

[0061] 3. The server analyzes the user's information and searches for suitable dresses from online shopping sites.

[0062] 4. The server sends the search results to the terminal and suggests them to the user.

[0063] 5. The user browses the details and photos of the proposed dresses and selects the one they like. 6. The server provides a link to the selected dress to proceed with the purchase process.

[0064] As described above, by transferring and processing data between the server and the terminal, it is possible to realize proposals of products that are optimally tailored to the user's needs.

[0065] The process flow of each embodiment will be described below.

[0066] Step 1: Collect user responses

[0067] The server initiates a dialogue with the user via the terminal.

[0068] The terminal receives the answer from the user and transmits it to the server.

[0069] The server receives and analyzes the user's answers.

[0070] Step 2: Product search and suggestions

[0071] The server searches for suitable products from online shopping sites based on the user's answers.

[0072] The server transmits the search results to the terminal and suggests them to the user.

[0073] The terminal displays the suggested products for the user to view.

[0074] Step 3: Check product details

[0075] The user views detailed information and photos of the suggested products.

[0076] The terminal transmits information about the product selected by the user to the server.

[0077] Step 4: Providing checkout

[0078] The server provides a link to proceed with the purchase for the selected item.

[0079] The terminal displays the link provided by the server, allowing the user to complete the purchase procedure.

[0080] Step 5: Collect and analyze data

[0081] The server collects the user's past purchase history and reactions to products and stores them in a database.

[0082] The server analyzes the collected data to identify trends regarding users' preferences and needs.

[0083] As described above, a series of processes are carried out, from collecting the user's answers, searching for and suggesting products, confirming product details, providing purchasing procedures, and collecting and analyzing data. This allows the system to suggest products that are optimally tailored to the user's needs.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0086] Conventional online shopping systems have difficulty in accurately grasping a user's preferences and needs and proposing optimal products. In addition, there is a lack of a means to effectively use a user's purchase history and reactions to products to improve the accuracy of product suggestions, which makes it difficult to increase user satisfaction. Furthermore, there is a lack of links to help users smoothly proceed through the purchase process.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes a means for understanding the preferences and needs of a user and proposing optimal products, a means for understanding the needs, age, and gender of the user through dialogue with the user, a means for providing details of a product the user wants to purchase and a suggestion of clothing to match a specific event, a means for analyzing user information using a generative AI model and searching for suitable products from an online database, a means for transmitting search results obtained by the generative AI model to the user's terminal and displaying them to the user, and a means for providing a link for proceeding with a product purchase procedure based on the user's selection. This makes it possible to propose products based on the user's needs, improves the accuracy of the proposal, and enables the user to smoothly complete the purchase procedure.

[0089] A "generative AI model" is an artificial intelligence algorithm that analyzes information provided by a user and searches for and suggests appropriate products.

[0090] The means of "understanding the user's preferences and needs" refers to collecting and analyzing specific information such as the user's needs, age, and gender from the information input and dialogue of the user.

[0091] "Means of understanding the user's needs, age, and gender through dialogue" is the process of grasping basic information with the user through interactive communication.

[0092] The means of "suggesting clothing according to details or specific events" is a process of collecting detailed information about products that a user desires according to a specific purpose or event, and then suggesting appropriate products based on the information.

[0093] "Means of acquiring past purchase history and reactions to products to improve the accuracy of product suggestions" refers to collecting and analyzing users' past purchase history and product reviews, etc., and using this as a basis to improve the accuracy of future product suggestions.

[0094] "Means for searching for suitable products from an online database" refers to the process of using collected user information to search for products that meet the user's requirements from an online database.

[0095] The "means for transmitting the search results to the user's terminal and displaying them" refers to a process in which the server processes the searched product information and transmits it to the user's device for display.

[0096] The "means for providing a link to proceed with the product purchase procedure" refers to the process of providing an interface such as a web link or button that allows the user to purchase the selected product.

[0097] The system of this invention has the function of understanding the preferences and needs of users and proposing optimal products. This system is mainly composed of three elements: a server, a terminal, and a user, and operates according to the following specific flow.

[0098] Hardware and Software Used

[0099] Server: Cloud services such as Amazon Web Services (AWS(TM)), Google(TM) Cloud Platform (GCP)

[0100] Devices: Smartphones, tablets, personal computers (PCs), etc.

[0101] Program: Programming languages ​​such as Python, JavaScript (registered trademark), Node.js (registered trademark), etc.

[0102] Online database: An online shopping site that sells products over the Internet.

[0103] Server Features

[0104] The server initiates a dialogue with the user and collects information such as the user's needs, age, gender, etc. It then uses a generative AI model to analyze this information and search an online database for products that match the user's preferences. The analysis uses natural language processing techniques and keyword extraction algorithms.

[0105] Device Features

[0106] The terminal provides an interface for interactive dialogue with the user. It transmits information input by the user to the server and displays product information received from the server to the user. This information includes product images, prices, detailed information, etc.

[0107] User Actions

[0108] The user interacts with the system through a terminal and inputs details of the desired product and information related to a specific event. For example, a specific prompt such as "I'm looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen" is used. Based on this information, the system suggests appropriate products.

[0109] Examples

[0110] 1. Example prompt:

[0111] The user inputs his / her preference using the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[0112] 2. Server analysis:

[0113] The server uses a generative AI model to analyze the user's preferences and extract the keyword "blue vintage dress."

[0114] Next, it uses the API of an online shopping site to search for a suitable dress.

[0115] 3. Submit search results:

[0116] The server sends the search result "Blue vintage dress (price: 28,000 yen)" in JSON format to the terminal.

[0117] The terminal receives this information and displays it to the user.

[0118] 4. User Choices and Purchase Procedures:

[0119] The user can view the proposed dresses through the terminal and select the one they like.

[0120] The terminal transmits information selected by the user to the server, and the server generates a link for proceeding with the purchase procedure and transmits it to the terminal.

[0121] The user clicks on this link to complete the purchase.

[0122] In this way, the server and the terminal work together to suggest optimal products based on the user's needs and to process the purchase.

[0123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0124] Step 1:

[0125] Prompt input from the user

[0126] A user accesses the system through a terminal and inputs detailed information about the product he or she desires. Specifically, he or she inputs the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[0127] Input: The prompt text entered by the user.

[0128] Output: The terminal sends the prompt to the server.

[0129] Step 2:

[0130] Prompt by terminal

[0131] The terminal sends the prompt text entered by the user to the server using an HTTP POST request.

[0132] Input: The prompt text entered by the user

[0133] Output: The server receives the prompt.

[0134] Step 3:

[0135] Prompt parsing by the server

[0136] The server analyzes the received prompt using a generative AI model (e.g., natural language processing model), extracts specific keywords (e.g., "vintage," "blue," "under 30,000 yen"), and understands the user's needs.

[0137] Input: Prompt sent from the terminal

[0138] Data Processing: Text Analysis with Generative AI Models

[0139] Output: Extracted keywords and user needs information

[0140] Step 4:

[0141] Product search by server

[0142] The server searches for suitable products using an online database (e.g., an online shopping site API) based on the extracted keywords and the user's needs information. For example, a search for "blue vintage dresses" can be performed using an online shopping site API.

[0143] Input: Extracted keywords and user needs information

[0144] Data Computing: Sending queries to online databases and retrieving search results

[0145] Output: A list of searched products

[0146] Step 5:

[0147] Server sends search results

[0148] The server processes the list of products obtained in JSON format and sends it to the terminal. For example, the information may be formatted as "Blue vintage dress (price: 28,000 yen)."

[0149] Input: A list of products retrieved from an online database

[0150] Data processing: Formatting data into JSON format

[0151] Output: List of processed product information

[0152] Step 6:

[0153] Displaying search results by device

[0154] The terminal displays the product information received from the server to the user. The user can view product images, prices, and detailed information. For example, a photo of the product, its price, and detailed information are displayed.

[0155] Input: A list of product information sent from the server

[0156] Output: Product information displayed to the user

[0157] Step 7:

[0158] User selection of products

[0159] The user selects a product he likes from the displayed products, for example by clicking a "Buy this dress" button.

[0160] Input: User browses and selects products

[0161] Output: Selected product information

[0162] Step 8:

[0163] Sending selection information by the terminal

[0164] The terminal transmits the product information selected by the user to the server.

[0165] Input: Product information selected by the user

[0166] Output: Selections sent to the server

[0167] Step 9:

[0168] Server-generated checkout link

[0169] The server generates a checkout link based on the product selected by the user and sends it to the terminal. For example, it generates a link that says "Click here to proceed with the checkout."

[0170] Input: Product information selected by the user

[0171] Data processing: generating checkout link

[0172] Output: Checkout link

[0173] Step 10:

[0174] Displaying links by terminal

[0175] The terminal displays the purchase procedure link received from the server to the user, who can click on this link to proceed with the purchase.

[0176] Input: Checkout link sent by the server

[0177] Output: The checkout link shown to the user.

[0178] In this way, the server and the terminal work together to suggest products that meet the user's needs and facilitate a smooth purchasing process.

[0179] (Application example 1)

[0180] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0181] Conventional online shopping systems have limitations in their ability to suggest optimal products based on users' preferences and needs, so customers often have difficulty finding the products they actually want. Even in brick-and-mortar stores, it is difficult for store clerks to instantly suggest appropriate products to each individual customer, which can lead to a poor customer experience.

[0182] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0183] In this invention, the server includes a means for understanding the needs, age, and sex of a user through dialogue with the user, a means for providing details of a product the user wants to purchase and clothing suggestions for a specific event, a means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions, and a means for suggesting products to customers in the store in real time through smart glasses and displaying them on a display. This makes it possible to quickly and accurately suggest products that customers want and improve the shopping experience in a physical store.

[0184] "User" refers to an individual who uses the System to search for and purchase Products.

[0185] "Preferences" refer to personal tastes and preferences that a user has for particular products or services.

[0186] "Needs" refers to the requirements or desires that a user has in a specific situation or for a specific purpose.

[0187] "Product" refers to a specific item or service offered to a user.

[0188] "Server" refers to a central computer that handles user interaction and processes information.

[0189] "Dialogue" refers to the exchange of information or communication between a user and a server or terminal.

[0190] "Purchase history" refers to a record of products and services a user has previously purchased.

[0191] "Response" refers to the evaluation or feedback a user gives to a proposed product or service.

[0192] "Smart glasses" refers to a wearable device that has a built-in display that displays information and can interact with the user.

[0193] "Real-time" means that information or data is provided immediately.

[0194] "Display" refers to a display device for visually presenting information to a user.

[0195] The system that realizes this application example mainly consists of three elements: a server, a terminal wearing smart glasses, and a user.

[0196] The server plays a key role in interacting with users, collecting and analyzing their information, and proposing appropriate products. Specifically, the server uses the following hardware and software:

[0197] Central computer (server): This is the heart of the system and processes and stores data.

[0198] Database Management System (DBMS): Used to store and manage users' purchase history and past responses.

[0199] AI model (generative AI model): An artificial intelligence system that analyzes user requirements and makes product suggestions.

[0200] The server first interacts with the user to collect information such as their needs, age, and gender, through voice recognition technology and text input, and then analyzes the collected data to provide the user with details about the products they need and suggestions tailored to specific events.

[0201] The smart glasses used as a terminal act as an intermediary between the user and the server. The smart glasses use the following hardware and software:

[0202] Smart Glasses Display: Displaying information to the user in real time.

[0203] Speech recognition functionality: Captures customer requests as voice input.

[0204] Communication module: Used to send and receive data to and from the server.

[0205] As a specific example of its use, a user puts on the smart glasses in a store and issues a voice command such as, "I'm looking for an elegant wedding dress for under 50,000 yen." This command is sent to the server, which searches for suitable products in an online database and displays the results on the smart glasses' display.

[0206] Users can view detailed information and photos of the suggested products through the smart glasses. This allows users to compare multiple products without moving around the store. If they find a product they like, they can proceed with the purchase on the spot.

[0207] Specific examples of prompts for this system are as follows:

[0208] "I am looking for an elegant dress under 50000 for a wedding. Please let me know my options."

[0209] This will dramatically improve the customer experience in physical stores and contribute to increasing customer satisfaction.

[0210] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0211] Step 1:

[0212] The user makes a request through the smart glasses by voice input or text input. For example, the user may say, "I'm looking for an elegant dress for a wedding under 50,000 yen." The smart glasses convert this request into text data using voice recognition and send the text data to the server.

[0213] Input: User's voice command

[0214] Output: Instruction data in text format

[0215] Step 2:

[0216] The server analyzes the received text data and performs natural language processing to identify the user's requirements (needs, preferences, budget, etc.) The analysis results are sent to a database as a specific search query.

[0217] Input: Instruction data in text format

[0218] Output: The search query

[0219] Step 3:

[0220] The server searches an online database for suitable products based on the search query, taking into account the user's past purchase history and reactions, and uses a generative AI model to list the most suitable products.

[0221] Input: Search query, user's past purchase history, response data

[0222] Output: Product list

[0223] Step 4:

[0224] The server converts the search results into a data format suitable for display on the smart glasses display and sends it to the device (smart glasses). This data includes detailed product information and photos.

[0225] Input: Product list

[0226] Output: Product details, photos

[0227] Step 5:

[0228] The terminal (smart glasses) visually displays the data received from the server, and the user can view detailed information and photos of the proposed products through the smart glasses display.

[0229] Input: Product details, photo

[0230] Output: Information displayed on the smart glasses display

[0231] Step 6:

[0232] The user selects the product they like from the suggested products, and the selection result is sent to the server via the smart glasses.

[0233] Input: User's choice

[0234] Output: Selected product data

[0235] Step 7:

[0236] The server generates a link and information for the selected product to proceed with the purchase procedure and sends it to the device (smart glasses), allowing the user to complete the purchase procedure on the spot.

[0237] Input: Selected product data

[0238] Output: Checkout information

[0239] In this way, users can receive product suggestions in real time through the smart glasses and proceed with the purchase process for products that interest them, providing a convenient and efficient shopping experience.

[0240] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0241] An embodiment for implementing the present invention includes the following elements.

[0242] 1. Emotion Engine:

[0243] To provide a means for recognizing emotions from a user's speech, facial expressions, etc.

[0244] Provide a means for transmitting recognized emotion information to a server.

[0245] To provide a means for adjusting product suggestions and dialogue progress based on a user's emotions.

[0246] 2. Server:

[0247] It provides an interface for interacting with the user.

[0248] Receive and process user responses and emotional information.

[0249] Adjust product suggestions and conversation progression based on emotional information.

[0250] 3. Terminal:

[0251] It provides an interface for interacting with the user.

[0252] It is responsible for communicating with the server and sending the user's answers and emotion information to the server.

[0253] Receives suggestions and information from the server and displays them to the user.

[0254] 4. Program:

[0255] Controls the transfer and processing of data between the server and the terminal.

[0256] User responses and emotional information are analyzed to search for and suggest appropriate products.

[0257] The progress of the dialogue is adjusted based on emotional information, and appropriate services and support are provided to the user.

[0258] (Specific examples)

[0259] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[0260] 1. The device recognizes emotions from the user's speech, facial expressions, etc. and transmits them to the emotion engine.

[0261] 2. The emotion engine analyzes the user's emotion information and sends the information to the server.

[0262] 3. The server receives the user's emotional information and adjusts product suggestions and the progress of the dialogue.

[0263] 4. The server searches and suggests suitable dresses from online shopping sites based on the user's answers and emotional information.

[0264] 5. The device displays detailed information and photos of the suggested dresses to the user.

[0265] 6. The user selects the suggested dress and proceeds with the purchase.

[0266] 7. The server provides a checkout link for the selected dress.

[0267] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[0268] The process flow of each embodiment will be described below.

[0269] Step 1: Recognizing emotions and sending information

[0270] The terminal recognizes the user's emotions from their speech and facial expressions.

[0271] The terminal transmits the recognized emotion information to the emotion engine.

[0272] The emotion engine analyzes the received emotion information and transmits it to the server.

[0273] Step 2: Make suggestions and tailor dialogue based on emotions

[0274] The server receives the emotion information received from the emotion engine.

[0275] The server analyzes the emotion information and adjusts the progress of the dialogue and suggests products appropriate to the user's emotions.

[0276] The server searches and suggests suitable products from online shopping sites.

[0277] Step 3: View suggested products

[0278] The terminal receives detailed information and photos of the suggested products from the server.

[0279] The terminal displays the suggested items to the user.

[0280] The user browses the suggested products and makes a selection.

[0281] Step 4: Proceed with your purchase

[0282] The terminal receives a link from the server to proceed with the purchase procedure for the product selected by the user.

[0283] The terminal displays a link to the user, allowing them to complete the purchase procedure.

[0284] Step 5: Collect and analyze sentiment information

[0285] The server collects the user's emotion information and stores it in a database.

[0286] The server analyzes the collected emotion information to understand the user's emotional tendencies and preferences.

[0287] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[0288] Example 2

[0289] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0290] Conventional online shopping systems suggest products without considering the user's emotions or mood at the time, making it difficult to suggest optimal products to users. In addition, since dialogue progression and product suggestions are not based on the user's emotions, there is a problem of a poor user experience. Furthermore, there is a lack of personalized suggestions to increase the user's motivation to purchase. For this reason, it is necessary to realize optimal product suggestions and dialogue progression that take into account the user's emotions.

[0291] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0292] In this invention, the server includes a means for analyzing the user's speech and facial expressions to recognize emotions, a means for transmitting the recognized emotion information to the server, and a means for adjusting the content of product proposals and the progress of the dialogue based on the user's emotions, thereby enabling optimal product proposals and dialogue progress that take the user's emotions into account.

[0293] "User" refers to an individual who uses the System to search for, select, and purchase Products.

[0294] "Dialogue" refers to interaction between a user and a system with the purpose of exchanging information.

[0295] "Needs" refers to the demands and desires for products and services required by users.

[0296] "Age" refers to the number of years calculated from the user's date of birth.

[0297] "Gender" refers to the sex of the user, such as male or female.

[0298] "Utterance" refers to the act of a user providing information to a system via voice.

[0299] "Facial expressions" refer to the movements and expressions of a user's face and are used to recognize emotions.

[0300] "Emotion" refers to the psychological state a user exhibits in response to a particular situation or stimulus.

[0301] An "emotion engine" refers to software that analyzes emotions from a user's speech and facial expressions.

[0302] "Server" refers to the central system that processes user information, makes product suggestions, and advances dialogue.

[0303] "Online shopping site" refers to a website that sells products over the Internet.

[0304] "Purchase history" refers to a record of products and services a user has previously purchased.

[0305] "Product recommendation accuracy" refers to the ability to appropriately recommend products based on a user's needs and emotions.

[0306] "Search conditions" refer to product search parameters that are set based on a user's needs and feelings.

[0307] "Purchase procedure" refers to the process by which a user officially purchases a product selected by the user.

[0308] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0309] A specific embodiment of the present invention is an online shopping system in which a user utilizes a terminal to receive product suggestions and progress in dialogue via a server. The system aims to improve the user experience by recognizing the user's emotions and making optimal product suggestions based on those emotions. A specific method for implementing the present invention will be described in detail below.

[0310] Emotion Engine

[0311] A user connects to the system using a device (such as a smartphone or tablet). The device has a built-in camera and microphone. The device collects the user's speech and facial expressions in real time via the camera and microphone, and transmits them to the emotion engine. The emotion engine uses voice recognition software and facial expression recognition software to analyze emotions from the user's speech and facial expressions.

[0312] The emotion engine generates the analysis results as numbers or categories (e.g., "happiness," "sadness," "surprise," etc.) and sends this information to a server. The software used by the emotion engine is a cloud-based API that enables real-time emotion analysis.

[0313] Server Features

[0314] The server receives the emotion information sent from the emotion engine. Based on this information, the server adjusts the dialogue scenario with the user and the product suggestions. For example, if the user expresses the emotion of "joy," the server will proceed with the dialogue with content optimized for the user's emotion, such as suggesting a dress with a gorgeous design and bright colors.

[0315] The server uses the user's speech and emotional information to search for suitable products on the online shopping site. It then selects the most suitable product from the search results and sends it to the terminal. This allows the server to provide the best product suggestions for the user.

[0316] Device Features

[0317] The terminal provides an interface that displays the product suggestions received from the server and the progress of the dialogue to the user. The user can view detailed information and photos of the suggested products on the terminal, select a product as necessary, and proceed with the purchase procedure.

[0318] As a concrete example, consider the case where a user is looking for a dress to attend a wedding. The user connects to the system through a terminal and says, "I'm looking for a dress for a wedding." The user's actions are captured by the camera and recorded by the microphone.

[0319] The device sends this information to the emotion engine, which detects emotions such as "happiness" and "anticipation." The server searches for suitable dresses from online shopping sites based on this emotion information and suggests them to the user. The user checks the details and photos of the suggested dresses on the device, selects the one they like, and proceeds with the purchase process.

[0320] Examples of prompt statements

[0321] For example, the following is an example of a prompt that a user may enter into the system:

[0322] I'm looking for a recommended dress to wear to a wedding.

[0323] "Please suggest a black formal dress that would please our users."

[0324] In this way, the entire system works together to recognize the user's emotions and make optimal product suggestions.

[0325] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0326] Step 1:

[0327] A user logs into the system through a terminal and inputs a request to search for a wedding dress.

[0328] Specifically, the user enters their user ID and password into the login screen of the terminal. Then, they enter "I'm looking for a wedding dress" into the search bar and click the search button. This sends a search request to the system.

[0329] Input: User login information and search query

[0330] Output: A search request to the server.

[0331] Step 2:

[0332] The device collects the user's speech and facial expressions using a camera and microphone and sends them to the emotion engine.

[0333] Specifically, the device captures the user's facial expressions with a built-in camera and records audio with a microphone, and these data are sent to the emotion engine in real time.

[0334] Input: User's speech and facial expression data

[0335] Output: Sending data to the emotion engine

[0336] Step 3:

[0337] The emotion engine analyzes the user's emotions and sends the information to the server.

[0338] Specifically, the emotion engine converts voice data into text using an API that converts voice data into text data, and analyzes facial expression data using an API that analyzes facial expression data. This allows emotion categories such as "happiness" and "expectation" to be identified. The analysis results are sent to the server as emotion information.

[0339] Input: Text data of utterances and facial expression feature data

[0340] Output: Emotion information

[0341] Step 4:

[0342] The server adjusts product search conditions based on the emotion information received.

[0343] Specifically, the server receives the emotion information and sets the characteristics of products that have attributes that the user finds pleasing (e.g., gorgeousness or color) as search conditions. It also takes into account the user's past purchase history and preferences to personalize the search conditions.

[0344] Input: Emotional information and past purchase history data

[0345] Output: Optimized search criteria

[0346] Step 5:

[0347] The server searches for suitable dresses on an online shopping site and transmits the results to the terminal.

[0348] Specifically, the server uses the API of the online shopping site to perform a product search based on the search criteria set by the user, selects the product that best matches the emotion information from the search results, and sends its details to the device.

[0349] Input: Optimized search criteria

[0350] Output:Details of selected product

[0351] Step 6:

[0352] The device will display detailed information and photos of the suggested dresses to the user.

[0353] Specifically, the device displays the dress information received from the server on a GUI, allowing the user to swipe through photos of the dress or tap on detailed information to view it.

[0354] Input:Details of selected product

[0355] Output: Information displayed to the user

[0356] Step 7:

[0357] The user selects a suggested dress and proceeds with the purchase.

[0358] Specifically, the user selects a dress from the suggestions and clicks the "Purchase" button. The terminal displays a checkout screen, allowing the user to enter credit card information and a shipping address. The user then confirms the purchase and clicks the "Confirm Purchase" button.

[0359] Input: User selection and purchase procedure information

[0360] Output: Purchase confirmation notice

[0361] This allows the entire system to work together to provide optimal product suggestions and purchasing procedures that take the user's emotions into account.

[0362] (Application example 2)

[0363] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] Conventional online shopping systems have difficulty in accurately understanding a user's preferences and needs and suggesting appropriate products. In addition, since product recommendations cannot be made taking into account the user's emotions, user satisfaction may decrease. Furthermore, in dialogue-based interactions, the progress of the dialogue cannot be adjusted according to the user's emotions, which may result in a poor user experience.

[0365] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for recognizing emotions from the user's speech and facial expressions, a means for transmitting to the server based on emotion information and making appropriate product recommendations, a means for understanding the needs, age, and gender through dialogue with the user, a means for providing details of a product the user wants to purchase and outfit suggestions suited to a specific event, a means for acquiring the user's past purchase history and reactions to products and improving the accuracy of product suggestions, and a means for adjusting the progress of the dialogue based on emotion information. This makes it possible to realize appropriate product suggestions and optimal interactions based on the user's emotions, improving the user experience.

[0366] "Means for recognizing emotions from a user's speech and facial expressions" refers to a method for identifying emotions by analyzing a user's facial expressions and voice using sensors such as a camera and microphone.

[0367] The "means for transmitting to a server based on emotion information and making appropriate product recommendations" is a method for transmitting recognized emotion data to a server and proposing optimal products based on that information.

[0368] "Means for understanding the needs, age, and gender of a user through dialogue" refers to a method for using a chatbot or voice assistant to interact with a user and collect information about the user's needs, age, and gender.

[0369] "Means for providing details of products that a user wishes to purchase and suggestions of appropriate attire for a specific event" refers to a method for providing detailed information of products that a user wishes to purchase and suggestions of appropriate attire for a specific event or situation.

[0370] "Means of acquiring a user's past purchase history and reactions to products to improve the accuracy of product suggestions" refers to a method of collecting a user's history of past purchases and the user's impressions and feedback about those products, and using that information to improve the accuracy of product recommendations.

[0371] The "means for adjusting the progress of a dialogue based on emotional information" is a method for optimizing the speed and content of a dialogue based on the user's current emotional state.

[0372] The embodiment of the present invention is a smart shopping assistant system that recommends products based on the user's emotions. This system analyzes the user's facial expressions and speech when the user uses a smartphone to shop online, and the server suggests appropriate products based on the emotional information.

[0373] First, the user's device is equipped with a camera and a microphone, and these sensors are used to capture the user's facial expressions and voice. The specific hardware used here is a smartphone's front camera and built-in microphone. The captured data is processed using OpenCV or a similar image processing library to recognize emotions from facial expressions.

[0374] A pre-trained emotion recognition model (e.g., a deep learning model built with Keras) is used to recognize user emotions. This model analyzes input image data and identifies the user's emotional state (e.g., joy, sadness, anger, etc.).

[0375] The recognized emotion information is then sent to a server via the Internet. The server receives this emotion information and combines it with the user's past purchase history and reaction data to search for and recommend the most suitable products. The server also takes into account information registered by the user, such as needs, age, and gender, to improve the accuracy of product recommendations.

[0376] In addition, the script in charge of the dialogue with the user is executed by the server, and the dialogue progress is adjusted based on the emotion information. For example, if the user is interested but has a slightly dissatisfied expression, the dialogue script is adjusted to carefully explain the product's features and benefits. On the other hand, if the user has a happy expression, a more relaxed dialogue style is selected.

[0377] A distinctive feature of this system is the adjustment of the dialogue progress based on emotional information, which allows users to experience product recommendations and interfaces optimized for their own emotional state, improving the overall satisfaction of the shopping experience.

[0378] As a concrete example, consider a case where a user is looking for a dress to attend a wedding. When the user speaks into the smartphone camera, the facial expression and speech are analyzed by the emotion engine. The analyzed emotion information is then sent to the server, which suggests dresses that match the user's needs. This process makes it easier for the user to find a dress that fits their emotions.

[0379] Example of a prompt to be input to a generative AI model:

[0380] Create a smartphone application for an online shopping site that provides a product recommendation service based on the user's emotions. The application will recognize emotions from the user's facial expressions and voice, and suggest the most suitable products based on those emotions.

[0381] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0382] Step 1:

[0383] A user starts a smartphone app and turns on the camera and microphone. The user starts talking to the screen or searching for information about a product. The input data here is the user's facial image and voice, which are captured by the camera and microphone. Facial expression data and voice data are acquired through the user's operations.

[0384] Step 2:

[0385] The device processes the captured video and audio in real time to perform face and voice recognition. Specifically, it uses the OpenCV library to detect faces and crop specific facial regions. The cropped facial images are input to a pre-trained emotion recognition model (a deep learning model using Keras). Here, the input data is the image of the facial region, and the output is emotion information (e.g., anger, joy, sadness, etc.).

[0386] Step 3:

[0387] The device sends the recognized emotion information and voice data to the server, where the device formats the data and sends it to the server using an HTTP request. The input is the payload of emotion information and voice data, which is sent to the server. The server receives the request and starts to analyze the emotion information and voice data.

[0388] Step 4:

[0389] The server understands the user's needs based on the received emotion information and voice data, and searches for appropriate products by referring to past purchase history and user feedback. In this step, the server executes a database query to search the user profile and filters products based on the obtained emotion information. The inputs are emotion information, voice data, and user profile information from the database. The output is a list of recommended products.

[0390] Step 5:

[0391] The server sends the recommended product list to the terminal. Here, the product list generated by the server is converted to JSON format or similar and sent to the terminal as an HTTP response. The input is the recommended product list, and the output is the response sent to the terminal.

[0392] Step 6:

[0393] The terminal displays the received product list to the user. The user can check the product details on the screen and select the items he likes. Here, the input is the product list received from the server, and the output is the product information displayed to the user. The terminal updates the user interface and allows the user to select products.

[0394] Step 7:

[0395] When the user selects a product, the terminal sends the selection information to the server. The server then proceeds with the purchase procedure based on the received information. Specifically, it securely processes credit card information and links with the ordering system. The input here is the user's selection information, and the output is a notification that the purchase procedure has been completed.

[0396] Through the above steps, optimal product recommendations based on emotion information and smooth interaction with the user are realized.

[0397] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0398] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0399] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0400] [Second embodiment]

[0401] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0402] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0403] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0404] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0405] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0406] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0407] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0408] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0409] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0410] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0411] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0412] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0413] An embodiment for implementing the present invention includes the following elements.

[0414] 1. Server:

[0415] It provides an interface for interacting with the user.

[0416] Collect and process user responses and requests.

[0417] Search for and suggest appropriate products from online shopping sites, etc.

[0418] 2. Terminal:

[0419] It provides an interface for interacting with the user.

[0420] It is responsible for communicating with the server and sending the user's responses and requests to the server.

[0421] Receives suggestions and information from the server and displays them to the user.

[0422] 3. User:

[0423] It interacts with servers and terminals through an interface.

[0424] Provide details about a product you want to purchase or information related to a particular event.

[0425] Browse and select suggested products.

[0426] 4. Program:

[0427] Controls the transfer and processing of data between the server and the terminal.

[0428] Analyze the user's responses and requests to search for and suggest appropriate products.

[0429] Acquire and analyze data from online shopping sites, etc.

[0430] (Specific examples)

[0431] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[0432] 1. The server initiates a dialogue with the user via the terminal.

[0433] 2. The user provides information such as wedding theme, dress preferences, budget, etc.

[0434] 3. The server analyzes the user's information and searches for suitable dresses from online shopping sites.

[0435] 4. The server sends the search results to the terminal and suggests them to the user.

[0436] 5. The user browses the details and photos of the proposed dresses and selects the one they like. 6. The server provides a link to the selected dress to proceed with the purchase process.

[0437] As described above, by transferring and processing data between the server and the terminal, it is possible to realize proposals of products that are optimally tailored to the user's needs.

[0438] The process flow will be explained below.

[0439] Step 1: Collect user responses

[0440] The server initiates a dialogue with the user via the terminal.

[0441] The terminal receives the answer from the user and transmits it to the server.

[0442] The server receives and analyzes the user's answers.

[0443] Step 2: Product search and suggestions

[0444] The server searches for suitable products from online shopping sites based on the user's answers.

[0445] The server transmits the search results to the terminal and suggests them to the user.

[0446] The terminal displays the suggested products for the user to view.

[0447] Step 3: Check product details

[0448] The user views detailed information and photos of the suggested products.

[0449] The terminal transmits information about the product selected by the user to the server.

[0450] Step 4: Providing checkout

[0451] The server provides a link to proceed with the purchase for the selected item.

[0452] The terminal displays the link provided by the server, allowing the user to complete the purchase procedure.

[0453] Step 5: Collect and analyze data

[0454] The server collects the user's past purchase history and reactions to products and stores them in a database.

[0455] The server analyzes the collected data to identify trends regarding users' preferences and needs.

[0456] As described above, a series of processes are carried out, from collecting the user's answers, searching for and suggesting products, confirming product details, providing purchasing procedures, and collecting and analyzing data. This allows the system to suggest products that are optimally tailored to the user's needs.

[0457] Example 1

[0458] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0459] Conventional online shopping systems have difficulty in accurately grasping a user's preferences and needs and proposing optimal products. In addition, there is a lack of a means to effectively use a user's purchase history and reactions to products to improve the accuracy of product suggestions, which makes it difficult to increase user satisfaction. Furthermore, there is a lack of links to help users smoothly proceed through the purchase process.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0461] In this invention, the server includes a means for understanding the preferences and needs of a user and proposing optimal products, a means for understanding the needs, age, and gender of the user through dialogue with the user, a means for providing details of a product the user wants to purchase and a suggestion of clothing to match a specific event, a means for analyzing user information using a generative AI model and searching for suitable products from an online database, a means for transmitting search results obtained by the generative AI model to the user's terminal and displaying them to the user, and a means for providing a link for proceeding with a product purchase procedure based on the user's selection. This makes it possible to propose products based on the user's needs, improves the accuracy of the proposal, and enables the user to smoothly complete the purchase procedure.

[0462] A "generative AI model" is an artificial intelligence algorithm that analyzes information provided by a user and searches for and suggests appropriate products.

[0463] The means of "understanding the user's preferences and needs" refers to collecting and analyzing specific information such as the user's needs, age, and gender from the information input and dialogue of the user.

[0464] "Means of understanding the user's needs, age, and gender through dialogue" is the process of grasping basic information with the user through interactive communication.

[0465] The means of "suggesting clothing according to details or specific events" is a process of collecting detailed information about products that a user desires according to a specific purpose or event, and then suggesting appropriate products based on the information.

[0466] "Means of acquiring past purchase history and reactions to products to improve the accuracy of product suggestions" refers to collecting and analyzing users' past purchase history and product reviews, etc., and using this as a basis to improve the accuracy of future product suggestions.

[0467] "Means for searching for suitable products from an online database" refers to the process of using collected user information to search for products that meet the user's requirements from an online database.

[0468] The "means for transmitting the search results to the user's terminal and displaying them" refers to a process in which the server processes the searched product information and transmits it to the user's device for display.

[0469] The "means for providing a link to proceed with the product purchase procedure" refers to the process of providing an interface such as a web link or button that allows the user to purchase the selected product.

[0470] The system of this invention has the function of understanding the preferences and needs of users and proposing optimal products. This system is mainly composed of three elements: a server, a terminal, and a user, and operates according to the following specific flow.

[0471] Hardware and Software Used

[0472] Server: Cloud services such as Amazon Web Services (AWS) and Google Cloud Platform (GCP)

[0473] Devices: Smartphones, tablets, personal computers (PCs), etc.

[0474] Programming: Programming languages ​​such as Python, JavaScript, Node.js, etc.

[0475] Online database: An online shopping site that sells products over the Internet.

[0476] Server Features

[0477] The server initiates a dialogue with the user and collects information such as the user's needs, age, gender, etc. It then uses a generative AI model to analyze this information and search an online database for products that match the user's preferences. The analysis uses natural language processing techniques and keyword extraction algorithms.

[0478] Device Features

[0479] The terminal provides an interface for interactive dialogue with the user. It transmits information input by the user to the server and displays product information received from the server to the user. This information includes product images, prices, detailed information, etc.

[0480] User Actions

[0481] The user interacts with the system through a terminal and inputs details of the desired product and information related to a specific event. For example, a specific prompt such as "I'm looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen" is used. Based on this information, the system suggests appropriate products.

[0482] Examples

[0483] 1. Example prompt:

[0484] The user inputs his / her preference using the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[0485] 2. Server analysis:

[0486] The server uses a generative AI model to analyze the user's preferences and extract the keyword "blue vintage dress."

[0487] Next, it uses the API of an online shopping site to search for a suitable dress.

[0488] 3. Submit search results:

[0489] The server sends the search result "Blue vintage dress (price: 28,000 yen)" in JSON format to the terminal.

[0490] The terminal receives this information and displays it to the user.

[0491] 4. User Choices and Purchase Procedures:

[0492] The user can view the proposed dresses through the terminal and select the one they like.

[0493] The terminal transmits information selected by the user to the server, and the server generates a link for proceeding with the purchase procedure and transmits it to the terminal.

[0494] The user clicks on this link to complete the purchase.

[0495] In this way, the server and the terminal work together to suggest optimal products based on the user's needs and to process the purchase.

[0496] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0497] Step 1:

[0498] Prompt input from the user

[0499] A user accesses the system through a terminal and inputs detailed information about the product he or she desires. Specifically, he or she inputs the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[0500] Input: The prompt text entered by the user.

[0501] Output: The terminal sends the prompt to the server.

[0502] Step 2:

[0503] Prompt by terminal

[0504] The terminal sends the prompt text entered by the user to the server using an HTTP POST request.

[0505] Input: The prompt text entered by the user

[0506] Output: The server receives the prompt.

[0507] Step 3:

[0508] Prompt parsing by the server

[0509] The server analyzes the received prompt using a generative AI model (e.g., natural language processing model), extracts specific keywords (e.g., "vintage," "blue," "under 30,000 yen"), and understands the user's needs.

[0510] Input: Prompt sent from the terminal

[0511] Data Processing: Text Analysis with Generative AI Models

[0512] Output: Extracted keywords and user needs information

[0513] Step 4:

[0514] Product search by server

[0515] The server searches for suitable products using online databases (e.g., online shopping site APIs) based on the extracted keywords and user needs information. For example, searching for "blue vintage dresses" using the Amazon API.

[0516] Input: Extracted keywords and user needs information

[0517] Data Computing: Sending queries to online databases and retrieving search results

[0518] Output: A list of searched products

[0519] Step 5:

[0520] Server sends search results

[0521] The server processes the list of products obtained in JSON format and sends it to the terminal. For example, the information may be formatted as "Blue vintage dress (price: 28,000 yen)."

[0522] Input: A list of products retrieved from an online database

[0523] Data processing: Data reformatting to JSON format

[0524] Output: List of processed product information

[0525] Step 6:

[0526] Displaying search results by device

[0527] The terminal displays the product information received from the server to the user. The user can view product images, prices, and detailed information. For example, a photo of the product, its price, and detailed information are displayed.

[0528] Input: A list of product information sent from the server

[0529] Output: Product information displayed to the user

[0530] Step 7:

[0531] User selection of products

[0532] The user selects a product he likes from the displayed products, for example by clicking a "Buy this dress" button.

[0533] Input: User browses and selects products

[0534] Output: Selected product information

[0535] Step 8:

[0536] Sending selection information by the terminal

[0537] The terminal transmits the product information selected by the user to the server.

[0538] Input: Product information selected by the user

[0539] Output: Selections sent to the server

[0540] Step 9:

[0541] Server-generated checkout link

[0542] The server generates a checkout link based on the product selected by the user and sends it to the terminal. For example, it generates a link that says "Click here to proceed with the checkout."

[0543] Input: Product information selected by the user

[0544] Data processing: generating checkout link

[0545] Output: Checkout link

[0546] Step 10:

[0547] Displaying links by terminal

[0548] The terminal displays the purchase procedure link received from the server to the user, who can click on this link to proceed with the purchase.

[0549] Input: Checkout link sent by the server

[0550] Output: The checkout link shown to the user.

[0551] In this way, the server and the terminal work together to suggest products that meet the user's needs and facilitate a smooth purchasing process.

[0552] (Application example 1)

[0553] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0554] Conventional online shopping systems have limitations in their ability to suggest optimal products based on users' preferences and needs, so customers often have difficulty finding the products they actually want. Even in brick-and-mortar stores, it is difficult for store clerks to instantly suggest appropriate products to each individual customer, which can lead to a poor customer experience.

[0555] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0556] In this invention, the server includes a means for understanding the needs, age, and sex of a user through dialogue with the user, a means for providing details of a product the user wants to purchase and clothing suggestions for a specific event, a means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions, and a means for suggesting products to customers in the store in real time through smart glasses and displaying them on a display. This makes it possible to quickly and accurately suggest products that customers want and improve the shopping experience in a physical store.

[0557] "User" refers to an individual who uses the System to search for and purchase Products.

[0558] "Preferences" refer to personal tastes and preferences that a user has for particular products or services.

[0559] "Needs" refers to the requirements or desires that a user has in a specific situation or for a specific purpose.

[0560] "Product" refers to a specific item or service offered to a user.

[0561] "Server" refers to a central computer that handles user interaction and processes information.

[0562] "Dialogue" refers to the exchange of information or communication between a user and a server or terminal.

[0563] "Purchase history" refers to a record of products and services a user has previously purchased.

[0564] "Response" refers to the evaluation or feedback a user gives to a proposed product or service.

[0565] "Smart glasses" refers to a wearable device that has a built-in display that displays information and can interact with the user.

[0566] "Real-time" means that information or data is provided immediately.

[0567] "Display" refers to a display device for visually presenting information to a user.

[0568] The system that realizes this application example mainly consists of three elements: a server, a terminal wearing smart glasses, and a user.

[0569] The server plays a key role in interacting with users, collecting and analyzing their information, and proposing appropriate products. Specifically, the server uses the following hardware and software:

[0570] Central computer (server): This is the heart of the system and processes and stores data.

[0571] Database Management System (DBMS): Used to store and manage users' purchase history and past responses.

[0572] AI model (generative AI model): An artificial intelligence system that analyzes user requirements and makes product suggestions.

[0573] The server first interacts with the user to gather information about the user's needs, age, gender, etc. This interaction is carried out through voice recognition technology and text input, and then analyzes the collected data to provide the user with details about the products they need and suggestions tailored to specific events.

[0574] The smart glasses used as a terminal act as an intermediary between the user and the server. The smart glasses use the following hardware and software:

[0575] Smart Glasses Display: Displaying information to the user in real time.

[0576] Speech recognition functionality: Captures customer requests as voice input.

[0577] Communication module: Used to send and receive data to and from the server.

[0578] As a specific example of its use, a user puts on the smart glasses in a store and issues a voice command such as, "I'm looking for an elegant wedding dress for under 50,000 yen." This command is sent to the server, which searches for suitable products in an online database and displays the results on the smart glasses' display.

[0579] Users can view detailed information and photos of the suggested products through the smart glasses. This allows users to compare multiple products without moving around the store. If they find a product they like, they can proceed with the purchase on the spot.

[0580] Specific examples of prompts for this system are as follows:

[0581] "I am looking for an elegant dress under 50000 for a wedding. Please let me know my options."

[0582] This will dramatically improve the customer experience in physical stores and contribute to increasing customer satisfaction.

[0583] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0584] Step 1:

[0585] The user makes a request through the smart glasses by voice input or text input. For example, the user may say, "I'm looking for an elegant dress for a wedding under 50,000 yen." The smart glasses convert this request into text data using voice recognition and send the text data to the server.

[0586] Input: User's voice command

[0587] Output: Instruction data in text format

[0588] Step 2:

[0589] The server analyzes the received text data and performs natural language processing to identify the user's requirements (needs, preferences, budget, etc.) The analysis results are sent to a database as a specific search query.

[0590] Input: Instruction data in text format

[0591] Output: The search query

[0592] Step 3:

[0593] The server searches an online database for suitable products based on the search query, taking into account the user's past purchase history and reactions, and uses a generative AI model to list the most suitable products.

[0594] Input: Search query, user's past purchase history, response data

[0595] Output: Product list

[0596] Step 4:

[0597] The server converts the search results into a data format suitable for display on the smart glasses display and sends it to the device (smart glasses). This data includes detailed product information and photos.

[0598] Input: Product list

[0599] Output: Product details, photos

[0600] Step 5:

[0601] The terminal (smart glasses) visually displays the data received from the server, and the user can view detailed information and photos of the proposed products through the smart glasses display.

[0602] Input: Product details, photo

[0603] Output: Information displayed on the smart glasses display

[0604] Step 6:

[0605] The user selects the product they like from the suggested products, and the selection result is sent to the server via the smart glasses.

[0606] Input: User's choice

[0607] Output: Selected product data

[0608] Step 7:

[0609] The server generates a link and information for the selected product to proceed with the purchase procedure and sends it to the device (smart glasses), allowing the user to complete the purchase procedure on the spot.

[0610] Input: Selected product data

[0611] Output: Checkout information

[0612] In this way, users can receive product suggestions in real time through the smart glasses and proceed with the purchase process for products that interest them, providing a convenient and efficient shopping experience.

[0613] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0614] An embodiment for implementing the present invention includes the following elements.

[0615] 1. Emotion Engine:

[0616] To provide a means for recognizing emotions from a user's speech, facial expressions, etc.

[0617] Provide a means for transmitting recognized emotion information to a server.

[0618] To provide a means for adjusting product suggestions and dialogue progress based on a user's emotions.

[0619] 2. Server:

[0620] It provides an interface for interacting with the user.

[0621] Receive and process user responses and emotional information.

[0622] Adjust product suggestions and conversation progression based on emotional information.

[0623] 3. Terminal:

[0624] It provides an interface for interacting with the user.

[0625] It is responsible for communicating with the server and sending the user's answers and emotion information to the server.

[0626] Receives suggestions and information from the server and displays them to the user.

[0627] 4. Program:

[0628] Controls the transfer and processing of data between the server and the terminal.

[0629] User responses and emotional information are analyzed to search for and suggest appropriate products.

[0630] The progress of the dialogue is adjusted based on emotional information, and appropriate services and support are provided to the user.

[0631] (Specific examples)

[0632] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[0633] 1. The device recognizes emotions from the user's speech, facial expressions, etc. and transmits them to the emotion engine.

[0634] 2. The emotion engine analyzes the user's emotion information and sends the information to the server.

[0635] 3. The server receives the user's emotional information and adjusts product suggestions and the progress of the dialogue.

[0636] 4. The server searches and suggests suitable dresses from online shopping sites based on the user's answers and emotional information.

[0637] 5. The device displays detailed information and photos of the suggested dresses to the user.

[0638] 6. The user selects the suggested dress and proceeds with the purchase.

[0639] 7. The server provides a checkout link for the selected dress.

[0640] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[0641] The process flow will be explained below.

[0642] Step 1: Recognizing emotions and sending information

[0643] The terminal recognizes the user's emotions from their speech and facial expressions.

[0644] The terminal transmits the recognized emotion information to the emotion engine.

[0645] The emotion engine analyzes the received emotion information and transmits it to the server.

[0646] Step 2: Make suggestions and tailor dialogue based on emotions

[0647] The server receives the emotion information received from the emotion engine.

[0648] The server analyzes the emotion information and adjusts the progress of the dialogue and suggests products appropriate to the user's emotions.

[0649] The server searches and suggests suitable products from online shopping sites.

[0650] Step 3: View suggested products

[0651] The terminal receives detailed information and photos of the suggested products from the server.

[0652] The terminal displays the suggested items to the user.

[0653] The user browses the suggested products and makes a selection.

[0654] Step 4: Proceed with your purchase

[0655] The terminal receives a link from the server to proceed with the purchase procedure for the product selected by the user.

[0656] The terminal displays a link to the user, allowing them to complete the purchase procedure.

[0657] Step 5: Collect and analyze sentiment information

[0658] The server collects the user's emotion information and stores it in a database.

[0659] The server analyzes the collected emotion information to understand the user's emotional tendencies and preferences.

[0660] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[0661] Example 2

[0662] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0663] Conventional online shopping systems suggest products without considering the user's emotions or mood at the time, making it difficult to suggest optimal products to users. In addition, since dialogue progression and product suggestions are not based on the user's emotions, there is a problem of a poor user experience. Furthermore, there is a lack of personalized suggestions to increase the user's motivation to purchase. For this reason, it is necessary to realize optimal product suggestions and dialogue progression that take into account the user's emotions.

[0664] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0665] In this invention, the server includes a means for analyzing the user's speech and facial expressions to recognize emotions, a means for transmitting the recognized emotion information to the server, and a means for adjusting the content of product proposals and the progress of the dialogue based on the user's emotions, thereby enabling optimal product proposals and dialogue progress that take the user's emotions into account.

[0666] "User" refers to an individual who uses the System to search for, select, and purchase Products.

[0667] "Dialogue" refers to interaction between a user and a system with the purpose of exchanging information.

[0668] "Needs" refers to the requests and desires for products and services that users require.

[0669] "Age" refers to the number of years calculated from the user's date of birth.

[0670] "Gender" refers to the sex of the user, such as male or female.

[0671] "Utterance" refers to the act of a user providing information to a system via voice.

[0672] "Facial expressions" refer to the movements and expressions of a user's face and are used to recognize emotions.

[0673] "Emotion" refers to the psychological state a user exhibits in response to a particular situation or stimulus.

[0674] An "emotion engine" refers to software that analyzes emotions from a user's speech and facial expressions.

[0675] "Server" refers to the central system that processes user information, makes product suggestions, and advances dialogue.

[0676] "Online shopping site" refers to a website that sells products over the Internet.

[0677] "Purchase history" refers to a record of products and services a user has previously purchased.

[0678] "Product recommendation accuracy" refers to the ability to appropriately recommend products based on a user's needs and emotions.

[0679] "Search conditions" refer to product search parameters that are set based on a user's needs and feelings.

[0680] "Purchase procedure" refers to the process by which a user officially purchases a product selected by the user.

[0681] A specific embodiment of the present invention is an online shopping system in which a user utilizes a terminal to receive product suggestions and progress in dialogue via a server. The system aims to improve the user experience by recognizing the user's emotions and making optimal product suggestions based on those emotions. A specific method for implementing the present invention will be described in detail below.

[0682] Emotion Engine

[0683] A user connects to the system using a device (such as a smartphone or tablet). The device has a built-in camera and microphone. The device collects the user's speech and facial expressions in real time via the camera and microphone, and transmits them to the emotion engine. The emotion engine uses voice recognition software and facial expression recognition software to analyze emotions from the user's speech and facial expressions.

[0684] The emotion engine generates the analysis results as numbers or categories (e.g., "happiness," "sadness," "surprise," etc.) and sends this information to a server. The software used by the emotion engine is a cloud-based API that enables real-time emotion analysis.

[0685] Server Features

[0686] The server receives the emotion information sent from the emotion engine. Based on this information, the server adjusts the dialogue scenario with the user and the product suggestions. For example, if the user expresses the emotion of "joy," the server will proceed with the dialogue with content optimized for the user's emotion, such as suggesting a dress with a gorgeous design and bright colors.

[0687] The server uses the user's speech and emotional information to search for suitable products on the online shopping site. It then selects the most suitable product from the search results and sends it to the terminal. This allows the server to provide the best product suggestions for the user.

[0688] Device Features

[0689] The terminal provides an interface that displays the product suggestions received from the server and the progress of the dialogue to the user. The user can view detailed information and photos of the suggested products on the terminal, select a product as necessary, and proceed with the purchase procedure.

[0690] As a concrete example, consider the case where a user is looking for a dress to attend a wedding. The user connects to the system through a terminal and says, "I'm looking for a dress for a wedding." The user's actions are captured by the camera and recorded by the microphone.

[0691] The device sends this information to the emotion engine, which detects emotions such as "happiness" and "anticipation." The server searches for suitable dresses from online shopping sites based on this emotion information and suggests them to the user. The user checks the details and photos of the suggested dresses on the device, selects the one they like, and proceeds with the purchase process.

[0692] Examples of prompt statements

[0693] For example, the following is an example of a prompt that a user may enter into the system:

[0694] I'm looking for a recommended dress to wear to a wedding.

[0695] "Please suggest a black formal dress that would please our users."

[0696] In this way, the entire system works together to recognize the user's emotions and make optimal product suggestions.

[0697] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0698] Step 1:

[0699] A user logs into the system through a terminal and inputs a request to search for a wedding dress.

[0700] Specifically, the user enters their user ID and password into the login screen of the terminal. Then, they enter "I'm looking for a wedding dress" into the search bar and click the search button. This sends a search request to the system.

[0701] Input: User login information and search query

[0702] Output: A search request to the server.

[0703] Step 2:

[0704] The device collects the user's speech and facial expressions using a camera and microphone and sends them to the emotion engine.

[0705] Specifically, the device captures the user's facial expressions with a built-in camera and records audio with a microphone, and these data are sent to the emotion engine in real time.

[0706] Input: User's speech and facial expression data

[0707] Output: Sending data to the emotion engine

[0708] Step 3:

[0709] The emotion engine analyzes the user's emotions and sends the information to the server.

[0710] Specifically, the emotion engine converts voice data to text using an API that converts voice data to text, and analyzes facial expression data using an API that analyzes facial expression data. This identifies emotion categories such as "happiness" or "expectation." The analysis results are sent to the server as emotion information.

[0711] Input: Text data of utterances and facial expression feature data

[0712] Output: Emotion information

[0713] Step 4:

[0714] The server adjusts product search conditions based on the emotion information received.

[0715] Specifically, the server receives the emotion information and sets the characteristics of products that have attributes that the user finds pleasing (e.g., gorgeousness or color) as search conditions. It also takes into account the user's past purchase history and preferences to personalize the search conditions.

[0716] Input: Emotional information and past purchase history data

[0717] Output: Optimized search criteria

[0718] Step 5:

[0719] The server searches for suitable dresses on an online shopping site and transmits the results to the terminal.

[0720] Specifically, the server uses the API of the online shopping site to perform a product search based on the search criteria set by the user, selects the product that best matches the emotion information from the search results, and sends its details to the device.

[0721] Input: Optimized search criteria

[0722] Output:Details of selected product

[0723] Step 6:

[0724] The device will display detailed information and photos of the suggested dresses to the user.

[0725] Specifically, the device displays the dress information received from the server on a GUI, allowing the user to swipe through photos of the dress or tap on detailed information to view it.

[0726] Input:Details of selected product

[0727] Output: Information displayed to the user

[0728] Step 7:

[0729] The user selects a suggested dress and proceeds with the purchase.

[0730] Specifically, the user selects a dress from the suggestions and clicks the "Purchase" button. The terminal displays a checkout screen, allowing the user to enter credit card information and a shipping address. The user then confirms the purchase and clicks the "Confirm Purchase" button.

[0731] Input: User selection and purchase procedure information

[0732] Output: Purchase confirmation notice

[0733] This allows the entire system to work together to provide optimal product suggestions and purchasing procedures that take the user's emotions into account.

[0734] (Application example 2)

[0735] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0736] Conventional online shopping systems have difficulty in accurately understanding a user's preferences and needs and suggesting appropriate products. In addition, since product recommendations cannot be made taking into account the user's emotions, user satisfaction may decrease. Furthermore, in dialogue-based interactions, the progress of the dialogue cannot be adjusted according to the user's emotions, which may result in a poor user experience.

[0737] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for recognizing emotions from the user's speech and facial expressions, a means for transmitting to the server based on emotion information and making appropriate product recommendations, a means for understanding the needs, age, and gender through dialogue with the user, a means for providing details of a product the user wants to purchase and outfit suggestions suited to a specific event, a means for acquiring the user's past purchase history and reactions to products and improving the accuracy of product suggestions, and a means for adjusting the progress of the dialogue based on emotion information. This makes it possible to realize appropriate product suggestions and optimal interactions based on the user's emotions, improving the user experience.

[0738] "Means for recognizing emotions from a user's speech and facial expressions" refers to a method for identifying emotions by analyzing a user's facial expressions and voice using sensors such as a camera and microphone.

[0739] The "means for transmitting to a server based on emotion information and making appropriate product recommendations" is a method for transmitting recognized emotion data to a server and proposing optimal products based on that information.

[0740] "Means for understanding the needs, age, and gender of a user through dialogue" refers to a method for using a chatbot or voice assistant to interact with a user and collect information about the user's needs, age, and gender.

[0741] "Means for providing details of products that a user wishes to purchase and suggestions of appropriate attire for a specific event" refers to a method for providing detailed information of products that a user wishes to purchase and suggestions of appropriate attire for a specific event or situation.

[0742] "Means of acquiring a user's past purchase history and reactions to products to improve the accuracy of product suggestions" refers to a method of collecting a user's history of past purchases and the user's impressions and feedback about those products, and using that information to improve the accuracy of product recommendations.

[0743] The "means for adjusting the progress of a dialogue based on emotional information" is a method for optimizing the speed and content of a dialogue based on the user's current emotional state.

[0744] The embodiment of the present invention is a smart shopping assistant system that recommends products based on the user's emotions. This system analyzes the user's facial expressions and speech when the user uses a smartphone to shop online, and the server suggests appropriate products based on the emotional information.

[0745] First, the user's device is equipped with a camera and a microphone, and these sensors are used to capture the user's facial expressions and voice. The specific hardware used here is a smartphone's front camera and built-in microphone. The captured data is processed using OpenCV or a similar image processing library to recognize emotions from facial expressions.

[0746] A pre-trained emotion recognition model (e.g., a deep learning model built with Keras) is used to recognize user emotions. This model analyzes input image data and identifies the user's emotional state (e.g., joy, sadness, anger, etc.).

[0747] The recognized emotion information is then sent to a server via the Internet. The server receives this emotion information and combines it with the user's past purchase history and reaction data to search for and recommend the most suitable products. The server also takes into account information registered by the user, such as needs, age, and gender, to improve the accuracy of product recommendations.

[0748] In addition, the script in charge of the dialogue with the user is executed by the server, and the dialogue progress is adjusted based on the emotion information. For example, if the user is interested but has a slightly dissatisfied expression, the dialogue script is adjusted to carefully explain the product's features and benefits. On the other hand, if the user has a happy expression, a more relaxed dialogue style is selected.

[0749] A distinctive feature of this system is the adjustment of the dialogue progress based on emotional information, which allows users to experience product recommendations and interfaces optimized for their own emotional state, improving the overall satisfaction of the shopping experience.

[0750] As a concrete example, consider a case where a user is looking for a dress to attend a wedding. When the user speaks into the smartphone camera, the facial expression and speech are analyzed by the emotion engine. The analyzed emotion information is then sent to the server, which suggests dresses that match the user's needs. This process makes it easier for the user to find a dress that fits their emotions.

[0751] Example of a prompt to be input to a generative AI model:

[0752] Create a smartphone application for an online shopping site that provides a product recommendation service based on the user's emotions. The application will recognize emotions from the user's facial expressions and voice, and suggest the most suitable products based on those emotions.

[0753] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0754] Step 1:

[0755] A user starts a smartphone app and turns on the camera and microphone. The user starts talking to the screen or searching for information about a product. The input data here is the user's facial image and voice, which are captured by the camera and microphone. Facial expression data and voice data are acquired through the user's operations.

[0756] Step 2:

[0757] The device processes the captured video and audio in real time to perform face and voice recognition. Specifically, it uses the OpenCV library to detect faces and crop specific facial regions. The cropped facial images are input to a pre-trained emotion recognition model (a deep learning model using Keras). Here, the input data is the image of the facial region, and the output is emotion information (e.g., anger, joy, sadness, etc.).

[0758] Step 3:

[0759] The device sends the recognized emotion information and voice data to the server, where the device formats the data and sends it to the server using an HTTP request. The input is the payload of emotion information and voice data, which is sent to the server. The server receives the request and starts to analyze the emotion information and voice data.

[0760] Step 4:

[0761] The server understands the user's needs based on the received emotion information and voice data, and searches for appropriate products by referring to past purchase history and user feedback. In this step, the server executes a database query to search the user profile and filters products based on the obtained emotion information. The inputs are emotion information, voice data, and user profile information from the database. The output is a list of recommended products.

[0762] Step 5:

[0763] The server sends the recommended product list to the terminal. Here, the product list generated by the server is converted to JSON format or similar and sent to the terminal as an HTTP response. The input is the recommended product list, and the output is the response sent to the terminal.

[0764] Step 6:

[0765] The terminal displays the received product list to the user. The user can check the product details on the screen and select the items he likes. Here, the input is the product list received from the server, and the output is the product information displayed to the user. The terminal updates the user interface and allows the user to select products.

[0766] Step 7:

[0767] When the user selects a product, the terminal sends the selection information to the server. The server then proceeds with the purchase procedure based on the received information. Specifically, it securely processes credit card information and links with the ordering system. The input here is the user's selection information, and the output is a notification that the purchase procedure has been completed.

[0768] Through the above steps, optimal product recommendations based on emotion information and smooth interaction with the user are realized.

[0769] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0770] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0771] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0772] [Third embodiment]

[0773] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0774] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0775] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0776] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0777] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0778] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0779] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0780] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0781] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0782] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0783] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0784] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server", and the headset type terminal 314 will be referred to as the "terminal".

[0785] An embodiment for implementing the present invention includes the following elements.

[0786] 1. Server:

[0787] It provides an interface for interacting with the user.

[0788] Collect and process user responses and requests.

[0789] Search for and suggest appropriate products from online shopping sites, etc.

[0790] 2. Terminal:

[0791] It provides an interface for interacting with the user.

[0792] It is responsible for communicating with the server and sending the user's responses and requests to the server.

[0793] Receives suggestions and information from the server and displays them to the user.

[0794] 3. User:

[0795] It interacts with servers and terminals through an interface.

[0796] Provide details about a product you want to purchase or information related to a particular event.

[0797] Browse and select suggested products.

[0798] 4. Program:

[0799] Controls the transfer and processing of data between the server and the terminal.

[0800] Analyze the user's responses and requests to search for and suggest appropriate products.

[0801] Acquire and analyze data from online shopping sites, etc.

[0802] (Specific examples)

[0803] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[0804] 1. The server initiates a dialogue with the user via the terminal.

[0805] 2. The user provides information such as wedding theme, dress preferences, budget, etc.

[0806] 3. The server analyzes the user's information and searches for suitable dresses from online shopping sites.

[0807] 4. The server sends the search results to the terminal and suggests them to the user.

[0808] 5. The user browses the details and photos of the proposed dresses and selects the one they like. 6. The server provides a link to the selected dress to proceed with the purchase process.

[0809] As described above, by transferring and processing data between the server and the terminal, it is possible to realize proposals of products that are optimally tailored to the user's needs.

[0810] The process flow will be explained below.

[0811] Step 1: Collect user responses

[0812] The server initiates a dialogue with the user via the terminal.

[0813] The terminal receives the answer from the user and transmits it to the server.

[0814] The server receives and analyzes the user's answers.

[0815] Step 2: Product search and suggestions

[0816] The server searches for suitable products from online shopping sites based on the user's answers.

[0817] The server transmits the search results to the terminal and suggests them to the user.

[0818] The terminal displays the suggested products for the user to view.

[0819] Step 3: Check product details

[0820] The user views detailed information and photos of the suggested products.

[0821] The terminal transmits information about the product selected by the user to the server.

[0822] Step 4: Providing checkout

[0823] The server provides a link to proceed with the purchase for the selected item.

[0824] The terminal displays the link provided by the server, allowing the user to complete the purchase procedure.

[0825] Step 5: Collect and analyze data

[0826] The server collects the user's past purchase history and reactions to products and stores them in a database.

[0827] The server analyzes the collected data to identify trends regarding users' preferences and needs.

[0828] As described above, a series of processes are carried out, from collecting the user's answers, searching for and suggesting products, confirming product details, providing purchasing procedures, and collecting and analyzing data. This allows the system to suggest products that are optimally tailored to the user's needs.

[0829] Example 1

[0830] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0831] Conventional online shopping systems have difficulty in accurately grasping a user's preferences and needs and proposing optimal products. In addition, there is a lack of a means to effectively use a user's purchase history and reactions to products to improve the accuracy of product suggestions, which makes it difficult to increase user satisfaction. Furthermore, there is a lack of links to help users smoothly proceed through the purchase process.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0833] In this invention, the server includes a means for understanding the preferences and needs of a user and proposing optimal products, a means for understanding the needs, age, and gender of the user through dialogue with the user, a means for providing details of a product the user wants to purchase and a suggestion of clothing to match a specific event, a means for analyzing user information using a generative AI model and searching for suitable products from an online database, a means for transmitting search results obtained by the generative AI model to the user's terminal and displaying them to the user, and a means for providing a link for proceeding with a product purchase procedure based on the user's selection. This makes it possible to propose products based on the user's needs, improves the accuracy of the proposal, and enables the user to smoothly complete the purchase procedure.

[0834] A "generative AI model" is an artificial intelligence algorithm that analyzes information provided by a user and searches for and suggests appropriate products.

[0835] The means of "understanding the user's preferences and needs" refers to collecting and analyzing specific information such as the user's needs, age, and gender from the information input and dialogue of the user.

[0836] "Means of understanding the user's needs, age, and gender through dialogue" is the process of grasping basic information with the user through interactive communication.

[0837] The means of "suggesting clothing according to details or specific events" is a process of collecting detailed information about products that a user desires according to a specific purpose or event, and then suggesting appropriate products based on the information.

[0838] "Means of acquiring past purchase history and reactions to products to improve the accuracy of product suggestions" refers to collecting and analyzing users' past purchase history and product reviews, etc., and using this as a basis to improve the accuracy of future product suggestions.

[0839] "Means for searching for suitable products from an online database" refers to the process of using collected user information to search for products that meet the user's requirements from an online database.

[0840] The "means for transmitting the search results to the user's terminal and displaying them" refers to a process in which the server processes the searched product information and transmits it to the user's device for display.

[0841] The "means for providing a link to proceed with the product purchase procedure" refers to the process of providing an interface such as a web link or button that allows the user to purchase the selected product.

[0842] The system of this invention has the function of understanding the preferences and needs of users and proposing optimal products. This system is mainly composed of three elements: a server, a terminal, and a user, and operates according to the following specific flow.

[0843] Hardware and Software Used

[0844] Server: Cloud services such as Amazon Web Services (AWS) and Google Cloud Platform (GCP)

[0845] Devices: Smartphones, tablets, personal computers (PCs), etc.

[0846] Programming: Programming languages ​​such as Python, JavaScript, Node.js, etc.

[0847] Online database: An online shopping site that sells products over the Internet.

[0848] Server Features

[0849] The server initiates a dialogue with the user and collects information such as the user's needs, age, gender, etc. It then uses a generative AI model to analyze this information and search an online database for products that match the user's preferences. The analysis uses natural language processing techniques and keyword extraction algorithms.

[0850] Device Features

[0851] The terminal provides an interface for interactive dialogue with the user. It transmits information input by the user to the server and displays product information received from the server to the user. This information includes product images, prices, detailed information, etc.

[0852] User Actions

[0853] The user interacts with the system through a terminal and inputs details of the desired product and information related to a specific event. For example, a specific prompt such as "I'm looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen" is used. Based on this information, the system suggests appropriate products.

[0854] Examples

[0855] 1. Example prompt:

[0856] The user inputs his / her preference using the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[0857] 2. Server analysis:

[0858] The server uses a generative AI model to analyze the user's preferences and extract the keyword "blue vintage dress."

[0859] Next, it uses the API of an online shopping site to search for a suitable dress.

[0860] 3. Submit search results:

[0861] The server sends the search result "Blue vintage dress (price: 28,000 yen)" in JSON format to the terminal.

[0862] The terminal receives this information and displays it to the user.

[0863] 4. User Choices and Purchase Procedures:

[0864] The user can view the proposed dresses through the terminal and select the one they like.

[0865] The terminal transmits information selected by the user to the server, and the server generates a link for proceeding with the purchase procedure and transmits it to the terminal.

[0866] The user clicks on this link to complete the purchase.

[0867] In this way, the server and the terminal work together to suggest optimal products based on the user's needs and to process the purchase.

[0868] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0869] Step 1:

[0870] Prompt input from the user

[0871] A user accesses the system through a terminal and inputs detailed information about the product he or she desires. Specifically, he or she inputs the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[0872] Input: The prompt text entered by the user.

[0873] Output: The terminal sends the prompt to the server.

[0874] Step 2:

[0875] Prompt by terminal

[0876] The terminal sends the prompt text entered by the user to the server using an HTTP POST request.

[0877] Input: The prompt text entered by the user

[0878] Output: The server receives the prompt.

[0879] Step 3:

[0880] Prompt parsing by the server

[0881] The server analyzes the received prompt using a generative AI model (e.g., natural language processing model), extracts specific keywords (e.g., "vintage," "blue," "under 30,000 yen"), and understands the user's needs.

[0882] Input: Prompt sent from the terminal

[0883] Data Processing: Text Analysis with Generative AI Models

[0884] Output: Extracted keywords and user needs information

[0885] Step 4:

[0886] Product search by server

[0887] The server searches for suitable products using an online database (e.g., an online shopping site API) based on the extracted keywords and the user's needs information. For example, a search for "blue vintage dresses" can be performed using an online shopping site API.

[0888] Input: Extracted keywords and user needs information

[0889] Data Computing: Sending queries to online databases and retrieving search results

[0890] Output: A list of searched products

[0891] Step 5:

[0892] Server sends search results

[0893] The server processes the list of products obtained in JSON format and sends it to the terminal. For example, the information may be formatted as "Blue vintage dress (price: 28,000 yen)."

[0894] Input: A list of products retrieved from an online database

[0895] Data processing: Formatting data into JSON format

[0896] Output: List of processed product information

[0897] Step 6:

[0898] Displaying search results by device

[0899] The terminal displays the product information received from the server to the user. The user can view product images, prices, and detailed information. For example, a photo of the product, its price, and detailed information are displayed.

[0900] Input: A list of product information sent from the server

[0901] Output: Product information displayed to the user

[0902] Step 7:

[0903] User selection of products

[0904] The user selects a product he likes from the displayed products, for example by clicking a "Buy this dress" button.

[0905] Input: User browses and selects products

[0906] Output: Selected product information

[0907] Step 8:

[0908] Sending selection information by the terminal

[0909] The terminal transmits the product information selected by the user to the server.

[0910] Input: Product information selected by the user

[0911] Output: Selections sent to the server

[0912] Step 9:

[0913] Server-generated checkout link

[0914] The server generates a checkout link based on the product selected by the user and sends it to the terminal. For example, it generates a link that says "Click here to proceed with the checkout."

[0915] Input: Product information selected by the user

[0916] Data processing: generating checkout link

[0917] Output: Checkout link

[0918] Step 10:

[0919] Displaying links by terminal

[0920] The terminal displays the purchase procedure link received from the server to the user, who can click on this link to proceed with the purchase.

[0921] Input: Checkout link sent by the server

[0922] Output: The checkout link shown to the user.

[0923] In this way, the server and the terminal work together to suggest products that meet the user's needs and facilitate a smooth purchasing process.

[0924] (Application example 1)

[0925] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0926] Conventional online shopping systems have limitations in their ability to suggest optimal products based on users' preferences and needs, so customers often have difficulty finding the products they actually want. Even in brick-and-mortar stores, it is difficult for store clerks to instantly suggest appropriate products to each individual customer, which can lead to a poor customer experience.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0928] In this invention, the server includes a means for understanding the needs, age, and sex of a user through dialogue with the user, a means for providing details of a product the user wants to purchase and clothing suggestions for a specific event, a means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions, and a means for suggesting products to customers in the store in real time through smart glasses and displaying them on a display. This makes it possible to quickly and accurately suggest products that customers want and improve the shopping experience in a physical store.

[0929] "User" refers to an individual who uses the System to search for and purchase Products.

[0930] "Preferences" refer to personal tastes and preferences that a user has for particular products or services.

[0931] "Needs" refers to the requirements or desires that a user has in a specific situation or for a specific purpose.

[0932] "Product" refers to a specific item or service offered to a user.

[0933] "Server" refers to a central computer that handles user interaction and processes information.

[0934] "Dialogue" refers to the exchange of information or communication between a user and a server or terminal.

[0935] "Purchase history" refers to a record of products and services a user has previously purchased.

[0936] "Response" refers to the evaluation or feedback a user gives to a proposed product or service.

[0937] "Smart glasses" refers to a wearable device that has a built-in display that displays information and can interact with the user.

[0938] "Real-time" means that information or data is provided immediately.

[0939] "Display" refers to a display device for visually presenting information to a user.

[0940] The system that realizes this application example mainly consists of three elements: a server, a terminal wearing smart glasses, and a user.

[0941] The server plays a key role in interacting with users, collecting and analyzing their information, and proposing appropriate products. Specifically, the server uses the following hardware and software:

[0942] Central computer (server): This is the heart of the system and processes and stores data.

[0943] Database Management System (DBMS): Used to store and manage users' purchase history and past responses.

[0944] AI model (generative AI model): An artificial intelligence system that analyzes user requirements and makes product suggestions.

[0945] The server first interacts with the user to collect information such as their needs, age, and gender, through voice recognition technology and text input, and then analyzes the collected data to provide the user with details about the products they need and suggestions tailored to specific events.

[0946] The smart glasses used as a terminal act as an intermediary between the user and the server. The smart glasses use the following hardware and software:

[0947] Smart Glasses Display: Displaying information to the user in real time.

[0948] Speech recognition functionality: Captures customer requests as voice input.

[0949] Communication module: Used to send and receive data to and from the server.

[0950] As a specific example of its use, a user puts on the smart glasses in a store and issues a voice command such as, "I'm looking for an elegant wedding dress for under 50,000 yen." This command is sent to the server, which searches for suitable products in an online database and displays the results on the smart glasses' display.

[0951] Users can view detailed information and photos of the suggested products through the smart glasses. This allows users to compare multiple products without moving around the store. If they find a product they like, they can proceed with the purchase on the spot.

[0952] Specific examples of prompts for this system are as follows:

[0953] "I am looking for an elegant dress under 50000 for a wedding. Please let me know my options."

[0954] This will dramatically improve the customer experience in physical stores and contribute to increasing customer satisfaction.

[0955] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0956] Step 1:

[0957] The user makes a request through the smart glasses by voice input or text input. For example, the user may say, "I'm looking for an elegant dress for a wedding under 50,000 yen." The smart glasses convert this request into text data using voice recognition and send the text data to the server.

[0958] Input: User's voice command

[0959] Output: Instruction data in text format

[0960] Step 2:

[0961] The server analyzes the received text data and performs natural language processing to identify the user's requirements (needs, preferences, budget, etc.) The analysis results are sent to a database as a specific search query.

[0962] Input: Instruction data in text format

[0963] Output: The search query

[0964] Step 3:

[0965] The server searches an online database for suitable products based on the search query, taking into account the user's past purchase history and reactions, and uses a generative AI model to list the most suitable products.

[0966] Input: Search query, user's past purchase history, response data

[0967] Output: Product list

[0968] Step 4:

[0969] The server converts the search results into a data format suitable for display on the smart glasses display and sends it to the device (smart glasses). This data includes detailed product information and photos.

[0970] Input: Product list

[0971] Output: Product details, photos

[0972] Step 5:

[0973] The terminal (smart glasses) visually displays the data received from the server, and the user can view detailed information and photos of the proposed products through the smart glasses display.

[0974] Input: Product details, photo

[0975] Output: Information displayed on the smart glasses display

[0976] Step 6:

[0977] The user selects the product they like from the suggested products, and the selection result is sent to the server via the smart glasses.

[0978] Input: User's choice

[0979] Output: Selected product data

[0980] Step 7:

[0981] The server generates a link and information for the selected product to proceed with the purchase procedure and sends it to the device (smart glasses), allowing the user to complete the purchase procedure on the spot.

[0982] Input: Selected product data

[0983] Output: Checkout information

[0984] In this way, users can receive product suggestions in real time through the smart glasses and proceed with the purchase process for products that interest them, providing a convenient and efficient shopping experience.

[0985] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0986] An embodiment for implementing the present invention includes the following elements.

[0987] 1. Emotion Engine:

[0988] To provide a means for recognizing emotions from a user's speech, facial expressions, etc.

[0989] Provide a means for transmitting recognized emotion information to a server.

[0990] To provide a means for adjusting product suggestions and dialogue progress based on a user's emotions.

[0991] 2. Server:

[0992] It provides an interface for interacting with the user.

[0993] Receive and process user responses and emotional information.

[0994] Adjust product suggestions and conversation progression based on emotional information.

[0995] 3. Terminal:

[0996] It provides an interface for interacting with the user.

[0997] It is responsible for communicating with the server and sending the user's answers and emotion information to the server.

[0998] Receives suggestions and information from the server and displays them to the user.

[0999] 4. Program:

[1000] Controls the transfer and processing of data between the server and the terminal.

[1001] User responses and emotional information are analyzed to search for and suggest appropriate products.

[1002] The progress of the dialogue is adjusted based on emotional information, and appropriate services and support are provided to the user.

[1003] (Specific examples)

[1004] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[1005] 1. The device recognizes emotions from the user's speech, facial expressions, etc. and transmits them to the emotion engine.

[1006] 2. The emotion engine analyzes the user's emotion information and sends the information to the server.

[1007] 3. The server receives the user's emotional information and adjusts product suggestions and the progress of the dialogue.

[1008] 4. The server searches and suggests suitable dresses from online shopping sites based on the user's answers and emotional information.

[1009] 5. The device displays detailed information and photos of the suggested dresses to the user.

[1010] 6. The user selects the suggested dress and proceeds with the purchase.

[1011] 7. The server provides a checkout link for the selected dress.

[1012] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[1013] The process flow will be explained below.

[1014] Step 1: Recognizing emotions and sending information

[1015] The terminal recognizes the user's emotions from their speech and facial expressions.

[1016] The terminal transmits the recognized emotion information to the emotion engine.

[1017] The emotion engine analyzes the received emotion information and transmits it to the server.

[1018] Step 2: Make suggestions and tailor dialogue based on emotions

[1019] The server receives the emotion information received from the emotion engine.

[1020] The server analyzes the emotion information and adjusts the progress of the dialogue and suggests products appropriate to the user's emotions.

[1021] The server searches and suggests suitable products from online shopping sites.

[1022] Step 3: View suggested products

[1023] The terminal receives detailed information and photos of the suggested products from the server.

[1024] The terminal displays the suggested items to the user.

[1025] The user browses the suggested products and makes a selection.

[1026] Step 4: Proceed with your purchase

[1027] The terminal receives a link from the server to proceed with the purchase procedure for the product selected by the user.

[1028] The terminal displays a link to the user, allowing them to complete the purchase procedure.

[1029] Step 5: Collect and analyze sentiment information

[1030] The server collects the user's emotion information and stores it in a database.

[1031] The server analyzes the collected emotion information to understand the user's emotional tendencies and preferences.

[1032] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[1033] Example 2

[1034] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1035] Conventional online shopping systems suggest products without considering the user's emotions or mood at the time, making it difficult to suggest optimal products to users. In addition, since dialogue progression and product suggestions are not based on the user's emotions, there is a problem of a poor user experience. Furthermore, there is a lack of personalized suggestions to increase the user's motivation to purchase. For this reason, it is necessary to realize optimal product suggestions and dialogue progression that take into account the user's emotions.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1037] In this invention, the server includes a means for analyzing the user's speech and facial expressions to recognize emotions, a means for transmitting the recognized emotion information to the server, and a means for adjusting the content of product proposals and the progress of the dialogue based on the user's emotions, thereby enabling optimal product proposals and dialogue progress that take the user's emotions into account.

[1038] "User" refers to an individual who uses the System to search for, select, and purchase Products.

[1039] "Dialogue" refers to interaction between a user and a system with the purpose of exchanging information.

[1040] "Needs" refers to the requests and desires for products and services that users require.

[1041] "Age" refers to the number of years calculated from the user's date of birth.

[1042] "Gender" refers to the sex of the user, such as male or female.

[1043] "Utterance" refers to the act of a user providing information to a system via voice.

[1044] "Facial expressions" refer to the movements and expressions of a user's face and are used to recognize emotions.

[1045] "Emotion" refers to the psychological state a user exhibits in response to a particular situation or stimulus.

[1046] An "emotion engine" refers to software that analyzes emotions from a user's speech and facial expressions.

[1047] "Server" refers to the central system that processes user information, makes product suggestions, and advances dialogue.

[1048] "Online shopping site" refers to a website that sells products over the Internet.

[1049] "Purchase history" refers to a record of products and services a user has previously purchased.

[1050] "Product recommendation accuracy" refers to the ability to appropriately recommend products based on a user's needs and emotions.

[1051] "Search conditions" refer to product search parameters that are set based on a user's needs and feelings.

[1052] "Purchase procedure" refers to the process by which a user officially purchases a product selected by the user.

[1053] A specific embodiment of the present invention is an online shopping system in which a user utilizes a terminal to receive product suggestions and progress in dialogue via a server. The system aims to improve the user experience by recognizing the user's emotions and making optimal product suggestions based on those emotions. A specific method for implementing the present invention will be described in detail below.

[1054] Emotion Engine

[1055] A user connects to the system using a device (such as a smartphone or tablet). The device has a built-in camera and microphone. The device collects the user's speech and facial expressions in real time via the camera and microphone, and transmits them to the emotion engine. The emotion engine uses voice recognition software and facial expression recognition software to analyze emotions from the user's speech and facial expressions.

[1056] The emotion engine generates the analysis results as numbers or categories (e.g., "happiness," "sadness," "surprise," etc.) and sends this information to a server. The software used by the emotion engine is a cloud-based API that enables real-time emotion analysis.

[1057] Server Features

[1058] The server receives the emotion information sent from the emotion engine. Based on this information, the server adjusts the dialogue scenario with the user and the product suggestions. For example, if the user expresses the emotion of "joy," the server will proceed with the dialogue with content optimized for the user's emotion, such as suggesting a dress with a gorgeous design and bright colors.

[1059] The server uses the user's speech and emotional information to search for suitable products on the online shopping site. It then selects the most suitable product from the search results and sends it to the terminal. This allows the server to provide the best product suggestions for the user.

[1060] Device Features

[1061] The terminal provides an interface that displays the product suggestions received from the server and the progress of the dialogue to the user. The user can view detailed information and photos of the suggested products on the terminal, select a product as necessary, and proceed with the purchase procedure.

[1062] As a concrete example, consider the case where a user is looking for a dress to attend a wedding. The user connects to the system through a terminal and says, "I'm looking for a dress for a wedding." The user's actions are captured by the camera and recorded by the microphone.

[1063] The device sends this information to the emotion engine, which detects emotions such as "happiness" and "anticipation." The server searches for suitable dresses from online shopping sites based on this emotion information and suggests them to the user. The user checks the details and photos of the suggested dresses on the device, selects the one they like, and proceeds with the purchase process.

[1064] Examples of prompt statements

[1065] For example, the following is an example of a prompt that a user may enter into the system:

[1066] I'm looking for a recommended dress to wear to a wedding.

[1067] "Please suggest a black formal dress that would please our users."

[1068] In this way, the entire system works together to recognize the user's emotions and make optimal product suggestions.

[1069] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1070] Step 1:

[1071] A user logs into the system through a terminal and inputs a request to search for a wedding dress.

[1072] Specifically, the user enters their user ID and password into the login screen of the terminal. Then, they enter "I'm looking for a wedding dress" into the search bar and click the search button. This sends a search request to the system.

[1073] Input: User login information and search query

[1074] Output: A search request to the server.

[1075] Step 2:

[1076] The device collects the user's speech and facial expressions using a camera and microphone and sends them to the emotion engine.

[1077] Specifically, the device captures the user's facial expressions with a built-in camera and records audio with a microphone, and these data are sent to the emotion engine in real time.

[1078] Input: User's speech and facial expression data

[1079] Output: Sending data to the emotion engine

[1080] Step 3:

[1081] The emotion engine analyzes the user's emotions and sends the information to the server.

[1082] Specifically, the emotion engine converts voice data into text using an API that converts voice data into text, and analyzes facial expression data using an API that analyzes facial expression data. This allows emotion categories such as "happiness" and "expectation" to be identified. The analysis results are sent to the server as emotion information.

[1083] Input: Text data of utterances and facial expression feature data

[1084] Output: Emotion information

[1085] Step 4:

[1086] The server adjusts product search conditions based on the emotion information received.

[1087] Specifically, the server receives the emotion information and sets the characteristics of products that have attributes that the user finds pleasing (e.g., gorgeousness or color) as search conditions. It also takes into account the user's past purchase history and preferences to personalize the search conditions.

[1088] Input: Emotional information and past purchase history data

[1089] Output: Optimized search criteria

[1090] Step 5:

[1091] The server searches for suitable dresses on an online shopping site and transmits the results to the terminal.

[1092] Specifically, the server uses the API of the online shopping site to perform a product search based on the search criteria set by the user, selects the product that best matches the emotion information from the search results, and sends its details to the device.

[1093] Input: Optimized search criteria

[1094] Output:Details of selected product

[1095] Step 6:

[1096] The device will display detailed information and photos of the suggested dresses to the user.

[1097] Specifically, the device displays the dress information received from the server on a GUI, allowing the user to swipe through photos of the dress or tap on detailed information to view it.

[1098] Input:Details of selected product

[1099] Output: Information displayed to the user

[1100] Step 7:

[1101] The user selects a suggested dress and proceeds with the purchase.

[1102] Specifically, the user selects a dress from the suggestions and clicks the "Purchase" button. The terminal displays a checkout screen, allowing the user to enter credit card information and a shipping address. The user then confirms the purchase and clicks the "Confirm Purchase" button.

[1103] Input: User selection and purchase procedure information

[1104] Output: Purchase confirmation notice

[1105] This allows the entire system to work together to provide optimal product suggestions and purchasing procedures that take the user's emotions into account.

[1106] (Application example 2)

[1107] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1108] Conventional online shopping systems have difficulty in accurately understanding a user's preferences and needs and suggesting appropriate products. In addition, since product recommendations cannot be made taking into account the user's emotions, user satisfaction may decrease. Furthermore, in dialogue-based interactions, the progress of the dialogue cannot be adjusted according to the user's emotions, which may result in a poor user experience.

[1109] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for recognizing emotions from the user's speech and facial expressions, a means for transmitting to the server based on emotion information and making appropriate product recommendations, a means for understanding the needs, age, and gender through dialogue with the user, a means for providing details of a product the user wants to purchase and outfit suggestions suited to a specific event, a means for acquiring the user's past purchase history and reactions to products and improving the accuracy of product suggestions, and a means for adjusting the progress of the dialogue based on emotion information. This makes it possible to realize appropriate product suggestions and optimal interactions based on the user's emotions, improving the user experience.

[1110] "Means for recognizing emotions from a user's speech and facial expressions" refers to a method for identifying emotions by analyzing a user's facial expressions and voice using sensors such as a camera and microphone.

[1111] The "means for transmitting to a server based on emotion information and making appropriate product recommendations" is a method for transmitting recognized emotion data to a server and proposing optimal products based on that information.

[1112] "Means for understanding the needs, age, and gender of a user through dialogue" refers to a method for using a chatbot or voice assistant to interact with a user and collect information about the user's needs, age, and gender.

[1113] "Means for providing details of products that a user wishes to purchase and suggestions of appropriate attire for a specific event" refers to a method for providing detailed information of products that a user wishes to purchase and suggestions of appropriate attire for a specific event or situation.

[1114] "Means of acquiring a user's past purchase history and reactions to products to improve the accuracy of product suggestions" refers to a method of collecting a user's history of past purchases and the user's impressions and feedback about those products, and using that information to improve the accuracy of product recommendations.

[1115] The "means for adjusting the progress of a dialogue based on emotional information" is a method for optimizing the speed and content of a dialogue based on the user's current emotional state.

[1116] The embodiment of the present invention is a smart shopping assistant system that recommends products based on the user's emotions. This system analyzes the user's facial expressions and speech when the user uses a smartphone to shop online, and the server suggests appropriate products based on the emotional information.

[1117] First, the user's device is equipped with a camera and a microphone, and these sensors are used to capture the user's facial expressions and voice. The specific hardware used here is a smartphone's front camera and built-in microphone. The captured data is processed using OpenCV or a similar image processing library to recognize emotions from facial expressions.

[1118] A pre-trained emotion recognition model (e.g., a deep learning model built with Keras) is used to recognize user emotions. This model analyzes input image data and identifies the user's emotional state (e.g., joy, sadness, anger, etc.).

[1119] The recognized emotion information is then sent to a server via the Internet. The server receives this emotion information and combines it with the user's past purchase history and reaction data to search for and recommend the most suitable products. The server also takes into account information registered by the user, such as needs, age, and gender, to improve the accuracy of product recommendations.

[1120] In addition, the script in charge of the dialogue with the user is executed by the server, and the dialogue progress is adjusted based on the emotion information. For example, if the user is interested but has a slightly dissatisfied expression, the dialogue script is adjusted to carefully explain the product's features and benefits. On the other hand, if the user has a happy expression, a more relaxed dialogue style is selected.

[1121] A distinctive feature of this system is the adjustment of the dialogue progress based on emotional information, which allows users to experience product recommendations and interfaces optimized for their own emotional state, improving the overall satisfaction of the shopping experience.

[1122] As a concrete example, consider a case where a user is looking for a dress to attend a wedding. When the user speaks into the smartphone camera, the facial expression and speech are analyzed by the emotion engine. The analyzed emotion information is then sent to the server, which suggests dresses that match the user's needs. This process makes it easier for the user to find a dress that fits their emotions.

[1123] Example of a prompt to be input to a generative AI model:

[1124] Create a smartphone application for an online shopping site that provides a product recommendation service based on the user's emotions. The application will recognize emotions from the user's facial expressions and voice, and suggest the most suitable products based on those emotions.

[1125] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1126] Step 1:

[1127] A user starts a smartphone app and turns on the camera and microphone. The user starts talking to the screen or searching for information about a product. The input data here is the user's facial image and voice, which are captured by the camera and microphone. Facial expression data and voice data are acquired through the user's operations.

[1128] Step 2:

[1129] The device processes the captured video and audio in real time to perform face and voice recognition. Specifically, it uses the OpenCV library to detect faces and crop specific facial regions. The cropped facial images are input to a pre-trained emotion recognition model (a deep learning model using Keras). Here, the input data is the image of the facial region, and the output is emotion information (e.g., anger, joy, sadness, etc.).

[1130] Step 3:

[1131] The device sends the recognized emotion information and voice data to the server, where the device formats the data and sends it to the server using an HTTP request. The input is the payload of emotion information and voice data, which is sent to the server. The server receives the request and starts to analyze the emotion information and voice data.

[1132] Step 4:

[1133] The server understands the user's needs based on the received emotion information and voice data, and searches for appropriate products by referring to past purchase history and user feedback. In this step, the server executes a database query to search the user profile and filters products based on the obtained emotion information. The inputs are emotion information, voice data, and user profile information from the database. The output is a list of recommended products.

[1134] Step 5:

[1135] The server sends the recommended product list to the terminal. Here, the product list generated by the server is converted to JSON format or similar and sent to the terminal as an HTTP response. The input is the recommended product list, and the output is the response sent to the terminal.

[1136] Step 6:

[1137] The terminal displays the received product list to the user. The user can check the product details on the screen and select the items he likes. Here, the input is the product list received from the server, and the output is the product information displayed to the user. The terminal updates the user interface and allows the user to select products.

[1138] Step 7:

[1139] When the user selects a product, the terminal sends the selection information to the server. The server then proceeds with the purchase procedure based on the received information. Specifically, it securely processes credit card information and links with the ordering system. The input here is the user's selection information, and the output is a notification that the purchase procedure has been completed.

[1140] Through the above steps, optimal product recommendations based on emotion information and smooth interaction with the user are realized.

[1141] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1142] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1143] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1144] [Fourth embodiment]

[1145] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1146] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1147] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1148] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1149] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1150] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1151] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1152] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1153] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1154] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1155] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1156] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1157] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1158] An embodiment for implementing the present invention includes the following elements.

[1159] 1. Server:

[1160] It provides an interface for interacting with the user.

[1161] Collect and process user responses and requests.

[1162] Search for and suggest appropriate products from online shopping sites, etc.

[1163] 2. Terminal:

[1164] It provides an interface for interacting with the user.

[1165] It is responsible for communicating with the server and sending the user's responses and requests to the server.

[1166] Receives suggestions and information from the server and displays them to the user.

[1167] 3. User:

[1168] It interacts with servers and terminals through an interface.

[1169] Provide details about a product you want to purchase or information related to a particular event.

[1170] Browse and select suggested products.

[1171] 4. Program:

[1172] Controls the transfer and processing of data between the server and the terminal.

[1173] Analyze the user's responses and requests to search for and suggest appropriate products.

[1174] Acquire and analyze data from online shopping sites, etc.

[1175] (Specific examples)

[1176] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[1177] 1. The server initiates a dialogue with the user via the terminal.

[1178] 2. The user provides information such as wedding theme, dress preferences, budget, etc.

[1179] 3. The server analyzes the user's information and searches for suitable dresses from online shopping sites.

[1180] 4. The server sends the search results to the terminal and suggests them to the user.

[1181] 5. The user browses the details and photos of the proposed dresses and selects the one they like. 6. The server provides a link to the selected dress to proceed with the purchase process.

[1182] As described above, by transferring and processing data between the server and the terminal, it is possible to realize proposals of products that are optimally tailored to the user's needs.

[1183] The process flow will be explained below.

[1184] Step 1: Collect user responses

[1185] The server initiates a dialogue with the user via the terminal.

[1186] The terminal receives the answer from the user and transmits it to the server.

[1187] The server receives and analyzes the user's answers.

[1188] Step 2: Product search and suggestions

[1189] The server searches for suitable products from online shopping sites based on the user's answers.

[1190] The server transmits the search results to the terminal and suggests them to the user.

[1191] The terminal displays the suggested products for the user to view.

[1192] Step 3: Check product details

[1193] The user views detailed information and photos of the suggested products.

[1194] The terminal transmits information about the product selected by the user to the server.

[1195] Step 4: Providing checkout

[1196] The server provides a link to proceed with the purchase for the selected item.

[1197] The terminal displays the link provided by the server, allowing the user to complete the purchase procedure.

[1198] Step 5: Collect and analyze data

[1199] The server collects the user's past purchase history and reactions to products and stores them in a database.

[1200] The server analyzes the collected data to identify trends regarding users' preferences and needs.

[1201] As described above, a series of processes are carried out, from collecting the user's answers, searching for and suggesting products, confirming product details, providing purchasing procedures, and collecting and analyzing data. This allows the system to suggest products that are optimally tailored to the user's needs.

[1202] Example 1

[1203] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[1204] Conventional online shopping systems have difficulty in accurately grasping a user's preferences and needs and proposing optimal products. In addition, there is a lack of a means to effectively use a user's purchase history and reactions to products to improve the accuracy of product suggestions, which makes it difficult to increase user satisfaction. Furthermore, there is a lack of links to help users smoothly proceed through the purchase process.

[1205] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1206] In this invention, the server includes a means for understanding the preferences and needs of a user and proposing optimal products, a means for understanding the needs, age, and gender of the user through dialogue with the user, a means for providing details of a product the user wants to purchase and a suggestion of clothing to match a specific event, a means for analyzing user information using a generative AI model and searching for suitable products from an online database, a means for transmitting search results obtained by the generative AI model to the user's terminal and displaying them to the user, and a means for providing a link for proceeding with a product purchase procedure based on the user's selection. This makes it possible to propose products based on the user's needs, improves the accuracy of the proposal, and enables the user to smoothly complete the purchase procedure.

[1207] A "generative AI model" is an artificial intelligence algorithm that analyzes information provided by a user and searches for and suggests appropriate products.

[1208] The means of "understanding the user's preferences and needs" refers to collecting and analyzing specific information such as the user's needs, age, and gender from the information input and dialogue of the user.

[1209] "Means of understanding the user's needs, age, and gender through dialogue" is the process of grasping basic information with the user through interactive communication.

[1210] The means of "suggesting clothing according to details or specific events" is a process of collecting detailed information about products that a user desires according to a specific purpose or event, and then suggesting appropriate products based on the information.

[1211] "Means of acquiring past purchase history and reactions to products to improve the accuracy of product suggestions" refers to collecting and analyzing users' past purchase history and product reviews, etc., and using this as a basis to improve the accuracy of future product suggestions.

[1212] "Means for searching for suitable products from an online database" refers to the process of using collected user information to search for products that meet the user's requirements from an online database.

[1213] The "means for transmitting the search results to the user's terminal and displaying them" refers to a process in which the server processes the searched product information and transmits it to the user's device for display.

[1214] The "means for providing a link to proceed with the product purchase procedure" refers to the process of providing an interface such as a web link or button that allows the user to purchase the selected product.

[1215] The system of this invention has the function of understanding the preferences and needs of users and proposing optimal products. This system is mainly composed of three elements: a server, a terminal, and a user, and operates according to the following specific flow.

[1216] Hardware and Software Used

[1217] Server: Cloud services such as Amazon Web Services (AWS) and Google Cloud Platform (GCP)

[1218] Devices: Smartphones, tablets, personal computers (PCs), etc.

[1219] Programming: Programming languages ​​such as Python, JavaScript, Node.js, etc.

[1220] Online database: An online shopping site that sells products over the Internet.

[1221] Server Features

[1222] The server initiates a dialogue with the user and collects information such as the user's needs, age, gender, etc. It then uses a generative AI model to analyze this information and search an online database for products that match the user's preferences. The analysis uses natural language processing techniques and keyword extraction algorithms.

[1223] Device Features

[1224] The terminal provides an interface for interactive dialogue with the user. It transmits information input by the user to the server and displays product information received from the server to the user. This information includes product images, prices, detailed information, etc.

[1225] User Actions

[1226] The user interacts with the system through a terminal and inputs details of the desired product and information related to a specific event. For example, a specific prompt such as "I'm looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen" is used. Based on this information, the system suggests appropriate products.

[1227] Examples

[1228] 1. Example prompt:

[1229] The user inputs his / her preference using the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[1230] 2. Server analysis:

[1231] The server uses a generative AI model to analyze the user's preferences and extract the keyword "blue vintage dress."

[1232] Next, it uses the API of an online shopping site to search for a suitable dress.

[1233] 3. Submit search results:

[1234] The server sends the search result "Blue vintage dress (price: 28,000 yen)" in JSON format to the terminal.

[1235] The terminal receives this information and displays it to the user.

[1236] 4. User Choices and Purchase Procedures:

[1237] The user can view the proposed dresses through the terminal and select the one they like.

[1238] The terminal transmits information selected by the user to the server, and the server generates a link for proceeding with the purchase procedure and transmits it to the terminal.

[1239] The user clicks on this link to complete the purchase.

[1240] In this way, the server and the terminal work together to suggest optimal products based on the user's needs and to process the purchase.

[1241] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1242] Step 1:

[1243] Prompt input from the user

[1244] A user accesses the system through a terminal and inputs detailed information about the product he or she desires. Specifically, he or she inputs the prompt sentence, "I am looking for a dress to wear to a wedding. The theme is vintage, the color is blue, and the budget is within 30,000 yen."

[1245] Input: The prompt text entered by the user.

[1246] Output: The terminal sends the prompt to the server.

[1247] Step 2:

[1248] Prompt by terminal

[1249] The terminal sends the prompt text entered by the user to the server using an HTTP POST request.

[1250] Input: The prompt text entered by the user

[1251] Output: The server receives the prompt.

[1252] Step 3:

[1253] Prompt parsing by the server

[1254] The server analyzes the received prompt using a generative AI model (e.g., natural language processing model), extracts specific keywords (e.g., "vintage," "blue," "under 30,000 yen"), and understands the user's needs.

[1255] Input: Prompt sent from the terminal

[1256] Data Processing: Text Analysis with Generative AI Models

[1257] Output: Extracted keywords and user needs information

[1258] Step 4:

[1259] Product search by server

[1260] The server searches for suitable products using online databases (e.g., online shopping site APIs) based on the extracted keywords and user needs information. For example, searching for "blue vintage dresses" using the Amazon API.

[1261] Input: Extracted keywords and user needs information

[1262] Data Computing: Sending queries to online databases and retrieving search results

[1263] Output: A list of searched products

[1264] Step 5:

[1265] Server sends search results

[1266] The server processes the list of products obtained in JSON format and sends it to the terminal. For example, the information may be formatted as "Blue vintage dress (price: 28,000 yen)."

[1267] Input: A list of products retrieved from an online database

[1268] Data processing: Formatting data into JSON format

[1269] Output: List of processed product information

[1270] Step 6:

[1271] Displaying search results by device

[1272] The terminal displays the product information received from the server to the user. The user can view product images, prices, and detailed information. For example, a photo of the product, its price, and detailed information are displayed.

[1273] Input: A list of product information sent from the server

[1274] Output: Product information displayed to the user

[1275] Step 7:

[1276] User selection of products

[1277] The user selects a product he likes from the displayed products, for example by clicking a "Buy this dress" button.

[1278] Input: User browses and selects products

[1279] Output: Selected product information

[1280] Step 8:

[1281] Sending selection information by the terminal

[1282] The terminal transmits the product information selected by the user to the server.

[1283] Input: Product information selected by the user

[1284] Output: Selections sent to the server

[1285] Step 9:

[1286] Server-generated checkout link

[1287] The server generates a checkout link based on the product selected by the user and sends it to the terminal. For example, it generates a link that says "Click here to proceed with the checkout."

[1288] Input: Product information selected by the user

[1289] Data processing: generating checkout link

[1290] Output: Checkout link

[1291] Step 10:

[1292] Displaying links by terminal

[1293] The terminal displays the purchase procedure link received from the server to the user, who can click on this link to proceed with the purchase.

[1294] Input: Checkout link sent by the server

[1295] Output: The checkout link shown to the user.

[1296] In this way, the server and the terminal work together to suggest products that meet the user's needs and facilitate a smooth purchasing process.

[1297] (Application example 1)

[1298] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1299] Conventional online shopping systems have limitations in their ability to suggest optimal products based on users' preferences and needs, so customers often have difficulty finding the products they actually want. Even in brick-and-mortar stores, it is difficult for store clerks to instantly suggest appropriate products to each individual customer, which can lead to a poor customer experience.

[1300] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1301] In this invention, the server includes a means for understanding the needs, age, and sex of a user through dialogue with the user, a means for providing details of a product the user wants to purchase and clothing suggestions for a specific event, a means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions, and a means for suggesting products to customers in the store in real time through smart glasses and displaying them on a display. This makes it possible to quickly and accurately suggest products that customers want and improve the shopping experience in a physical store.

[1302] "User" refers to an individual who uses the System to search for and purchase Products.

[1303] "Preferences" refer to personal tastes and preferences that a user has for particular products or services.

[1304] "Needs" refers to the requirements or desires that a user has in a specific situation or for a specific purpose.

[1305] "Product" refers to a specific item or service offered to a user.

[1306] "Server" refers to a central computer that handles user interaction and processes information.

[1307] "Dialogue" refers to the exchange of information or communication between a user and a server or terminal.

[1308] "Purchase history" refers to a record of products and services a user has previously purchased.

[1309] "Response" refers to the evaluation or feedback a user gives to a proposed product or service.

[1310] "Smart glasses" refers to a wearable device that has a built-in display that displays information and can interact with the user.

[1311] "Real-time" means that information or data is provided immediately.

[1312] "Display" refers to a display device for visually presenting information to a user.

[1313] The system that realizes this application example mainly consists of three elements: a server, a terminal wearing smart glasses, and a user.

[1314] The server plays a key role in interacting with users, collecting and analyzing their information, and proposing appropriate products. Specifically, the server uses the following hardware and software:

[1315] Central computer (server): This is the heart of the system and processes and stores data.

[1316] Database Management System (DBMS): Used to store and manage users' purchase history and past responses.

[1317] AI model (generative AI model): An artificial intelligence system that analyzes user requirements and makes product suggestions.

[1318] The server first interacts with the user to gather information about the user's needs, age, gender, etc. This interaction is carried out through voice recognition technology and text input, and then analyzes the collected data to provide the user with details about the products they need and suggestions tailored to specific events.

[1319] The smart glasses used as a terminal act as an intermediary between the user and the server. The smart glasses use the following hardware and software:

[1320] Smart Glasses Display: Displaying information to the user in real time.

[1321] Speech recognition functionality: Captures customer requests as voice input.

[1322] Communication module: Used to send and receive data to and from the server.

[1323] As a specific example of its use, a user puts on the smart glasses in a store and issues a voice command such as, "I'm looking for an elegant wedding dress for under 50,000 yen." This command is sent to the server, which searches for suitable products in an online database and displays the results on the smart glasses' display.

[1324] Users can view detailed information and photos of the suggested products through the smart glasses. This allows users to compare multiple products without moving around the store. If they find a product they like, they can proceed with the purchase on the spot.

[1325] Specific examples of prompts for this system are as follows:

[1326] "I am looking for an elegant dress under 50000 for a wedding. Please let me know my options."

[1327] This will dramatically improve the customer experience in physical stores and contribute to increasing customer satisfaction.

[1328] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1329] Step 1:

[1330] The user makes a request through the smart glasses by voice input or text input. For example, the user may say, "I'm looking for an elegant dress for a wedding under 50,000 yen." The smart glasses convert this request into text data using voice recognition and send the text data to the server.

[1331] Input: User's voice command

[1332] Output: Instruction data in text format

[1333] Step 2:

[1334] The server analyzes the received text data and performs natural language processing to identify the user's requirements (needs, preferences, budget, etc.) The analysis results are sent to a database as a specific search query.

[1335] Input: Instruction data in text format

[1336] Output: The search query

[1337] Step 3:

[1338] The server searches an online database for suitable products based on the search query, taking into account the user's past purchase history and reactions, and uses a generative AI model to list the most suitable products.

[1339] Input: Search query, user's past purchase history, response data

[1340] Output: Product list

[1341] Step 4:

[1342] The server converts the search results into a data format suitable for display on the smart glasses display and sends it to the device (smart glasses). This data includes detailed product information and photos.

[1343] Input: Product list

[1344] Output: Product details, photos

[1345] Step 5:

[1346] The terminal (smart glasses) visually displays the data received from the server, and the user can view detailed information and photos of the proposed products through the smart glasses display.

[1347] Input: Product details, photo

[1348] Output: Information displayed on the smart glasses display

[1349] Step 6:

[1350] The user selects the product they like from the suggested products, and the selection result is sent to the server via the smart glasses.

[1351] Input: User's choice

[1352] Output: Selected product data

[1353] Step 7:

[1354] The server generates a link and information for the selected product to proceed with the purchase procedure and sends it to the device (smart glasses), allowing the user to complete the purchase procedure on the spot.

[1355] Input: Selected product data

[1356] Output: Checkout information

[1357] In this way, users can receive product suggestions in real time through the smart glasses and proceed with the purchase process for products that interest them, providing a convenient and efficient shopping experience.

[1358] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1359] An embodiment for implementing the present invention includes the following elements.

[1360] 1. Emotion Engine:

[1361] To provide a means for recognizing emotions from a user's speech, facial expressions, etc.

[1362] Provide a means for transmitting recognized emotion information to a server.

[1363] To provide a means for adjusting product suggestions and dialogue progress based on a user's emotions.

[1364] 2. Server:

[1365] It provides an interface for interacting with the user.

[1366] Receive and process user responses and emotional information.

[1367] Adjust product suggestions and conversation progression based on emotional information.

[1368] 3. Terminal:

[1369] It provides an interface for interacting with the user.

[1370] It is responsible for communicating with the server and sending the user's answers and emotion information to the server.

[1371] Receives suggestions and information from the server and displays them to the user.

[1372] 4. Program:

[1373] Controls the transfer and processing of data between the server and the terminal.

[1374] User responses and emotional information are analyzed to search for and suggest appropriate products.

[1375] The progress of the dialogue is adjusted based on emotional information, and appropriate services and support are provided to the user.

[1376] (Specific examples)

[1377] For example, when a user is searching for a dress to wear to a wedding ceremony through a terminal connected to the server, the following program processing is carried out.

[1378] 1. The device recognizes emotions from the user's speech, facial expressions, etc. and transmits them to the emotion engine.

[1379] 2. The emotion engine analyzes the user's emotion information and sends the information to the server.

[1380] 3. The server receives the user's emotional information and adjusts product suggestions and the progress of the dialogue.

[1381] 4. The server searches and suggests suitable dresses from online shopping sites based on the user's answers and emotional information.

[1382] 5. The device displays detailed information and photos of the suggested dresses to the user.

[1383] 6. The user selects the suggested dress and proceeds with the purchase.

[1384] 7. The server provides a checkout link for the selected dress.

[1385] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[1386] The process flow will be explained below.

[1387] Step 1: Recognizing emotions and sending information

[1388] The terminal recognizes the user's emotions from their speech and facial expressions.

[1389] The terminal transmits the recognized emotion information to the emotion engine.

[1390] The emotion engine analyzes the received emotion information and transmits it to the server.

[1391] Step 2: Make suggestions and tailor dialogue based on emotions

[1392] The server receives the emotion information received from the emotion engine.

[1393] The server analyzes the emotion information and adjusts the progress of the dialogue and suggests products appropriate to the user's emotions.

[1394] The server searches and suggests suitable products from online shopping sites.

[1395] Step 3: View suggested products

[1396] The terminal receives detailed information and photos of the suggested products from the server.

[1397] The terminal displays the suggested items to the user.

[1398] The user browses the suggested products and makes a selection.

[1399] Step 4: Proceed with your purchase

[1400] The terminal receives a link from the server to proceed with the purchase procedure for the product selected by the user.

[1401] The terminal displays a link to the user, allowing them to complete the purchase procedure.

[1402] Step 5: Collect and analyze sentiment information

[1403] The server collects the user's emotion information and stores it in a database.

[1404] The server analyzes the collected emotion information to understand the user's emotional tendencies and preferences.

[1405] As described above, by processing a program that combines an emotion engine, it is possible to recognize the user's emotions and adjust product suggestions and the progress of the dialogue based on those emotions. This makes it possible to provide optimal services and support that are tailored to the user's emotions.

[1406] Example 2

[1407] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1408] Conventional online shopping systems suggest products without considering the user's emotions or mood at the time, making it difficult to suggest optimal products to users. In addition, since dialogue progression and product suggestions are not based on the user's emotions, there is a problem of a poor user experience. Furthermore, there is a lack of personalized suggestions to increase the user's motivation to purchase. For this reason, it is necessary to realize optimal product suggestions and dialogue progression that take into account the user's emotions.

[1409] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1410] In this invention, the server includes a means for analyzing the user's speech and facial expressions to recognize emotions, a means for transmitting the recognized emotion information to the server, and a means for adjusting the content of product proposals and the progress of the dialogue based on the user's emotions, thereby enabling optimal product proposals and dialogue progress that take the user's emotions into account.

[1411] "User" refers to an individual who uses the System to search for, select, and purchase Products.

[1412] "Dialogue" refers to interaction between a user and a system with the purpose of exchanging information.

[1413] "Needs" refers to the requests and desires for products and services that users require.

[1414] "Age" refers to the number of years calculated from the user's date of birth.

[1415] "Gender" refers to the sex of the user, such as male or female.

[1416] "Utterance" refers to the act of a user providing information to a system via voice.

[1417] "Facial expressions" refer to the movements and expressions of a user's face and are used to recognize emotions.

[1418] "Emotion" refers to the psychological state a user exhibits in response to a particular situation or stimulus.

[1419] An "emotion engine" refers to software that analyzes emotions from a user's speech and facial expressions.

[1420] "Server" refers to the central system that processes user information, makes product suggestions, and advances dialogue.

[1421] "Online shopping site" refers to a website that sells products over the Internet.

[1422] "Purchase history" refers to a record of products and services a user has previously purchased.

[1423] "Product recommendation accuracy" refers to the ability to appropriately recommend products based on a user's needs and emotions.

[1424] "Search conditions" refer to product search parameters that are set based on a user's needs and feelings.

[1425] "Purchase procedure" refers to the process by which a user officially purchases a product selected by the user.

[1426] A specific embodiment of the present invention is an online shopping system in which a user utilizes a terminal to receive product suggestions and progress in dialogue via a server. The system aims to improve the user experience by recognizing the user's emotions and making optimal product suggestions based on those emotions. A specific method for implementing the present invention will be described in detail below.

[1427] Emotion Engine

[1428] A user connects to the system using a device (such as a smartphone or tablet). The device has a built-in camera and microphone. The device collects the user's speech and facial expressions in real time via the camera and microphone, and transmits them to the emotion engine. The emotion engine uses voice recognition software and facial expression recognition software to analyze emotions from the user's speech and facial expressions.

[1429] The emotion engine generates the analysis results as numbers or categories (e.g., "happiness," "sadness," "surprise," etc.) and sends this information to a server. The software used by the emotion engine is a cloud-based API that enables real-time emotion analysis.

[1430] Server Features

[1431] The server receives the emotion information sent from the emotion engine. Based on this information, the server adjusts the dialogue scenario with the user and the product suggestions. For example, if the user expresses the emotion of "joy," the server will proceed with the dialogue with content optimized for the user's emotion, such as suggesting a dress with a gorgeous design and bright colors.

[1432] The server uses the user's speech and emotional information to search for suitable products on online shopping sites (e.g., Amazon API, Rakuten API). It then selects the most suitable product from the search results and sends it to the terminal. This allows the server to suggest the best product for the user.

[1433] Device Features

[1434] The terminal provides an interface that displays the product suggestions received from the server and the progress of the dialogue to the user. The user can view detailed information and photos of the suggested products on the terminal, select a product as necessary, and proceed with the purchase procedure.

[1435] As a concrete example, consider the case where a user is looking for a dress to attend a wedding. The user connects to the system through a terminal and says, "I'm looking for a dress for a wedding." The user's actions are captured by the camera and recorded by the microphone.

[1436] The device sends this information to the emotion engine, which detects emotions such as "happiness" and "anticipation." The server searches for suitable dresses from online shopping sites based on this emotion information and suggests them to the user. The user checks the details and photos of the suggested dresses on the device, selects the one they like, and proceeds with the purchase process.

[1437] Examples of prompt statements

[1438] For example, the following is an example of a prompt that a user may enter into the system:

[1439] I'm looking for a recommended dress to wear to a wedding.

[1440] "Please suggest a black formal dress that would please our users."

[1441] In this way, the entire system works together to recognize the user's emotions and make optimal product suggestions.

[1442] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1443] Step 1:

[1444] A user logs into the system through a terminal and inputs a request to search for a wedding dress.

[1445] Specifically, the user enters their user ID and password into the login screen of the terminal. Then, they enter "I'm looking for a wedding dress" into the search bar and click the search button. This sends a search request to the system.

[1446] Input: User login information and search query

[1447] Output: A search request to the server.

[1448] Step 2:

[1449] The device collects the user's speech and facial expressions using a camera and microphone and sends them to the emotion engine.

[1450] Specifically, the device captures the user's facial expressions with a built-in camera and records audio with a microphone, and these data are sent to the emotion engine in real time.

[1451] Input: User's speech and facial expression data

[1452] Output: Sending data to the emotion engine

[1453] Step 3:

[1454] The emotion engine analyzes the user's emotions and sends the information to the server.

[1455] Specifically, the emotion engine converts voice data to text using an API that converts voice data to text, and analyzes facial expression data using an API that analyzes facial expression data. This identifies emotion categories such as "happiness" or "expectation." The analysis results are sent to the server as emotion information.

[1456] Input: Text data of utterances and facial expression feature data

[1457] Output: Emotion information

[1458] Step 4:

[1459] The server adjusts product search conditions based on the emotion information received.

[1460] Specifically, the server receives the emotion information and sets the characteristics of products that have attributes that the user finds pleasing (e.g., gorgeousness or color) as search conditions. It also takes into account the user's past purchase history and preferences to personalize the search conditions.

[1461] Input: Emotional information and past purchase history data

[1462] Output: Optimized search criteria

[1463] Step 5:

[1464] The server searches for suitable dresses on an online shopping site and transmits the results to the terminal.

[1465] Specifically, the server uses the API of the online shopping site to perform a product search based on the search criteria set by the user, selects the product that best matches the emotion information from the search results, and sends its details to the device.

[1466] Input: Optimized search criteria

[1467] Output:Details of selected product

[1468] Step 6:

[1469] The device will display detailed information and photos of the suggested dresses to the user.

[1470] Specifically, the device displays the dress information received from the server on a GUI, allowing the user to swipe through photos of the dress or tap on detailed information to view it.

[1471] Input:Details of selected product

[1472] Output: Information displayed to the user

[1473] Step 7:

[1474] The user selects a suggested dress and proceeds with the purchase.

[1475] Specifically, the user selects a dress from the suggestions and clicks the "Purchase" button. The terminal displays a checkout screen, allowing the user to enter credit card information and a shipping address. The user then confirms the purchase and clicks the "Confirm Purchase" button.

[1476] Input: User selection and purchase procedure information

[1477] Output: Purchase confirmation notice

[1478] This allows the entire system to work together to provide optimal product suggestions and purchasing procedures that take the user's emotions into account.

[1479] (Application example 2)

[1480] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1481] Conventional online shopping systems have difficulty in accurately understanding a user's preferences and needs and suggesting appropriate products. In addition, since product recommendations cannot be made taking into account the user's emotions, user satisfaction may decrease. Furthermore, in dialogue-based interactions, the progress of the dialogue cannot be adjusted according to the user's emotions, which may result in a poor user experience.

[1482] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for recognizing emotions from the user's speech and facial expressions, a means for transmitting to the server based on emotion information and making appropriate product recommendations, a means for understanding the needs, age, and gender through dialogue with the user, a means for providing details of a product the user wants to purchase and outfit suggestions suited to a specific event, a means for acquiring the user's past purchase history and reactions to products and improving the accuracy of product suggestions, and a means for adjusting the progress of the dialogue based on emotion information. This makes it possible to realize appropriate product suggestions and optimal interactions based on the user's emotions, improving the user experience.

[1483] "Means for recognizing emotions from a user's speech and facial expressions" refers to a method for identifying emotions by analyzing a user's facial expressions and voice using sensors such as a camera and microphone.

[1484] The "means for transmitting to a server based on emotion information and making appropriate product recommendations" is a method for transmitting recognized emotion data to a server and proposing optimal products based on that information.

[1485] "Means for understanding the needs, age, and gender of a user through dialogue" refers to a method for using a chatbot or voice assistant to interact with a user and collect information about the user's needs, age, and gender.

[1486] "Means for providing details of products that a user wishes to purchase and suggestions of appropriate attire for a specific event" refers to a method for providing detailed information of products that a user wishes to purchase and suggestions of appropriate attire for a specific event or situation.

[1487] "Means of acquiring a user's past purchase history and reactions to products to improve the accuracy of product suggestions" refers to a method of collecting a user's history of past purchases and the user's impressions and feedback about those products, and using that information to improve the accuracy of product recommendations.

[1488] The "means for adjusting the progress of a dialogue based on emotional information" is a method for optimizing the speed and content of a dialogue based on the user's current emotional state.

[1489] The embodiment of the present invention is a smart shopping assistant system that recommends products based on the user's emotions. This system analyzes the user's facial expressions and speech when the user uses a smartphone to shop online, and the server suggests appropriate products based on the emotional information.

[1490] First, the user's device is equipped with a camera and a microphone, and these sensors are used to capture the user's facial expressions and voice. The specific hardware used here is a smartphone's front camera and built-in microphone. The captured data is processed using OpenCV or a similar image processing library to recognize emotions from facial expressions.

[1491] A pre-trained emotion recognition model (e.g., a deep learning model built with Keras) is used to recognize user emotions. This model analyzes input image data and identifies the user's emotional state (e.g., joy, sadness, anger, etc.).

[1492] The recognized emotion information is then sent to a server via the Internet. The server receives this emotion information and combines it with the user's past purchase history and reaction data to search for and recommend the most suitable products. The server also takes into account information registered by the user, such as needs, age, and gender, to improve the accuracy of product recommendations.

[1493] In addition, the script in charge of the dialogue with the user is executed by the server, and the dialogue progress is adjusted based on the emotion information. For example, if the user is interested but has a slightly dissatisfied expression, the dialogue script is adjusted to carefully explain the product's features and benefits. On the other hand, if the user has a happy expression, a more relaxed dialogue style is selected.

[1494] A distinctive feature of this system is the adjustment of the dialogue progress based on emotional information, which allows users to experience product recommendations and interfaces optimized for their own emotional state, improving the overall satisfaction of the shopping experience.

[1495] As a concrete example, consider a case where a user is looking for a dress to attend a wedding. When the user speaks into the smartphone camera, the facial expression and speech are analyzed by the emotion engine. The analyzed emotion information is then sent to the server, which suggests dresses that match the user's needs. This process makes it easier for the user to find a dress that fits their emotions.

[1496] Example of a prompt to be input to a generative AI model:

[1497] Create a smartphone application for an online shopping site that provides a product recommendation service based on the user's emotions. The application will recognize emotions from the user's facial expressions and voice, and suggest the most suitable products based on those emotions.

[1498] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1499] Step 1:

[1500] A user starts a smartphone app and turns on the camera and microphone. The user starts talking to the screen or searching for information about a product. The input data here is the user's facial image and voice, which are captured by the camera and microphone. Facial expression data and voice data are acquired through the user's operations.

[1501] Step 2:

[1502] The device processes the captured video and audio in real time to perform face and voice recognition. Specifically, it uses the OpenCV library to detect faces and crop specific facial regions. The cropped facial images are input to a pre-trained emotion recognition model (a deep learning model using Keras). Here, the input data is the image of the facial region, and the output is emotion information (e.g., anger, joy, sadness, etc.).

[1503] Step 3:

[1504] The device sends the recognized emotion information and voice data to the server, where the device formats the data and sends it to the server using an HTTP request. The input is the payload of emotion information and voice data, which is sent to the server. The server receives the request and starts to analyze the emotion information and voice data.

[1505] Step 4:

[1506] The server understands the user's needs based on the received emotion information and voice data, and searches for appropriate products by referring to past purchase history and user feedback. In this step, the server executes a database query to search the user profile and filters products based on the obtained emotion information. The inputs are emotion information, voice data, and user profile information from the database. The output is a list of recommended products.

[1507] Step 5:

[1508] The server sends the recommended product list to the terminal. Here, the product list generated by the server is converted to JSON format or similar and sent to the terminal as an HTTP response. The input is the recommended product list, and the output is the response sent to the terminal.

[1509] Step 6:

[1510] The terminal displays the received product list to the user. The user can check the product details on the screen and select the items he likes. Here, the input is the product list received from the server, and the output is the product information displayed to the user. The terminal updates the user interface and allows the user to select products.

[1511] Step 7:

[1512] When the user selects a product, the terminal sends the selection information to the server. The server then proceeds with the purchase procedure based on the received information. Specifically, it securely processes credit card information and links with the ordering system. The input here is the user's selection information, and the output is a notification that the purchase procedure has been completed.

[1513] Through the above steps, optimal product recommendations based on emotion information and smooth interaction with the user are realized.

[1514] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1515] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1516] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the robot 414.

[1517] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[1518] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1519] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1520] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1521] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[1522] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[1523] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[1524] Although the system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[1525] In the above embodiment, an example is given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to input data.

[1526] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1527] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[1528] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1529] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[1530] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[1531] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1532] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[1533] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[1534] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[1535] The following is further disclosed regarding the above embodiment.

[1536] (Claim 1) A system for understanding a user's preferences and needs and proposing optimal products, A means for understanding the needs, age, and gender of a user through a dialogue with said user; A means for providing details of products the user wishes to purchase and suggestions on clothing suitable for a particular event; A means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions; A system comprising:

[1537] (Claim 2) Searching for suitable products from an online shopping site based on the user's answers; The system of claim 1 further comprising a means for making a suggestion.

[1538] (Claim 3) The system of claim 1 , further comprising means for providing detailed information or photos of the product that the user likes.

[1539] (Claim 4) A system that combines an emotion engine that recognizes user emotions, An emotion engine that recognizes the emotion of the user from the user's speech, facial expression, etc.; A means for adjusting product suggestions and dialogue progress based on the user's emotions; A means for providing appropriate services and support according to the user's feelings; A system comprising:

[1540] (Claim 5) 5. The system of claim 4, wherein the emotion engine includes means for recognizing the emotion of the user in real time and transmitting information of the emotion of the user to a server.

[1541] (Claim 6) The system of claim 4 , wherein the emotion engine includes means for adjusting product suggestions or dialogue progression based on the user's emotion.

[1542] "Example 1" (Claim 1) A system for understanding a user's preferences and needs and proposing optimal products, A means for understanding the needs, age, and gender of a user through a dialogue with said user; A means for providing details of products the user wishes to purchase and suggestions on clothing suitable for a particular event; A means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions; A means of analyzing user information using a generative AI model and searching an online database for suitable products; A means for transmitting search results obtained by the generative AI model to a user's terminal and displaying the search results to the user; means for providing a link to proceed with a purchase based on a user's selection; A system including:

[1543] (Claim 2) The system of claim 1, further comprising means for analyzing user information by the generative AI model, searching for suitable products from an online database, and suggesting them to the user.

[1544] (Claim 3) The system according to claim 1, further comprising a means for providing detailed information or an image of a product that the user likes, and facilitating a purchase procedure for the product.

[1545] "Application example 1" (Claim 1) A system for understanding a user's preferences and needs and proposing optimal products, A means for understanding the needs, age, and gender of a user through a dialogue with said user; A means for providing details of products the user wishes to purchase and suggestions on clothing suitable for a particular event; A means for acquiring the user's past purchase history and reactions to products and improving the accuracy of product suggestions; A means for suggesting products to customers in a store in real time through smart glasses and displaying them on a display; A system including:

[1546] (Claim 2) 2. The system of claim 1, further comprising: means for searching and suggesting suitable products from an online database based on the user's answers.

[1547] (Claim 3) The system of claim 1, further comprising: means for displaying detailed information or photos of the product liked by the user on a display of the smart glasses.

[1548] "Example 2 of combining emotion engines" (Claim 1) A means of understanding the needs, age and gender of users through dialogue with them; A means for users to provide details about products they want to purchase and outfit suggestions for specific events, etc. A means for acquiring a user's past purchase history and reactions to products to improve the accuracy of product suggestions; A means for analyzing a user's speech and facial expressions to recognize emotions; A means for transmitting the recognized emotion information to a server; A means for adjusting product suggestions and dialogue progress based on the user's emotions; A system including:

[1549] (Claim 2) 2. The system according to claim 1, further comprising: means for searching and suggesting suitable products from an online shopping site based on the user's answer.

[1550] (Claim 3) The system of claim 1, further comprising means for providing detailed information or photos of products that the user likes.

[1551] "Application example 2 when combining emotion engines" (Claim 1) A system for understanding a user's preferences and needs and proposing optimal products, A means for recognizing emotions from the user's speech and facial expressions; A means for transmitting the emotion information to a server and making appropriate product recommendations based on the emotion information; A means for understanding the needs, age, and gender of a user through a dialogue with said user; A means for providing details of products the user wishes to purchase and suggestions on clothing suitable for a particular event; A means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions; A means for adjusting a progress of a dialogue based on the emotion information; A system including:

[1552] (Claim 2) The system according to claim 1, further comprising: means for searching and suggesting suitable products from an online shopping site based on the user's answer and emotion information.

[1553] (Claim 3) The system of claim 1, further comprising means for providing detailed information or photos of the product that the user likes. [Explanation of symbols]

[1554] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A system for understanding a user's preferences or needs and proposing optimal products, A means for understanding the needs, age, and gender of a user through a dialogue with said user; A means for providing details of products the user wishes to purchase and suggestions on clothing suitable for a particular event; A means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions; A means for analyzing the user's information using a generative AI model and searching an online database for suitable products; A means for transmitting search results obtained by the generative AI model to a terminal of the user and displaying the search results on the terminal; means for providing a link for proceeding with a product purchase procedure based on the user's selection; A system comprising:

2. The system of claim 1 , further comprising means for analyzing information of the user by the generative AI model, searching for appropriate products from an online database, and suggesting them to the user.

3. The system according to claim 1 , further comprising a means for providing detailed information or images of a product that the user likes, and facilitating a purchase procedure for the product.

4. A system that combines an emotion engine that recognizes user emotions, A means for understanding the needs, age, and gender of a user through a dialogue with said user; A means for providing details of a product that the user wants to purchase or a suggestion of an outfit to be worn for a specific event based on the emotion of the user recognized by the emotion engine; A means for acquiring the user's past purchase history and reactions to products to improve the accuracy of product suggestions; A means for analyzing the user's speech and facial expressions using the emotion engine and recognizing emotions; A means for transmitting the recognized emotion information to a server; A means for adjusting the content of product suggestions and the progress of the dialogue based on the user's feelings; A system comprising:

5. The system according to claim 4, further comprising means for searching and suggesting suitable products from an online shopping site based on the user's answer.

6. The system according to claim 4 , further comprising means for providing detailed information or pictures of the products that the user likes.

Images