system

Abstract

Provide a system. A means for a user to select a product; A means for the server to manage product information; a means for the terminal to display product information to a user; a means for the server to add items to the cart; a means for a user to enter payment information; a means by which the server processes payments; a means by which the server confirms the order; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, with the spread of e-commerce, many users have begun purchasing products over the Internet. In this environment, users are required to easily select products and efficiently proceed with the purchasing process. However, conventional e-commerce systems require users to follow many procedures at each processing step, making operation complex and resulting in insufficient user convenience. Furthermore, data communication between the server and terminals tends to be cumbersome when managing product information and processing payments, which can slow down the operation of the entire system. This can discourage users from purchasing. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means: a system including a means for a user to select a product, a server managing product information, and a terminal displaying the product information to the user. The system further includes a means for the server to add products to a cart, a means for the user to input payment information, a means for the server to process payments, and a means for the server to confirm the order. This allows users to purchase products efficiently with fewer steps. The server also includes a means for storing product information in a database and retrieving product information from the database in response to a user's selection and transmitting it to the terminal, facilitating product information management and enabling rapid data communication. In this way, user convenience is enhanced and the system operates quickly and smoothly.

[0006] "User" means any person or entity that uses the System to browse, select, and purchase Products.

[0007] A "server" is a computer and associated software that provides functions such as managing product information, storing information in a database, processing payments, and confirming orders in an e-commerce system.

[0008] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that communicates with the server to display product information and provides an interface for carrying out the purchase process.

[0009] "Product information" refers to information such as the name, price, description, image, and stock status of a product registered in the system.

[0010] A "cart" is a virtual container that temporarily stores products that a user wishes to purchase, and is used for the final purchase procedure.

[0011] "Payment information" refers to information relating to payment methods such as credit card information, debit card information, and bank account information that a user enters when purchasing a product.

[0012] "Payment" is the process of electronically paying for the product based on the payment information selected by the user.

[0013] An "order" is a purchase request recorded by the server after a user completes a purchase of a product.

[0014] A "database" is a data storage system for centrally managing product information, user information, order information, and the like in an electronic commerce system. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. An embodiment of the present invention will be described below with specific examples.

[0037] System Configuration

[0038] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[0039] Product Selection

[0040] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[0041] Add to Cart

[0042] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[0043] Purchase procedure

[0044] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[0045] Payment Processing

[0046] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[0047] Order Confirmed

[0048] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[0049] Specific examples

[0050] For example, the process for a user purchasing a "smartphone case" is described as follows: The user accesses an e-commerce site and displays the product list page. The server obtains product information, generates a list page, and sends it to the user's device. The user clicks "smartphone case" to display detailed information. The server obtains the detailed information, generates a detail page, and sends it to the device. When the user clicks "Add to cart," the server adds the product information to the cart. When the user checks the cart and clicks "Proceed to checkout," the server generates a checkout page. When the user enters payment information and clicks "Complete payment," the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[0051] As described above, the system of the present invention allows users to purchase products efficiently and quickly. Because the entire system operates in cooperation with one another, user operation is simplified and the purchasing experience is improved.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[0055] Step 2:

[0056] The server generates HTML data for the homepage and sends it to the terminal, which receives the HTML data and displays it in its browser.

[0057] Step 3:

[0058] Users use the categories and search bar on the page to access a list of products they are interested in. The device sends a request to the server containing the selected category and search keywords.

[0059] Step 4:

[0060] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser.

[0061] Step 5:

[0062] A user clicks on a specific product link to view more information about that product. The device detects the click event and sends a request to the server that includes the product ID.

[0063] Step 6:

[0064] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser.

[0065] Step 7:

[0066] The user clicks the "Add to Cart" button. The device detects this action and sends a request to the server that includes the product ID and user ID.

[0067] Step 8:

[0068] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser.

[0069] Step 9:

[0070] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server containing the current cart contents and the user ID.

[0071] Step 10:

[0072] The server retrieves the user's cart contents from the cart table, generates an HTML page for the checkout process, and sends it to the terminal. The terminal receives this HTML and displays the checkout process page in the browser.

[0073] Step 11:

[0074] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[0075] Step 12:

[0076] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful.

[0077] Step 13:

[0078] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser.

[0079] Through these steps, users can select products, add them to their cart, and complete the purchase process. The entire system works together to enable users to purchase products intuitively and efficiently.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] In conventional e-commerce systems, the process of a user selecting a product, adding it to a cart, completing the purchase procedure, and completing the payment often involves multiple, cumbersome steps. Furthermore, if data exchange with the server during the shopping process is not smooth, the user experience can be poor and purchasing motivation can decrease. The present invention aims to solve these problems and provide a system that allows users to purchase products efficiently and quickly.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes means for retrieving product information from a database, generating an HTML web page, and sending it to the user's terminal, means for receiving a request including the cart contents and user ID, confirming the product information and user information, and generating a purchase procedure page, and means for sending payment information to the payment service and completing the payment procedure. This allows the user to smoothly and quickly complete operations from product selection to purchase confirmation.

[0085] A "user" is a general user who uses the system to select products and complete the purchase procedure.

[0086] A "server" is a computer system that manages product information, user information, and order information, and performs the necessary data processing in response to various requests.

[0087] "Product information" refers to detailed information stored in the database, such as product name, price, description, and stock quantity.

[0088] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that communicates with a server and displays web pages.

[0089] A "cart" is a temporary data storage space that temporarily stores products that a user is considering purchasing.

[0090] A "database" is a collection of data that stores product information and user information in an organized manner and can be searched and retrieved.

[0091] An "HTML web page" is a document format that is generated by a server, sent to a terminal, and can be viewed by a user using a browser.

[0092] A "product ID" is an identifier that uniquely identifies a product.

[0093] A "user ID" is an identifier that uniquely identifies a user.

[0094] A "web browser" is software for viewing HTML web pages on a terminal.

[0095] A "payment service" is an online service that processes actual payments based on payment information provided by the user.

[0096] "Payment success or failure" refers to whether the payment was successful or failed based on the payment information.

[0097] The "order table" is a part of the database that manages user order information.

[0098] "Stock information" is data that manages the current stock quantity for each product.

[0099] A "purchase procedure page" is a web page where a user enters the shipping address and payment information required to make a purchase.

[0100] The "order completion page" is a web page that allows a user to confirm that the purchase procedure has been completed.

[0101] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. The following describes an embodiment of the present invention with specific examples.

[0102] System Configuration

[0103] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[0104] Product Selection

[0105] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[0106] For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone case" from the database and displays that information to the user.

[0107] Add to Cart

[0108] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[0109] For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID and user ID of the "blue smartphone case" to the server, and the server adds it to the cart.

[0110] Purchase procedure

[0111] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[0112] For example, when a user clicks "Proceed to Checkout," the server generates a checkout page and displays a form for entering a shipping address and payment method.

[0113] Payment Processing

[0114] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[0115] For example, when a user enters credit card information and clicks "Complete Payment," the server sends that information to the payment service and, if the payment is successful, returns a success message.

[0116] Order Confirmed

[0117] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[0118] For example, when a user confirms the purchase of a "blue smartphone case," the server updates the order information, reduces the inventory quantity, and generates and sends an order completion page.

[0119] This allows users to purchase products efficiently and quickly. Because the entire system works in cooperation with one another, user operation is simple and the purchasing experience is improved.

[0120] Examples of prompt statements

[0121] Below is an example of a prompt to input to a generative AI model:

[0122] "Please explain the process from when a user selects a product on an e-commerce site to when they complete the purchase process. Please provide a detailed explanation, including each step: selecting a product, adding it to the cart, paying, and confirming the order. Please also use a 'blue smartphone case' as an example of a specific product."

[0123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0124] Step 1:

[0125] The user browses products on the device and opens the product selection page.

[0126] Input: The user enters the URL of the e-commerce site in a web browser and accesses it.

[0127] Data processing: The server receives the request, retrieves product information from the database, and generates an HTML web page.

[0128] Output: The server sends the generated HTML page to the user's terminal.

[0129] Specific operation: For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone cases" from the database and generates an HTML page.

[0130] Step 2:

[0131] The terminal displays the received page in a browser, and the user browses the product list.

[0132] Input: The HTML page sent by the server.

[0133] Data processing: The device renders the HTML page in a web browser.

[0134] Output: Product list page displayed in browser.

[0135] Specific operation: The user checks the list of "smartphone cases" in the browser.

[0136] Step 3:

[0137] A user adds a specific product to their cart.

[0138] Input: User clicks the "Add to Cart" button.

[0139] Data processing: The terminal sends a request including the product ID and user ID to the server.

[0140] Output: The request sent from the terminal to the server.

[0141] Specific operation: For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID of the "blue smartphone case" and the user ID to the server.

[0142] Step 4:

[0143] The server updates the cart information.

[0144] Input: A request sent from the device containing the product ID and user ID.

[0145] Data processing: The server receives the request and adds the user ID and product ID to the cart table.

[0146] Output: A success message is sent from the server to the device.

[0147] Specific operation: The server receives the request and adds the product information to the cart. If the addition is successful, the server generates a success message and sends it to the user's device.

[0148] Step 5:

[0149] The user checks the contents of the cart and clicks the "Proceed to checkout" button.

[0150] Input: The user clicks the "Proceed to Checkout" button.

[0151] Data processing: The terminal sends a request including the cart contents and the user ID to the server.

[0152] Output: The request sent from the terminal to the server.

[0153] Specific operation: The user presses the "Proceed to checkout" button on the cart page.

[0154] Step 6:

[0155] The server generates the checkout page.

[0156] Input: A request sent from the device containing the cart contents and user ID.

[0157] Data processing: The server checks the product information and user information and generates a purchase procedure page.

[0158] Output: The generated checkout page is sent from the server to the device.

[0159] Specific operation: Based on the acquired information, the server generates a purchase procedure page where the user can enter their shipping address and payment information, and sends it to the user's terminal.

[0160] Step 7:

[0161] The user enters payment information and clicks the "Complete Payment" button.

[0162] Enter: The user fills out the payment form and clicks the "Complete Payment" button.

[0163] Data processing: The terminal sends a request to the server containing the entered payment information.

[0164] Output: The request sent from the terminal to the server.

[0165] Specific operation: The user enters credit card information and clicks "Complete payment."

[0166] Step 8:

[0167] The server processes the payment.

[0168] Input: The request sent from the terminal containing payment information.

[0169] Data processing: The server sends the payment information received to the payment service and processes the payment.

[0170] Output: Receives a response from the payment service and sends a success / failure message from the server to the terminal.

[0171] Specific operation: The server sends payment information to the payment service, and if the payment is successful, it generates a success message and sends it to the user terminal.

[0172] Step 9:

[0173] The server confirms the order and generates an order completion page.

[0174] Input: Response from the payment service that the payment was successful.

[0175] Data processing: The server adds order information to the order table and updates inventory information.

[0176] Output: Order completion page sent from server to terminal.

[0177] Specific operations: The server updates the order information, reduces the stock quantity, generates an order completion page and sends it to the user terminal.

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] In conventional e-commerce systems, the processes of product selection, adding items to a cart, purchasing procedures, payment, and order confirmation are often time-consuming for users. In particular, when using smart devices, accessibility is limited because only a visual interface is used. Furthermore, voice command and gesture recognition technologies are not fully utilized, creating a need for improved user experience. This creates challenges for all users, including the elderly and people with disabilities, in making product purchases easily and quickly.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes a means for a user to select a product using a voice command or gesture recognition, a means for the server to manage product information, a means for a terminal to display product information to the user using voice commands or gesture recognition, a means for the server to add products to a cart based on the user's voice command or gesture, a means for the user to input payment information using a voice command, a means for the server to process voice payments, and a means for the server to confirm the order. This allows users to easily and quickly complete the process from product selection to purchase procedures by utilizing voice and gesture recognition. By improving the user experience and providing an operation method that does not rely on vision, an e-commerce system that is easy for all users to use is realized.

[0183] "Means for users to select products using voice commands or gesture recognition" refers to an interface that recognizes the voices and hand movements made by users toward the terminal and allows them to select products according to those instructions.

[0184] "Means for the server to manage product information" refers to a function that stores information such as product names, descriptions, prices, and stock status in a database, and allows for manipulation, updating, and retrieval as needed.

[0185] "Means for the terminal to display product information to the user through voice commands or gesture recognition" refers to a function that interprets the user's voice instructions or gestures and displays corresponding product information on the screen.

[0186] "Means for the server to add products to the cart based on the user's voice command or gesture" refers to a function in which, when the user issues an instruction to add a product to the cart using voice or gesture, the server receives that information and registers the product in the cart.

[0187] "A means for users to input payment information using voice commands" refers to a function in which a user verbally inputs a credit card number and payment method, which is then recognized and reflected in the system.

[0188] The "means for the server to process voice payments" is a function that obtains payment information based on the user's voice input and uses that information to complete the payment.

[0189] The "means for the server to confirm the order" is a function for officially registering order information and updating inventory information etc. after successful payment.

[0190] The present invention is a system that allows users to conduct e-commerce transactions using voice commands and gesture recognition, and is configured as follows: The main hardware consists of a user terminal such as a smartphone or smart glasses, a server, a voice recognition device, and a gesture recognition device. The software uses Google® Cloud Speech-to-Text, TENSORFLOW® & OpenCV, MySQL®, React, Node.js, and Stripe API.

[0191] System Configuration

[0192] User device: A smartphone or smart glasses accepts voice commands and gestures and sends the input to the server, using Google Cloud Speech-to-Text for voice recognition and TensorFlow & OpenCV for gesture recognition.

[0193] Server: The server manages products, adds them to the cart, processes payments, and confirms orders. To achieve this, a web interface built with React and Node.js is used. The database is MySQL.

[0194] Payment system: We use the Stripe API for payments, enabling secure voice payments.

[0195] Product Selection

[0196] When a user types a voice command into their device, such as "Find a smartphone case," the device converts the command into text using voice recognition and sends it to the server. The server retrieves product information from a database based on the received text and sends it to the user's device as a web page generated with React. The user can then select a product using voice or gestures.

[0197] Add to Cart

[0198] When a user issues a voice command such as "Add this to cart," the voice recognition system converts it into text and sends a request from the device to the server. The server updates the cart table based on the received product ID and user ID. If the addition is successful, the server generates a success message and sends it to the device.

[0199] Purchase procedure

[0200] When the user issues the voice command "Proceed to checkout," the device sends a request to the server based on the voice command, including the cart contents and user ID. The server processes the request and generates a checkout page. The page includes fields for entering shipping address and payment information, and is sent to the user's device. The user enters this information using voice commands.

[0201] Payment Processing

[0202] When the user issues the voice command "Complete payment," the terminal sends a request containing the collected payment information to the server. The server then sends it to the Stripe API to process the payment. Depending on whether the payment is successful or unsuccessful, the server performs the appropriate processing and notifies the user of the result.

[0203] Order Confirmed

[0204] If the payment is successful, the server adds new order information to the order table and updates the inventory information. Once the order is confirmed, the server generates an order completion page and sends it to the user's device. The user can view the order completion page to confirm that the purchase was successful.

[0205] Specific examples

[0206] For example, the process for a user purchasing a "smartphone case" is as follows: When the user issues a voice command such as "Find a smartphone case," the device converts the command into text and sends it to the server. The server retrieves product information from a database and sends it to the device. The user then instructs the server to "add this to cart," and the server adds the product to the cart. The user then issues a voice command such as "Proceed to checkout," and a checkout page is generated. When the user enters the necessary information and instructs the server to "Complete payment," the server processes the payment and the order is confirmed.

[0207] Prompt Sentence Examples

[0208] "Looking for a smartphone case"

[0209] "Add this to cart"

[0210] "Proceed to purchase"

[0211] "Complete payment"

[0212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0213] Step 1:

[0214] The user enters the voice command "Find my phone case" into the device.

[0215] Input: The user's voice command.

[0216] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice commands into text.

[0217] Output: The string "Find a phone case".

[0218] Step 2:

[0219] The terminal sends the string to the server.

[0220] Input: User's voice command converted into a string.

[0221] How it works: The device generates an HTTP request and sends it to the server.

[0222] Output: The request data received on the server side.

[0223] Step 3:

[0224] The server retrieves product information from the database and sends it to the user's terminal.

[0225] Input: Search query for the string "smartphone case".

[0226] How it works: The server searches the MySQL database for product information and retrieves the results.

[0227] Output: A list of product information sent to the user's device.

[0228] Step 4:

[0229] The terminal displays the product information to the user.

[0230] Input: A list of product information received from the server.

[0231] Operation: Generate and display a product list page using React in the device browser.

[0232] Output: A product listing page that can be viewed by the user.

[0233] Step 5:

[0234] The user enters the voice command "Add this to cart."

[0235] Input: The user's voice command.

[0236] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[0237] Output: The string "Add this to cart".

[0238] Step 6:

[0239] The device sends a request including the product ID and user ID to the server.

[0240] Input: Voice command converted into text, product ID, and user ID.

[0241] How it works: The device generates an HTTP request and sends it to the server.

[0242] Output: The request data received on the server side.

[0243] Step 7:

[0244] The server updates the cart table and generates a success message.

[0245] Input: Request data including product ID and user ID.

[0246] What happens: The server adds data to the cart table in the MySQL database and generates a success message.

[0247] Output: Sends a success message to the user's terminal.

[0248] Step 8:

[0249] The user enters the voice command "Proceed to checkout."

[0250] Input: The user's voice command.

[0251] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[0252] Output: The string "Proceed to checkout".

[0253] Step 9:

[0254] The device sends a purchase request to the server.

[0255] Input: Voice command converted into text, cart contents, and user ID.

[0256] How it works: The device generates an HTTP request and sends it to the server.

[0257] Output: The request data received on the server side.

[0258] Step 10:

[0259] The server generates a checkout page and sends it to the user's device.

[0260] Input: Request data including cart contents and user ID.

[0261] How it works: The server generates a checkout page, including fields for shipping address and payment information.

[0262] Output: Sends the completed checkout page to the user's device.

[0263] Step 11:

[0264] The user enters the voice command "Complete payment."

[0265] Input: The user's voice command.

[0266] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[0267] Output: The string "Complete payment".

[0268] Step 12:

[0269] The terminal sends the payment information to the server.

[0270] Input: Voice command converted to text and user payment information.

[0271] How it works: The device generates an HTTP request and sends it to the server.

[0272] Output: The request data received on the server side.

[0273] Step 13:

[0274] The server processes the payment and notifies the user of the result.

[0275] Input: Request data, including payment information.

[0276] What happens: Your server calls the Stripe API to process the payment and checks the result.

[0277] Output: Sends a notification message to the user's device based on the payment result.

[0278] Step 14:

[0279] The server confirms the order and generates an order completion page.

[0280] Input: The result of the payment process.

[0281] What it does: The server updates the orders table in the MySQL database and generates the order completion page.

[0282] Output: Sends the completed order completion page to the user's device.

[0283] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0284] The present invention provides a system that recognizes user emotions in an e-commerce system and efficiently performs product selection, cart addition, purchase procedure, payment, and order confirmation based on that information. An embodiment of the present invention will be described below with a specific example in which an emotion engine is combined.

[0285] System Configuration

[0286] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[0287] Product selection and display

[0288] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of product information.

[0289] Add to Cart

[0290] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[0291] Checkout and Emotion Recording

[0292] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, verifies the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[0293] Payment Processing

[0294] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[0295] Order Confirmed

[0296] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[0297] Specific examples

[0298] For example, let's consider the process of a user purchasing a "smartphone case." The user accesses an e-commerce site and displays the product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or expresses joy, it highlights products that the user is likely to be particularly interested in. When the user clicks on "smartphone case" and displays the detail page, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[0299] This invention uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works in tandem, allowing users to purchase products intuitively and efficiently.

[0300] The processing flow will be explained below.

[0301] Step 1:

[0302] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[0303] Step 2:

[0304] The server generates HTML data for the homepage and sends it to the device. The device receives this HTML data and displays it in the browser. At this time, the emotion engine is activated and analyzes the user's facial expressions and voice using the device's camera and microphone.

[0305] Step 3:

[0306] A user accesses a product list using a product category or search bar. The device sends a request to the server containing the selected category and search keywords.

[0307] Step 4:

[0308] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser. At this time, the emotion engine analyzes the user's reaction and optimizes the display order of the products according to their emotions.

[0309] Step 5:

[0310] A user clicks on a specific product link to view more information about it. The device detects the click event and sends a request including the product ID to the server. The emotion engine analyzes whether the user has a positive reaction to the product.

[0311] Step 6:

[0312] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser. The emotion engine analyzes the user's level of interest and suggests supplementary information or approaches as needed.

[0313] Step 7:

[0314] The user clicks the "Add to Cart" button. The device detects this action and sends a request including the product ID and user ID to the server. The emotion engine records the user's emotion when adding the item to the cart.

[0315] Step 8:

[0316] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser. If the emotion engine detects a positive reaction when the user adds a product to the cart, it displays a message to express joy.

[0317] Step 9:

[0318] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server including the current cart contents and the user ID. The emotion engine records the user's emotions as they proceed to checkout.

[0319] Step 10:

[0320] The server retrieves the user's cart contents from the cart table, generates the HTML for the checkout page, and sends it to the terminal. The terminal receives this HTML and displays the checkout page in the browser. The emotion engine analyzes the user's emotions and adjusts the interface to make the checkout process smoother.

[0321] Step 11:

[0322] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[0323] Step 12:

[0324] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful. The emotion engine records the user's emotion when making a payment.

[0325] Step 13:

[0326] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion page indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser. The emotion engine analyzes whether the user is satisfied when the order is completed, and displays an appropriate feedback message.

[0327] Through these steps, users can select products, add them to their cart, and complete the purchase process. By using the emotion engine, the purchasing experience can be customized to match the user's emotions, improving the purchasing experience.

[0328] Example 2

[0329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Conventional e-commerce systems lack the ability to recognize user emotions and optimize the purchasing process, which can lead to lower user satisfaction and purchasing motivation. They also fail to provide appropriate information, especially in difficult situations. Furthermore, they lack a mechanism for utilizing user emotion data to recommend future products.

[0331] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing a user's emotion, a means for optimizing product display based on emotion data, and a means for an emotion engine to record the user's emotion. This makes it possible to analyze the user's emotion in real time and optimize the display of product information and the content of messages, thereby improving user satisfaction and purchasing motivation. Furthermore, recording emotion data can improve the accuracy of future product recommendations.

[0332] "User" refers to a person who uses the system to select, view, and purchase products.

[0333] "Server" refers to a computer system that manages product information, processes data, optimizes emotional data, processes payments, and confirms orders.

[0334] A "terminal" refers to a device used by a user, such as a computer or smartphone, that displays product information and recognizes emotions.

[0335] "Product Information" refers to data including information such as product name, price, description, image, and availability.

[0336] A "cart" refers to a virtual shopping basket in which a user temporarily stores products that they wish to purchase.

[0337] "Payment Information" refers to information necessary to make a payment, such as credit card number, payment method, and billing address.

[0338] "Payment" refers to the process of paying for products using payment information entered by a user.

[0339] An "emotion engine" refers to software or hardware that uses the device's camera and microphone to analyze the user's facial expressions and voice and recognize the user's emotions.

[0340] "Emotion data" refers to data relating to a user's emotions recognized by the emotion engine.

[0341] "Product display optimization" refers to the process of dynamically adjusting the selection and order of products to be displayed, the content of messages, etc., based on user emotional data.

[0342] "Emotion recording" refers to the process of storing a user's emotional data in a database or other location and using it for later analysis or product recommendations.

[0343] The present invention is a system for recognizing a user's emotions in an e-commerce system, and based on this information, efficiently performs product selection, adding to cart, purchasing procedures, payment, and order confirmation. Specific embodiments of this system are described below.

[0344] System Configuration

[0345] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[0346] Product selection and display

[0347] When a user browses for a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of the product information.

[0348] Add to Cart

[0349] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[0350] Checkout and Emotion Recording

[0351] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[0352] Payment Processing

[0353] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[0354] Order Confirmed

[0355] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[0356] Specific examples

[0357] For example, let's consider the process of a user purchasing a smartphone case. The user accesses an e-commerce site and displays a product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or happy, it highlights products that the user is likely to be particularly interested in. When the user clicks on a smartphone case and the detail page is displayed, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[0358] This system uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works together, allowing users to purchase products intuitively and efficiently.

[0359] Prompt Sentence Examples

[0360] "We would like you to design a system that recognizes emotions by analyzing facial expressions and voice using a camera and microphone while a user is viewing a specific product, and optimizes the product display."

[0361] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0362] Product selection and display

[0363] Step 1:

[0364] The user accesses the e-commerce site on the device.

[0365] Input: The user enters the URL of the e-commerce site into a web browser.

[0366] How it works: The browser generates and sends a request to the server.

[0367] Output: A request to access the server.

[0368] Step 2:

[0369] The server retrieves product information from the database

[0370] Input: Access request and user request.

[0371] Operation: The server queries the database to retrieve product information.

[0372] Output: Retrieved product information data.

[0373] Step 3:

[0374] The server generates an HTML web page

[0375] Input: Retrieved product information data.

[0376] How it works: The server generates HTML code based on product information.

[0377] Output: The generated HTML code.

[0378] Step 4:

[0379] The server generates a web page and sends it to the user's device.

[0380] Input: The generated HTML code.

[0381] What it does: The server sends an HTTP response containing HTML code.

[0382] Output: HTML response to the user's device.

[0383] Step 5:

[0384] The page received by the user's device is displayed in the browser.

[0385] Input: The received HTML code.

[0386] What happens: The browser renders the HTML code and displays the product listing page.

[0387] Output: A product listing page viewable by the user.

[0388] Step 6:

[0389] The emotion engine uses the device's camera and microphone to recognize the user's emotions.

[0390] Input: Camera video and microphone audio.

[0391] How it works: The emotion engine uses video and audio analysis algorithms to analyze the user's emotions.

[0392] Output: Recognized emotion data.

[0393] Step 7:

[0394] The server optimizes product display based on emotional data

[0395] Input: Recognized emotion data.

[0396] How it works: The server analyzes the emotion data and sorts and highlights products based on the user's interests.

[0397] Output: Optimized product display data.

[0398] Add to Cart

[0399] Step 1:

[0400] User clicks the "Add to Cart" button

[0401] Input: User clicks.

[0402] Behavior: The browser triggers a click event and begins processing.

[0403] Output: The request data including the product ID and user ID.

[0404] Step 2:

[0405] The device sends a request to the server containing the product ID and user ID.

[0406] Input: Request data including product ID and user ID.

[0407] What happens: The browser sends the request data to the server.

[0408] Output: The request to the server.

[0409] Step 3:

[0410] The server receives the request and adds the information to the cart table.

[0411] Input: Request data to the server.

[0412] Operation: The server executes an operation to add the product ID and user ID to the cart table.

[0413] Output: The data added to the cart table.

[0414] Step 4:

[0415] The server generates a success message and sends it to the user's device.

[0416] Input: Result of adding data to the cart table.

[0417] What happens: The server generates a success message and sends it as an HTTP response.

[0418] Output: A success message to the user's terminal.

[0419] Step 5:

[0420] Emotion engine displays encouraging and humorous messages

[0421] Input: User's emotional data and operation details.

[0422] How it works: The emotion engine generates and displays messages based on the user's emotions.

[0423] Output: Encouraging and humorous messages.

[0424] Checkout and Emotion Recording

[0425] Step 1:

[0426] The user checks the contents of the cart and presses the "Proceed to checkout" button

[0427] Input: User clicks.

[0428] Behavior: The browser triggers a click event and begins processing.

[0429] Output: Request data including cart contents and user ID.

[0430] Step 2:

[0431] The device sends a request to the server containing the cart contents and the user ID.

[0432] Input: Request data including cart contents and user ID.

[0433] What happens: The browser sends the request data to the server.

[0434] Output: The request to the server.

[0435] Step 3:

[0436] The server receives the request and generates the checkout page

[0437] Input: Request data to the server.

[0438] How it works: The server parses the request data and generates a checkout page.

[0439] Output: HTML code for the checkout page.

[0440] Step 4:

[0441] Emotion engine records user emotions

[0442] Input: User emotion data during checkout.

[0443] How it works: The emotion engine analyzes the user's emotions and stores the data in a database.

[0444] Output: Saved emotion data.

[0445] Payment Processing

[0446] Step 1:

[0447] The user enters payment information and presses the "Complete payment" button

[0448] Input: When a user enters payment information and clicks a button.

[0449] Behavior: The browser triggers a click event and begins processing.

[0450] Output: The payment information entered.

[0451] Step 2:

[0452] The device sends the entered information to the server.

[0453] Input: Payment information entered.

[0454] How it works: The browser sends payment information to the server.

[0455] Output: Payment information request to server.

[0456] Step 3:

[0457] The server sends the data to the payment service and processes the payment.

[0458] Input: A request to the server for payment information.

[0459] What happens: The server sends the information to a payment service (e.g., a payment gateway) to execute the payment.

[0460] Output: Processing result from payment service.

[0461] Step 4:

[0462] The server checks whether the payment was successful and executes the process.

[0463] Input: Processing result from payment service.

[0464] Operation: The server checks whether the payment was successful or not, and performs the necessary next steps. For example, if successful, the order is confirmed, and if unsuccessful, an error message is generated.

[0465] Output: The processing result of the payment, whether successful or unsuccessful.

[0466] Order Confirmed

[0467] Step 1:

[0468] The server adds the order information to the orders table

[0469] Input: Payment successful processing result and order information.

[0470] Action: The server adds the order information to the orders table.

[0471] Output: The order information added to the orders table.

[0472] Step 2:

[0473] The server updates the inventory information

[0474] Input: Inventory information that needs to be updated.

[0475] Action: The server updates the inventory information in the database.

[0476] Output: Updated inventory information.

[0477] Step 3:

[0478] The server generates an order completion page and sends it to the user's device.

[0479] Input: Order and inventory information.

[0480] How it works: The server generates the HTML code for the order completion page and sends it to the terminal.

[0481] Output: HTML code for order completion page.

[0482] Step 4:

[0483] The user's device displays the order completion page in the browser.

[0484] Input: HTML code for order completion page.

[0485] What happens: The browser renders the HTML code and displays the order completion page.

[0486] Output: A user-viewable order completion page.

[0487] (Application example 2)

[0488] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0489] In modern e-commerce systems, there is a lack of emotionally-driven product recommendations and prompts to increase purchasing motivation when users make purchases. This leads to a suboptimal user experience, which can lead to a decline in purchasing motivation and a decrease in sales.

[0490] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on the emotions, means for collecting the user's facial expressions and voice via a camera and microphone, and means for adjusting the product recommendation method using the collected user emotion data. This makes it possible to recommend products and display messages optimally in accordance with the user's emotions.

[0491] "Means for user product selection" refers to an interface (website or application) that allows a user to select products online and designate specific products for purchase.

[0492] "Means for the server to manage product information" refers to a server or database system for storing, updating, deleting, and searching product data.

[0493] "Means by which a terminal displays product information to a user" refers to software or applications for displaying product information on a user's device (such as a smartphone, tablet, or computer).

[0494] "Means for the server to add products to the cart" refers to the server's function of adding products selected by the user to the shopping cart.

[0495] "Means for users to enter payment information" refers to the interface (form, input field, etc.) through which users enter the payment information required for purchase.

[0496] "Means by which the server processes payments" refers to the server's ability to actually process payments using the provided payment information.

[0497] "Means for the server to confirm the order" refers to the server's function of officially confirming the order after the payment is completed, and recording and processing the information.

[0498] "Means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on those emotions" refers to a system that analyzes a user's facial expressions and voice data collected using a camera and microphone, understands the user's emotional state, and displays product recommendations and customized messages based on that.

[0499] "Means for collecting the user's facial expressions and voice via a camera and microphone" refers to a camera and microphone device for acquiring the user's facial expressions and voice data in real time.

[0500] "Means for adjusting the product recommendation method using collected user emotional data" refers to a system that dynamically adjusts the product recommendation algorithm based on the user's emotional data and presents products that are optimal for the user's emotional state.

[0501] This invention provides a system for recognizing a user's emotions and efficiently selecting products, adding them to a cart, completing purchases, making payments, and finalizing orders within an e-commerce system. The system includes a server, a user terminal, an emotion recognition engine, a camera, and a microphone. A specific embodiment of this system will be described below.

[0502] The server uses an emotion recognition device to analyze the user's emotions and display product recommendations and messages based on those emotions. It also collects the user's facial expressions and voice via a camera and microphone, and adjusts the product recommendation method based on the collected data.

[0503] The device is a device that allows users to browse and select products. It can be a smartphone, tablet, or computer. The device's camera and microphone are used to collect user emotion data, which is then sent to a server. The server then uses this data to run an emotion recognition engine.

[0504] The emotion recognition engine analyzes the user's facial expressions and voice data to identify the user's emotions. This engine uses, for example, the machine learning library Keras and the image processing library OpenCV.

[0505] The server retrieves product information selected by the user from the database and sends it to the terminal. The user uses the terminal to select products and add them to the cart. At this time, the emotion recognition engine analyzes the user's emotions and displays a message to encourage the user to purchase.

[0506] The server then provides a means for the user to enter payment information and processes the payment. If the payment is successful, the server confirms the order information, updates inventory information, and finally generates an order completion page that is displayed on the terminal.

[0507] As a concrete example, imagine a situation where a user is browsing an online shopping site on a smartphone. As the user browses products using the smartphone's camera and microphone, an emotion recognition engine analyzes positive emotions such as joy and excitement from the user's facial expressions. Based on this emotional data, the server recommends the most suitable products for the user and displays messages to further encourage them to purchase.

[0508] An example of a prompt sentence when using a generative AI model is as follows:

[0509] "What products should we recommend if the user's emotions indicate delight or surprise?"

[0510] "If a user is expressing anger or sadness, what message can we show them to encourage them to buy?"

[0511] "Generate a blueprint for a real-time product recommendation system based on sentiment data."

[0512] In this way, by analyzing the user's emotional data in real time and using the results to recommend products and display messages, it is possible to significantly improve the user's purchasing experience.

[0513] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0514] Step 1:

[0515] The device uses a camera and microphone to record the user's facial expressions and voice in real time. The input is video and audio data from the camera and microphone. The output is the recorded video and audio data of the user. This data is used in the subsequent emotion recognition engine.

[0516] Step 2:

[0517] The terminal sends the collected user video and audio data to the server. The input is the user video and audio data recorded in step 1. The output is the video and audio data sent to the server.

[0518] Step 3:

[0519] The server inputs the received video and audio data into the emotion recognition engine to analyze the user's emotions. The input is the video and audio data sent to the server. The emotion recognition engine analyzes the data using OpenCV and Keras to identify the user's emotions (happiness, sadness, surprise, etc.). The output is the analyzed user's emotion data.

[0520] Step 4:

[0521] The server executes a process to recommend products based on the analyzed emotional data. The input is the emotional data obtained in step 3. The server uses a generative AI model to select the most suitable products based on the emotion. The output is a list of recommended products.

[0522] Step 5:

[0523] The server sends the list of recommended products to the terminal. The input is the list of recommended products generated in step 4. The output is the list of products sent to the terminal.

[0524] Step 6:

[0525] The terminal displays the received product list to the user. The input is the list of recommended products sent from the server. The output is the product information displayed to the user. Along with this display, the terminal also displays a message to encourage the user to purchase.

[0526] Step 7:

[0527] The user selects a product and clicks the "Add to Cart" button. The input is the operation data based on the user's selection. The output is the product information "added to cart."

[0528] Step 8:

[0529] The server adds items to the cart and updates their status. The input is the item selection data sent by the user. The server adds the user ID and item ID to the cart table and notifies the user that the cart has been updated. The output is the updated cart information.

[0530] Step 9:

[0531] The user clicks the "Proceed to Checkout" button and enters payment information. The input is the payment information the user enters into the terminal. The output is the payment information sent to the server.

[0532] Step 10:

[0533] The server receives payment information and processes the payment. The input is the payment information entered by the user. The server sends a request to the payment service to check whether the payment is successful. The output is information on whether the payment was successful or not.

[0534] Step 11:

[0535] If the payment is successful, the server confirms the order and updates the inventory information. The input is payment success / failure information and order information. The server adds the information to the order table and updates the inventory information. The output is the confirmed order information.

[0536] Step 12:

[0537] The server generates an order completion page and sends it to the terminal. The input is the confirmed order information. The terminal receives this page and displays it to the user. The output is the order completion page displayed to the user.

[0538] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0540] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0541] [Second embodiment]

[0542] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0543] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0544] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0545] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0546] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0547] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0548] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0549] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0550] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0551] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0552] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0553] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0554] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. An embodiment of the present invention will be described below with specific examples.

[0555] System Configuration

[0556] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[0557] Product Selection

[0558] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[0559] Add to Cart

[0560] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[0561] Purchase procedure

[0562] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[0563] Payment Processing

[0564] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[0565] Order Confirmed

[0566] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[0567] Specific examples

[0568] For example, the process for a user purchasing a "smartphone case" is described as follows: The user accesses an e-commerce site and displays the product list page. The server obtains product information, generates a list page, and sends it to the user's device. The user clicks "smartphone case" to display detailed information. The server obtains the detailed information, generates a detail page, and sends it to the device. When the user clicks "Add to cart," the server adds the product information to the cart. When the user checks the cart and clicks "Proceed to checkout," the server generates a checkout page. When the user enters payment information and clicks "Complete payment," the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[0569] As described above, the system of the present invention allows users to purchase products efficiently and quickly. Because the entire system operates in cooperation with one another, user operation is simplified and the purchasing experience is improved.

[0570] The processing flow will be explained below.

[0571] Step 1:

[0572] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[0573] Step 2:

[0574] The server generates HTML data for the homepage and sends it to the terminal, which receives the HTML data and displays it in its browser.

[0575] Step 3:

[0576] Users use the categories and search bar on the page to access a list of products they are interested in. The device sends a request to the server containing the selected category and search keywords.

[0577] Step 4:

[0578] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser.

[0579] Step 5:

[0580] A user clicks on a specific product link to view more information about that product. The device detects the click event and sends a request to the server that includes the product ID.

[0581] Step 6:

[0582] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser.

[0583] Step 7:

[0584] The user clicks the "Add to Cart" button. The device detects this action and sends a request to the server that includes the product ID and user ID.

[0585] Step 8:

[0586] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser.

[0587] Step 9:

[0588] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server containing the current cart contents and the user ID.

[0589] Step 10:

[0590] The server retrieves the user's cart contents from the cart table, generates an HTML page for the checkout process, and sends it to the terminal. The terminal receives this HTML and displays the checkout process page in the browser.

[0591] Step 11:

[0592] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[0593] Step 12:

[0594] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful.

[0595] Step 13:

[0596] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser.

[0597] Through these steps, users can select products, add them to their cart, and complete the purchase process. The entire system works together to enable users to purchase products intuitively and efficiently.

[0598] Example 1

[0599] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] In conventional e-commerce systems, the process of a user selecting a product, adding it to a cart, completing the purchase procedure, and completing the payment often involves multiple, cumbersome steps. Furthermore, if data exchange with the server during the shopping process is not smooth, the user experience can be poor and purchasing motivation can decrease. The present invention aims to solve these problems and provide a system that allows users to purchase products efficiently and quickly.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0602] In this invention, the server includes means for retrieving product information from a database, generating an HTML web page, and sending it to the user's terminal, means for receiving a request including the cart contents and user ID, confirming the product information and user information, and generating a purchase procedure page, and means for sending payment information to the payment service and completing the payment procedure. This allows the user to smoothly and quickly complete operations from product selection to purchase confirmation.

[0603] A "user" is a general user who uses the system to select products and complete the purchase procedure.

[0604] A "server" is a computer system that manages product information, user information, and order information, and performs the necessary data processing in response to various requests.

[0605] "Product information" refers to detailed information stored in the database, such as product name, price, description, and stock quantity.

[0606] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that communicates with a server and displays web pages.

[0607] A "cart" is a temporary data storage space that temporarily stores products that a user is considering purchasing.

[0608] A "database" is a collection of data that stores product information and user information in an organized manner and can be searched and retrieved.

[0609] An "HTML web page" is a document format that is generated by a server, sent to a terminal, and can be viewed by a user using a browser.

[0610] A "product ID" is an identifier that uniquely identifies a product.

[0611] A "user ID" is an identifier that uniquely identifies a user.

[0612] A "web browser" is software for viewing HTML web pages on a terminal.

[0613] A "payment service" is an online service that processes actual payments based on payment information provided by the user.

[0614] "Payment success or failure" refers to whether the payment was successful or failed based on the payment information.

[0615] The "order table" is a part of the database that manages user order information.

[0616] "Stock information" is data that manages the current stock quantity for each product.

[0617] A "purchase procedure page" is a web page where a user enters the shipping address and payment information required to make a purchase.

[0618] The "order completion page" is a web page that allows a user to confirm that the purchase procedure has been completed.

[0619] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. The following describes an embodiment of the present invention with specific examples.

[0620] System Configuration

[0621] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[0622] Product Selection

[0623] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[0624] For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone case" from the database and displays that information to the user.

[0625] Add to Cart

[0626] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[0627] For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID and user ID of the "blue smartphone case" to the server, and the server adds it to the cart.

[0628] Purchase procedure

[0629] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[0630] For example, when a user clicks "Proceed to Checkout," the server generates a checkout page and displays a form for entering a shipping address and payment method.

[0631] Payment Processing

[0632] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[0633] For example, when a user enters credit card information and clicks "Complete Payment," the server sends that information to the payment service and, if the payment is successful, returns a success message.

[0634] Order Confirmed

[0635] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[0636] For example, when a user confirms the purchase of a "blue smartphone case," the server updates the order information, reduces the inventory quantity, and generates and sends an order completion page.

[0637] This allows users to purchase products efficiently and quickly. Because the entire system works in cooperation with one another, user operation is simple and the purchasing experience is improved.

[0638] Examples of prompt statements

[0639] Below is an example of a prompt to input to a generative AI model:

[0640] "Please explain the process from when a user selects a product on an e-commerce site to when they complete the purchase process. Please provide a detailed explanation, including each step: selecting a product, adding it to the cart, paying, and confirming the order. Please also use a 'blue smartphone case' as an example of a specific product."

[0641] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0642] Step 1:

[0643] The user browses products on the device and opens the product selection page.

[0644] Input: The user enters the URL of the e-commerce site in a web browser and accesses it.

[0645] Data processing: The server receives the request, retrieves product information from the database, and generates an HTML web page.

[0646] Output: The server sends the generated HTML page to the user's terminal.

[0647] Specific operation: For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone cases" from the database and generates an HTML page.

[0648] Step 2:

[0649] The terminal displays the received page in a browser, and the user browses the product list.

[0650] Input: The HTML page sent by the server.

[0651] Data processing: The device renders the HTML page in a web browser.

[0652] Output: Product list page displayed in browser.

[0653] Specific operation: The user checks the list of "smartphone cases" in the browser.

[0654] Step 3:

[0655] A user adds a specific product to their cart.

[0656] Input: User clicks the "Add to Cart" button.

[0657] Data processing: The terminal sends a request including the product ID and user ID to the server.

[0658] Output: The request sent from the terminal to the server.

[0659] Specific operation: For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID of the "blue smartphone case" and the user ID to the server.

[0660] Step 4:

[0661] The server updates the cart information.

[0662] Input: A request sent from the device containing the product ID and user ID.

[0663] Data processing: The server receives the request and adds the user ID and product ID to the cart table.

[0664] Output: A success message is sent from the server to the device.

[0665] Specific operation: The server receives the request and adds the product information to the cart. If the addition is successful, the server generates a success message and sends it to the user's device.

[0666] Step 5:

[0667] The user checks the contents of the cart and clicks the "Proceed to checkout" button.

[0668] Input: The user clicks the "Proceed to Checkout" button.

[0669] Data processing: The terminal sends a request including the cart contents and the user ID to the server.

[0670] Output: The request sent from the terminal to the server.

[0671] Specific operation: The user presses the "Proceed to checkout" button on the cart page.

[0672] Step 6:

[0673] The server generates the checkout page.

[0674] Input: A request sent from the device containing the cart contents and user ID.

[0675] Data processing: The server checks the product information and user information and generates a purchase procedure page.

[0676] Output: The generated checkout page is sent from the server to the device.

[0677] Specific operation: Based on the acquired information, the server generates a purchase procedure page where the user can enter their shipping address and payment information, and sends it to the user's terminal.

[0678] Step 7:

[0679] The user enters payment information and clicks the "Complete Payment" button.

[0680] Enter: The user fills out the payment form and clicks the "Complete Payment" button.

[0681] Data processing: The terminal sends a request to the server containing the entered payment information.

[0682] Output: The request sent from the terminal to the server.

[0683] Specific operation: The user enters credit card information and clicks "Complete payment."

[0684] Step 8:

[0685] The server processes the payment.

[0686] Input: The request sent from the terminal containing payment information.

[0687] Data processing: The server sends the payment information received to the payment service and processes the payment.

[0688] Output: Receives a response from the payment service and sends a success / failure message from the server to the terminal.

[0689] Specific operation: The server sends payment information to the payment service, and if the payment is successful, it generates a success message and sends it to the user terminal.

[0690] Step 9:

[0691] The server confirms the order and generates an order completion page.

[0692] Input: Response from the payment service that the payment was successful.

[0693] Data processing: The server adds order information to the order table and updates inventory information.

[0694] Output: Order completion page sent from server to terminal.

[0695] Specific operations: The server updates the order information, reduces the stock quantity, generates an order completion page and sends it to the user terminal.

[0696] (Application example 1)

[0697] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0698] In conventional e-commerce systems, the processes of product selection, adding items to a cart, purchasing procedures, payment, and order confirmation are often time-consuming for users. In particular, when using smart devices, accessibility is limited because only a visual interface is used. Furthermore, voice command and gesture recognition technologies are not fully utilized, creating a need for improved user experience. This creates challenges for all users, including the elderly and people with disabilities, in making product purchases easily and quickly.

[0699] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0700] In this invention, the server includes a means for a user to select a product using a voice command or gesture recognition, a means for the server to manage product information, a means for a terminal to display product information to the user using voice commands or gesture recognition, a means for the server to add products to a cart based on the user's voice command or gesture, a means for the user to input payment information using a voice command, a means for the server to process voice payments, and a means for the server to confirm the order. This allows users to easily and quickly complete the process from product selection to purchase procedures by utilizing voice and gesture recognition. By improving the user experience and providing an operation method that does not rely on vision, an e-commerce system that is easy for all users to use is realized.

[0701] "Means for users to select products using voice commands or gesture recognition" refers to an interface that recognizes the voices and hand movements made by users toward the terminal and allows them to select products according to those instructions.

[0702] "Means for the server to manage product information" refers to a function that stores information such as product names, descriptions, prices, and stock status in a database, and allows for manipulation, updating, and retrieval as needed.

[0703] "Means for the terminal to display product information to the user through voice commands or gesture recognition" refers to a function that interprets the user's voice instructions or gestures and displays corresponding product information on the screen.

[0704] "Means for the server to add products to the cart based on the user's voice command or gesture" refers to a function in which, when the user issues an instruction to add a product to the cart using voice or gesture, the server receives that information and registers the product in the cart.

[0705] "A means for users to input payment information using voice commands" refers to a function in which a user verbally inputs a credit card number and payment method, which is then recognized and reflected in the system.

[0706] The "means for the server to process voice payments" is a function that obtains payment information based on the user's voice input and uses that information to complete the payment.

[0707] The "means for the server to confirm the order" is a function for officially registering order information and updating inventory information etc. after successful payment.

[0708] This invention is a system that allows users to conduct e-commerce transactions using voice commands and gesture recognition, and is configured as follows: The main hardware consists of a user terminal such as a smartphone or smart glasses, a server, a voice recognition device, and a gesture recognition device. The software uses Google Cloud Speech-to-Text, TensorFlow & OpenCV, MySQL, React, Node.js, and the Stripe API.

[0709] System Configuration

[0710] User device: A smartphone or smart glasses accepts voice commands and gestures and sends the input to the server, using Google Cloud Speech-to-Text for voice recognition and TensorFlow & OpenCV for gesture recognition.

[0711] Server: The server manages products, adds them to the cart, processes payments, and confirms orders. To achieve this, a web interface built with React and Node.js is used. The database is MySQL.

[0712] Payment system: We use the Stripe API for payments, enabling secure voice payments.

[0713] Product Selection

[0714] When a user types a voice command into their device, such as "Find a smartphone case," the device converts the command into text using voice recognition and sends it to the server. The server retrieves product information from a database based on the received text and sends it to the user's device as a web page generated with React. The user can then select a product using voice or gestures.

[0715] Add to Cart

[0716] When a user issues a voice command such as "Add this to cart," the voice recognition system converts it into text and sends a request from the device to the server. The server updates the cart table based on the received product ID and user ID. If the addition is successful, the server generates a success message and sends it to the device.

[0717] Purchase procedure

[0718] When the user issues the voice command "Proceed to checkout," the device sends a request to the server based on the voice command, including the cart contents and user ID. The server processes the request and generates a checkout page. The page includes fields for entering shipping address and payment information, and is sent to the user's device. The user enters this information using voice commands.

[0719] Payment Processing

[0720] When the user issues the voice command "Complete payment," the terminal sends a request containing the collected payment information to the server. The server then sends it to the Stripe API to process the payment. Depending on whether the payment is successful or unsuccessful, the server performs the appropriate processing and notifies the user of the result.

[0721] Order Confirmed

[0722] If the payment is successful, the server adds new order information to the order table and updates the inventory information. Once the order is confirmed, the server generates an order completion page and sends it to the user's device. The user can view the order completion page to confirm that the purchase was successful.

[0723] Specific examples

[0724] For example, the process for a user purchasing a "smartphone case" is as follows: When the user issues a voice command such as "Find a smartphone case," the device converts the command into text and sends it to the server. The server retrieves product information from a database and sends it to the device. The user then instructs the server to "add this to cart," and the server adds the product to the cart. The user then issues a voice command such as "Proceed to checkout," and a checkout page is generated. When the user enters the necessary information and instructs the server to "Complete payment," the server processes the payment and the order is confirmed.

[0725] Prompt Sentence Examples

[0726] "Looking for a smartphone case"

[0727] "Add this to cart"

[0728] "Proceed to purchase"

[0729] "Complete payment"

[0730] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0731] Step 1:

[0732] The user enters the voice command "Find my phone case" into the device.

[0733] Input: The user's voice command.

[0734] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice commands into text.

[0735] Output: The string "Find a phone case".

[0736] Step 2:

[0737] The terminal sends the string to the server.

[0738] Input: User's voice command converted into a string.

[0739] How it works: The device generates an HTTP request and sends it to the server.

[0740] Output: The request data received on the server side.

[0741] Step 3:

[0742] The server retrieves product information from the database and sends it to the user's terminal.

[0743] Input: Search query for the string "smartphone case".

[0744] How it works: The server searches the MySQL database for product information and retrieves the results.

[0745] Output: A list of product information sent to the user's device.

[0746] Step 4:

[0747] The terminal displays the product information to the user.

[0748] Input: A list of product information received from the server.

[0749] Operation: Generate and display a product list page using React in the device browser.

[0750] Output: A product listing page that can be viewed by the user.

[0751] Step 5:

[0752] The user enters the voice command "Add this to cart."

[0753] Input: The user's voice command.

[0754] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[0755] Output: The string "Add this to cart".

[0756] Step 6:

[0757] The device sends a request including the product ID and user ID to the server.

[0758] Input: Voice command converted into text, product ID, and user ID.

[0759] How it works: The device generates an HTTP request and sends it to the server.

[0760] Output: The request data received on the server side.

[0761] Step 7:

[0762] The server updates the cart table and generates a success message.

[0763] Input: Request data including product ID and user ID.

[0764] What happens: The server adds data to the cart table in the MySQL database and generates a success message.

[0765] Output: Sends a success message to the user's terminal.

[0766] Step 8:

[0767] The user enters the voice command "Proceed to checkout."

[0768] Input: The user's voice command.

[0769] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[0770] Output: The string "Proceed to checkout".

[0771] Step 9:

[0772] The device sends a purchase request to the server.

[0773] Input: Voice command converted into text, cart contents, and user ID.

[0774] How it works: The device generates an HTTP request and sends it to the server.

[0775] Output: The request data received on the server side.

[0776] Step 10:

[0777] The server generates a checkout page and sends it to the user's device.

[0778] Input: Request data including cart contents and user ID.

[0779] How it works: The server generates a checkout page, including fields for shipping address and payment information.

[0780] Output: Sends the completed checkout page to the user's device.

[0781] Step 11:

[0782] The user enters the voice command "Complete payment."

[0783] Input: The user's voice command.

[0784] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[0785] Output: The string "Complete payment".

[0786] Step 12:

[0787] The terminal sends the payment information to the server.

[0788] Input: Voice command converted to text and user payment information.

[0789] How it works: The device generates an HTTP request and sends it to the server.

[0790] Output: The request data received on the server side.

[0791] Step 13:

[0792] The server processes the payment and notifies the user of the result.

[0793] Input: Request data, including payment information.

[0794] What happens: Your server calls the Stripe API to process the payment and checks the result.

[0795] Output: Sends a notification message to the user's device based on the payment result.

[0796] Step 14:

[0797] The server confirms the order and generates an order completion page.

[0798] Input: The result of the payment process.

[0799] What it does: The server updates the orders table in the MySQL database and generates the order completion page.

[0800] Output: Sends the completed order completion page to the user's device.

[0801] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0802] The present invention provides a system that recognizes user emotions in an e-commerce system and efficiently performs product selection, cart addition, purchase procedure, payment, and order confirmation based on that information. An embodiment of the present invention will be described below with a specific example in which an emotion engine is combined.

[0803] System Configuration

[0804] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[0805] Product selection and display

[0806] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of product information.

[0807] Add to Cart

[0808] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[0809] Checkout and Emotion Recording

[0810] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, verifies the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[0811] Payment Processing

[0812] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[0813] Order Confirmed

[0814] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[0815] Specific examples

[0816] For example, let's consider the process of a user purchasing a "smartphone case." The user accesses an e-commerce site and displays the product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or expresses joy, it highlights products that the user is likely to be particularly interested in. When the user clicks on "smartphone case" and displays the detail page, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[0817] This invention uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works in tandem, allowing users to purchase products intuitively and efficiently.

[0818] The processing flow will be explained below.

[0819] Step 1:

[0820] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[0821] Step 2:

[0822] The server generates HTML data for the homepage and sends it to the device. The device receives this HTML data and displays it in the browser. At this time, the emotion engine is activated and analyzes the user's facial expressions and voice using the device's camera and microphone.

[0823] Step 3:

[0824] A user accesses a product list using a product category or search bar. The device sends a request to the server containing the selected category and search keywords.

[0825] Step 4:

[0826] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser. At this time, the emotion engine analyzes the user's reaction and optimizes the display order of the products according to their emotions.

[0827] Step 5:

[0828] A user clicks on a specific product link to view more information about it. The device detects the click event and sends a request including the product ID to the server. The emotion engine analyzes whether the user has a positive reaction to the product.

[0829] Step 6:

[0830] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser. The emotion engine analyzes the user's level of interest and suggests supplementary information or approaches as needed.

[0831] Step 7:

[0832] The user clicks the "Add to Cart" button. The device detects this action and sends a request including the product ID and user ID to the server. The emotion engine records the user's emotion when adding the item to the cart.

[0833] Step 8:

[0834] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser. If the emotion engine detects a positive reaction when the user adds a product to the cart, it displays a message to express joy.

[0835] Step 9:

[0836] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server including the current cart contents and the user ID. The emotion engine records the user's emotions as they proceed to checkout.

[0837] Step 10:

[0838] The server retrieves the user's cart contents from the cart table, generates the HTML for the checkout page, and sends it to the terminal. The terminal receives this HTML and displays the checkout page in the browser. The emotion engine analyzes the user's emotions and adjusts the interface to make the checkout process smoother.

[0839] Step 11:

[0840] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[0841] Step 12:

[0842] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful. The emotion engine records the user's emotion when making a payment.

[0843] Step 13:

[0844] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion page indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser. The emotion engine analyzes whether the user is satisfied when the order is completed, and displays an appropriate feedback message.

[0845] Through these steps, users can select products, add them to their cart, and complete the purchase process. By using the emotion engine, the purchasing experience can be customized to match the user's emotions, improving the purchasing experience.

[0846] Example 2

[0847] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0848] Conventional e-commerce systems lack the ability to recognize user emotions and optimize the purchasing process, which can lead to lower user satisfaction and purchasing motivation. They also fail to provide appropriate information, especially in difficult situations. Furthermore, they lack a mechanism for utilizing user emotion data to recommend future products.

[0849] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing a user's emotion, a means for optimizing product display based on emotion data, and a means for an emotion engine to record the user's emotion. This makes it possible to analyze the user's emotion in real time and optimize the display of product information and the content of messages, thereby improving user satisfaction and purchasing motivation. Furthermore, recording emotion data can improve the accuracy of future product recommendations.

[0850] "User" refers to a person who uses the system to select, view, and purchase products.

[0851] "Server" refers to a computer system that manages product information, processes data, optimizes emotional data, processes payments, and confirms orders.

[0852] A "terminal" refers to a device used by a user, such as a computer or smartphone, that displays product information and recognizes emotions.

[0853] "Product Information" refers to data including information such as product name, price, description, image, and availability.

[0854] A "cart" refers to a virtual shopping basket in which a user temporarily stores products that they wish to purchase.

[0855] "Payment Information" refers to information necessary to make a payment, such as credit card number, payment method, and billing address.

[0856] "Payment" refers to the process of paying for products using payment information entered by a user.

[0857] An "emotion engine" refers to software or hardware that uses the device's camera and microphone to analyze the user's facial expressions and voice and recognize the user's emotions.

[0858] "Emotion data" refers to data relating to a user's emotions recognized by the emotion engine.

[0859] "Product display optimization" refers to the process of dynamically adjusting the selection and order of products to be displayed, the content of messages, etc., based on user emotional data.

[0860] "Emotion recording" refers to the process of storing a user's emotional data in a database or other location and using it for later analysis or product recommendations.

[0861] The present invention is a system for recognizing a user's emotions in an e-commerce system, and based on this information, efficiently performs product selection, adding to cart, purchasing procedures, payment, and order confirmation. Specific embodiments of this system are described below.

[0862] System Configuration

[0863] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[0864] Product selection and display

[0865] When a user browses for a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of the product information.

[0866] Add to Cart

[0867] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[0868] Checkout and Emotion Recording

[0869] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[0870] Payment Processing

[0871] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[0872] Order Confirmed

[0873] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[0874] Specific examples

[0875] For example, let's consider the process of a user purchasing a smartphone case. The user accesses an e-commerce site and displays a product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or happy, it highlights products that the user is likely to be particularly interested in. When the user clicks on a smartphone case and the detail page is displayed, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[0876] This system uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works together, allowing users to purchase products intuitively and efficiently.

[0877] Prompt Sentence Examples

[0878] "We would like you to design a system that recognizes emotions by analyzing facial expressions and voice using a camera and microphone while a user is viewing a specific product, and optimizes the product display."

[0879] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0880] Product selection and display

[0881] Step 1:

[0882] The user accesses the e-commerce site on the device.

[0883] Input: The user enters the URL of the e-commerce site into a web browser.

[0884] How it works: The browser generates and sends a request to the server.

[0885] Output: A request to access the server.

[0886] Step 2:

[0887] The server retrieves product information from the database

[0888] Input: Access request and user request.

[0889] Operation: The server queries the database to retrieve product information.

[0890] Output: Retrieved product information data.

[0891] Step 3:

[0892] The server generates an HTML web page

[0893] Input: Retrieved product information data.

[0894] How it works: The server generates HTML code based on product information.

[0895] Output: The generated HTML code.

[0896] Step 4:

[0897] The server generates a web page and sends it to the user's device.

[0898] Input: The generated HTML code.

[0899] What it does: The server sends an HTTP response containing HTML code.

[0900] Output: HTML response to the user's device.

[0901] Step 5:

[0902] The page received by the user's device is displayed in the browser.

[0903] Input: The received HTML code.

[0904] What happens: The browser renders the HTML code and displays the product listing page.

[0905] Output: A product listing page viewable by the user.

[0906] Step 6:

[0907] The emotion engine uses the device's camera and microphone to recognize the user's emotions.

[0908] Input: Camera video and microphone audio.

[0909] How it works: The emotion engine uses video and audio analysis algorithms to analyze the user's emotions.

[0910] Output: Recognized emotion data.

[0911] Step 7:

[0912] The server optimizes product display based on emotional data

[0913] Input: Recognized emotion data.

[0914] How it works: The server analyzes the emotion data and sorts and highlights products based on the user's interests.

[0915] Output: Optimized product display data.

[0916] Add to Cart

[0917] Step 1:

[0918] User clicks the "Add to Cart" button

[0919] Input: User clicks.

[0920] Behavior: The browser triggers a click event and begins processing.

[0921] Output: The request data including the product ID and user ID.

[0922] Step 2:

[0923] The device sends a request to the server containing the product ID and user ID.

[0924] Input: Request data including product ID and user ID.

[0925] What happens: The browser sends the request data to the server.

[0926] Output: The request to the server.

[0927] Step 3:

[0928] The server receives the request and adds the information to the cart table.

[0929] Input: Request data to the server.

[0930] Operation: The server executes an operation to add the product ID and user ID to the cart table.

[0931] Output: The data added to the cart table.

[0932] Step 4:

[0933] The server generates a success message and sends it to the user's device.

[0934] Input: Result of adding data to the cart table.

[0935] What happens: The server generates a success message and sends it as an HTTP response.

[0936] Output: A success message to the user's terminal.

[0937] Step 5:

[0938] Emotion engine displays encouraging and humorous messages

[0939] Input: User's emotional data and operation details.

[0940] How it works: The emotion engine generates and displays messages based on the user's emotions.

[0941] Output: Encouraging and humorous messages.

[0942] Checkout and Emotion Recording

[0943] Step 1:

[0944] The user checks the contents of the cart and presses the "Proceed to checkout" button

[0945] Input: User clicks.

[0946] Behavior: The browser triggers a click event and begins processing.

[0947] Output: Request data including cart contents and user ID.

[0948] Step 2:

[0949] The device sends a request to the server containing the cart contents and the user ID.

[0950] Input: Request data including cart contents and user ID.

[0951] What happens: The browser sends the request data to the server.

[0952] Output: The request to the server.

[0953] Step 3:

[0954] The server receives the request and generates the checkout page

[0955] Input: Request data to the server.

[0956] How it works: The server parses the request data and generates a checkout page.

[0957] Output: HTML code for the checkout page.

[0958] Step 4:

[0959] Emotion engine records user emotions

[0960] Input: User emotion data during checkout.

[0961] How it works: The emotion engine analyzes the user's emotions and stores the data in a database.

[0962] Output: Saved emotion data.

[0963] Payment Processing

[0964] Step 1:

[0965] The user enters payment information and presses the "Complete payment" button

[0966] Input: When a user enters payment information and clicks a button.

[0967] Behavior: The browser triggers a click event and begins processing.

[0968] Output: The payment information entered.

[0969] Step 2:

[0970] The device sends the entered information to the server.

[0971] Input: Payment information entered.

[0972] How it works: The browser sends payment information to the server.

[0973] Output: Payment information request to server.

[0974] Step 3:

[0975] The server sends the data to the payment service and processes the payment.

[0976] Input: A request to the server for payment information.

[0977] What happens: The server sends the information to a payment service (e.g., a payment gateway) to execute the payment.

[0978] Output: Processing result from payment service.

[0979] Step 4:

[0980] The server checks whether the payment was successful and executes the process.

[0981] Input: Processing result from payment service.

[0982] Operation: The server checks whether the payment was successful or not, and performs the necessary next steps. For example, if successful, the order is confirmed, and if unsuccessful, an error message is generated.

[0983] Output: The processing result of the payment, whether successful or unsuccessful.

[0984] Order Confirmed

[0985] Step 1:

[0986] The server adds the order information to the orders table

[0987] Input: Payment successful processing result and order information.

[0988] Action: The server adds the order information to the orders table.

[0989] Output: The order information added to the orders table.

[0990] Step 2:

[0991] The server updates the inventory information

[0992] Input: Inventory information that needs to be updated.

[0993] Action: The server updates the inventory information in the database.

[0994] Output: Updated inventory information.

[0995] Step 3:

[0996] The server generates an order completion page and sends it to the user's device.

[0997] Input: Order and inventory information.

[0998] How it works: The server generates the HTML code for the order completion page and sends it to the terminal.

[0999] Output: HTML code for order completion page.

[1000] Step 4:

[1001] The user's device displays the order completion page in the browser.

[1002] Input: HTML code for order completion page.

[1003] What happens: The browser renders the HTML code and displays the order completion page.

[1004] Output: A user-viewable order completion page.

[1005] (Application example 2)

[1006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1007] In modern e-commerce systems, there is a lack of emotionally-driven product recommendations and prompts to increase purchasing motivation when users make purchases. This leads to a suboptimal user experience, which can lead to a decline in purchasing motivation and a decrease in sales.

[1008] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on the emotions, means for collecting the user's facial expressions and voice via a camera and microphone, and means for adjusting the product recommendation method using the collected user emotion data. This makes it possible to recommend products and display messages optimally in accordance with the user's emotions.

[1009] "Means for user product selection" refers to an interface (website or application) that allows a user to select products online and designate specific products for purchase.

[1010] "Means for the server to manage product information" refers to a server or database system for storing, updating, deleting, and searching product data.

[1011] "Means by which a terminal displays product information to a user" refers to software or applications for displaying product information on a user's device (such as a smartphone, tablet, or computer).

[1012] "Means for the server to add products to the cart" refers to the server's function of adding products selected by the user to the shopping cart.

[1013] "Means for users to enter payment information" refers to the interface (form, input field, etc.) through which users enter the payment information required for purchase.

[1014] "Means by which the server processes payments" refers to the server's ability to actually process payments using the provided payment information.

[1015] "Means for the server to confirm the order" refers to the server's function of officially confirming the order after the payment is completed, and recording and processing the information.

[1016] "Means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on those emotions" refers to a system that analyzes a user's facial expressions and voice data collected using a camera and microphone, understands the user's emotional state, and displays product recommendations and customized messages based on that.

[1017] "Means for collecting the user's facial expressions and voice via a camera and microphone" refers to a camera and microphone device for acquiring the user's facial expressions and voice data in real time.

[1018] "Means for adjusting the product recommendation method using collected user emotional data" refers to a system that dynamically adjusts the product recommendation algorithm based on the user's emotional data and presents products that are optimal for the user's emotional state.

[1019] This invention provides a system for recognizing a user's emotions and efficiently selecting products, adding them to a cart, completing purchases, making payments, and finalizing orders within an e-commerce system. The system includes a server, a user terminal, an emotion recognition engine, a camera, and a microphone. A specific embodiment of this system will be described below.

[1020] The server uses an emotion recognition device to analyze the user's emotions and display product recommendations and messages based on those emotions. It also collects the user's facial expressions and voice via a camera and microphone, and adjusts the product recommendation method based on the collected data.

[1021] The device is a device that allows users to browse and select products. It can be a smartphone, tablet, or computer. The device's camera and microphone are used to collect user emotion data, which is then sent to a server. The server then uses this data to run an emotion recognition engine.

[1022] The emotion recognition engine analyzes the user's facial expressions and voice data to identify the user's emotions. This engine uses, for example, the machine learning library Keras and the image processing library OpenCV.

[1023] The server retrieves product information selected by the user from the database and sends it to the terminal. The user uses the terminal to select products and add them to the cart. At this time, the emotion recognition engine analyzes the user's emotions and displays a message to encourage the user to purchase.

[1024] The server then provides a means for the user to enter payment information and processes the payment. If the payment is successful, the server confirms the order information, updates inventory information, and finally generates an order completion page that is displayed on the terminal.

[1025] As a concrete example, imagine a situation where a user is browsing an online shopping site on a smartphone. As the user browses products using the smartphone's camera and microphone, an emotion recognition engine analyzes positive emotions such as joy and excitement from the user's facial expressions. Based on this emotional data, the server recommends the most suitable products for the user and displays messages to further encourage them to purchase.

[1026] An example of a prompt sentence when using a generative AI model is as follows:

[1027] "What products should we recommend if the user's emotions indicate delight or surprise?"

[1028] "If a user is expressing anger or sadness, what message can we show them to encourage them to buy?"

[1029] "Generate a blueprint for a real-time product recommendation system based on sentiment data."

[1030] In this way, by analyzing the user's emotional data in real time and using the results to recommend products and display messages, it is possible to significantly improve the user's purchasing experience.

[1031] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1032] Step 1:

[1033] The device uses a camera and microphone to record the user's facial expressions and voice in real time. The input is video and audio data from the camera and microphone. The output is the recorded video and audio data of the user. This data is used in the subsequent emotion recognition engine.

[1034] Step 2:

[1035] The terminal sends the collected user video and audio data to the server. The input is the user video and audio data recorded in step 1. The output is the video and audio data sent to the server.

[1036] Step 3:

[1037] The server inputs the received video and audio data into the emotion recognition engine to analyze the user's emotions. The input is the video and audio data sent to the server. The emotion recognition engine analyzes the data using OpenCV and Keras to identify the user's emotions (happiness, sadness, surprise, etc.). The output is the analyzed user's emotion data.

[1038] Step 4:

[1039] The server executes a process to recommend products based on the analyzed emotional data. The input is the emotional data obtained in step 3. The server uses a generative AI model to select the most suitable products based on the emotion. The output is a list of recommended products.

[1040] Step 5:

[1041] The server sends the list of recommended products to the terminal. The input is the list of recommended products generated in step 4. The output is the list of products sent to the terminal.

[1042] Step 6:

[1043] The terminal displays the received product list to the user. The input is the list of recommended products sent from the server. The output is the product information displayed to the user. Along with this display, the terminal also displays a message to encourage the user to purchase.

[1044] Step 7:

[1045] The user selects a product and clicks the "Add to Cart" button. The input is the operation data based on the user's selection. The output is the product information "added to cart."

[1046] Step 8:

[1047] The server adds items to the cart and updates their status. The input is the item selection data sent by the user. The server adds the user ID and item ID to the cart table and notifies the user that the cart has been updated. The output is the updated cart information.

[1048] Step 9:

[1049] The user clicks the "Proceed to Checkout" button and enters payment information. The input is the payment information the user enters into the terminal. The output is the payment information sent to the server.

[1050] Step 10:

[1051] The server receives payment information and processes the payment. The input is the payment information entered by the user. The server sends a request to the payment service to check whether the payment is successful. The output is information on whether the payment was successful or not.

[1052] Step 11:

[1053] If the payment is successful, the server confirms the order and updates the inventory information. The input is payment success / failure information and order information. The server adds the information to the order table and updates the inventory information. The output is the confirmed order information.

[1054] Step 12:

[1055] The server generates an order completion page and sends it to the terminal. The input is the confirmed order information. The terminal receives this page and displays it to the user. The output is the order completion page displayed to the user.

[1056] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1057] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1058] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1059] [Third embodiment]

[1060] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1061] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1062] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1063] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1064] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1065] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1066] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1067] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1068] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1069] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1070] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1071] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1072] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. An embodiment of the present invention will be described below with specific examples.

[1073] System Configuration

[1074] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[1075] Product Selection

[1076] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[1077] Add to Cart

[1078] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[1079] Purchase procedure

[1080] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[1081] Payment Processing

[1082] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[1083] Order Confirmed

[1084] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[1085] Specific examples

[1086] For example, the process for a user purchasing a "smartphone case" is described as follows: The user accesses an e-commerce site and displays the product list page. The server obtains product information, generates a list page, and sends it to the user's device. The user clicks "smartphone case" to display detailed information. The server obtains the detailed information, generates a detail page, and sends it to the device. When the user clicks "Add to cart," the server adds the product information to the cart. When the user checks the cart and clicks "Proceed to checkout," the server generates a checkout page. When the user enters payment information and clicks "Complete payment," the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[1087] As described above, the system of the present invention allows users to purchase products efficiently and quickly. Because the entire system operates in cooperation with one another, user operation is simplified and the purchasing experience is improved.

[1088] The processing flow will be explained below.

[1089] Step 1:

[1090] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[1091] Step 2:

[1092] The server generates HTML data for the homepage and sends it to the terminal, which receives the HTML data and displays it in its browser.

[1093] Step 3:

[1094] Users use the categories and search bar on the page to access a list of products they are interested in. The device sends a request to the server containing the selected category and search keywords.

[1095] Step 4:

[1096] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser.

[1097] Step 5:

[1098] A user clicks on a specific product link to view more information about that product. The device detects the click event and sends a request to the server that includes the product ID.

[1099] Step 6:

[1100] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser.

[1101] Step 7:

[1102] The user clicks the "Add to Cart" button. The device detects this action and sends a request to the server that includes the product ID and user ID.

[1103] Step 8:

[1104] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser.

[1105] Step 9:

[1106] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server containing the current cart contents and the user ID.

[1107] Step 10:

[1108] The server retrieves the user's cart contents from the cart table, generates an HTML page for the checkout process, and sends it to the terminal. The terminal receives this HTML and displays the checkout process page in the browser.

[1109] Step 11:

[1110] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[1111] Step 12:

[1112] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful.

[1113] Step 13:

[1114] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser.

[1115] Through these steps, users can select products, add them to their cart, and complete the purchase process. The entire system works together to enable users to purchase products intuitively and efficiently.

[1116] Example 1

[1117] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1118] In conventional e-commerce systems, the process of a user selecting a product, adding it to a cart, completing the purchase procedure, and completing the payment often involves multiple, cumbersome steps. Furthermore, if data exchange with the server during the shopping process is not smooth, the user experience can be poor and purchasing motivation can decrease. The present invention aims to solve these problems and provide a system that allows users to purchase products efficiently and quickly.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1120] In this invention, the server includes means for retrieving product information from a database, generating an HTML web page, and sending it to the user's terminal, means for receiving a request including the cart contents and user ID, confirming the product information and user information, and generating a purchase procedure page, and means for sending payment information to the payment service and completing the payment procedure. This allows the user to smoothly and quickly complete operations from product selection to purchase confirmation.

[1121] A "user" is a general user who uses the system to select products and complete the purchase procedure.

[1122] A "server" is a computer system that manages product information, user information, and order information, and performs the necessary data processing in response to various requests.

[1123] "Product information" refers to detailed information stored in the database, such as product name, price, description, and stock quantity.

[1124] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that communicates with a server and displays web pages.

[1125] A "cart" is a temporary data storage space that temporarily stores products that a user is considering purchasing.

[1126] A "database" is a collection of data that stores product information and user information in an organized manner and can be searched and retrieved.

[1127] An "HTML web page" is a document format that is generated by a server, sent to a terminal, and can be viewed by a user using a browser.

[1128] A "product ID" is an identifier that uniquely identifies a product.

[1129] A "user ID" is an identifier that uniquely identifies a user.

[1130] A "web browser" is software for viewing HTML web pages on a terminal.

[1131] A "payment service" is an online service that processes actual payments based on payment information provided by the user.

[1132] "Payment success or failure" refers to whether the payment was successful or failed based on the payment information.

[1133] The "order table" is a part of the database that manages user order information.

[1134] "Stock information" is data that manages the current stock quantity for each product.

[1135] A "purchase procedure page" is a web page where a user enters the shipping address and payment information required to make a purchase.

[1136] The "order completion page" is a web page that allows a user to confirm that the purchase procedure has been completed.

[1137] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. The following describes an embodiment of the present invention with specific examples.

[1138] System Configuration

[1139] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[1140] Product Selection

[1141] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[1142] For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone case" from the database and displays that information to the user.

[1143] Add to Cart

[1144] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[1145] For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID and user ID of the "blue smartphone case" to the server, and the server adds it to the cart.

[1146] Purchase procedure

[1147] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[1148] For example, when a user clicks "Proceed to Checkout," the server generates a checkout page and displays a form for entering a shipping address and payment method.

[1149] Payment Processing

[1150] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[1151] For example, when a user enters credit card information and clicks "Complete Payment," the server sends that information to the payment service and, if the payment is successful, returns a success message.

[1152] Order Confirmed

[1153] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[1154] For example, when a user confirms the purchase of a "blue smartphone case," the server updates the order information, reduces the inventory quantity, and generates and sends an order completion page.

[1155] This allows users to purchase products efficiently and quickly. Because the entire system works in cooperation with one another, user operation is simple and the purchasing experience is improved.

[1156] Examples of prompt statements

[1157] Below is an example of a prompt to input to a generative AI model:

[1158] "Please explain the process from when a user selects a product on an e-commerce site to when they complete the purchase process. Please provide a detailed explanation, including each step: selecting a product, adding it to the cart, paying, and confirming the order. Please also use a 'blue smartphone case' as an example of a specific product."

[1159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1160] Step 1:

[1161] The user browses products on the device and opens the product selection page.

[1162] Input: The user enters the URL of the e-commerce site in a web browser and accesses it.

[1163] Data processing: The server receives the request, retrieves product information from the database, and generates an HTML web page.

[1164] Output: The server sends the generated HTML page to the user's terminal.

[1165] Specific operation: For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone cases" from the database and generates an HTML page.

[1166] Step 2:

[1167] The terminal displays the received page in a browser, and the user browses the product list.

[1168] Input: The HTML page sent by the server.

[1169] Data processing: The device renders the HTML page in a web browser.

[1170] Output: Product list page displayed in browser.

[1171] Specific operation: The user checks the list of "smartphone cases" in the browser.

[1172] Step 3:

[1173] A user adds a specific product to their cart.

[1174] Input: User clicks the "Add to Cart" button.

[1175] Data processing: The terminal sends a request including the product ID and user ID to the server.

[1176] Output: The request sent from the terminal to the server.

[1177] Specific operation: For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID of the "blue smartphone case" and the user ID to the server.

[1178] Step 4:

[1179] The server updates the cart information.

[1180] Input: A request sent from the device containing the product ID and user ID.

[1181] Data processing: The server receives the request and adds the user ID and product ID to the cart table.

[1182] Output: A success message is sent from the server to the device.

[1183] Specific operation: The server receives the request and adds the product information to the cart. If the addition is successful, the server generates a success message and sends it to the user's device.

[1184] Step 5:

[1185] The user checks the contents of the cart and clicks the "Proceed to checkout" button.

[1186] Input: The user clicks the "Proceed to Checkout" button.

[1187] Data processing: The terminal sends a request including the cart contents and the user ID to the server.

[1188] Output: The request sent from the terminal to the server.

[1189] Specific operation: The user presses the "Proceed to checkout" button on the cart page.

[1190] Step 6:

[1191] The server generates the checkout page.

[1192] Input: A request sent from the device containing the cart contents and user ID.

[1193] Data processing: The server checks the product information and user information and generates a purchase procedure page.

[1194] Output: The generated checkout page is sent from the server to the device.

[1195] Specific operation: Based on the acquired information, the server generates a purchase procedure page where the user can enter their shipping address and payment information, and sends it to the user's terminal.

[1196] Step 7:

[1197] The user enters payment information and clicks the "Complete Payment" button.

[1198] Enter: The user fills out the payment form and clicks the "Complete Payment" button.

[1199] Data processing: The terminal sends a request to the server containing the entered payment information.

[1200] Output: The request sent from the terminal to the server.

[1201] Specific operation: The user enters credit card information and clicks "Complete payment."

[1202] Step 8:

[1203] The server processes the payment.

[1204] Input: The request sent from the terminal containing payment information.

[1205] Data processing: The server sends the payment information received to the payment service and processes the payment.

[1206] Output: Receives a response from the payment service and sends a success / failure message from the server to the terminal.

[1207] Specific operation: The server sends payment information to the payment service, and if the payment is successful, it generates a success message and sends it to the user terminal.

[1208] Step 9:

[1209] The server confirms the order and generates an order completion page.

[1210] Input: Response from the payment service that the payment was successful.

[1211] Data processing: The server adds order information to the order table and updates inventory information.

[1212] Output: Order completion page sent from server to terminal.

[1213] Specific operations: The server updates the order information, reduces the stock quantity, generates an order completion page and sends it to the user terminal.

[1214] (Application example 1)

[1215] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1216] In conventional e-commerce systems, the processes of product selection, adding items to a cart, purchasing procedures, payment, and order confirmation are often time-consuming for users. In particular, when using smart devices, accessibility is limited because only a visual interface is used. Furthermore, voice command and gesture recognition technologies are not fully utilized, creating a need for improved user experience. This creates challenges for all users, including the elderly and people with disabilities, in making product purchases easily and quickly.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1218] In this invention, the server includes a means for a user to select a product using a voice command or gesture recognition, a means for the server to manage product information, a means for a terminal to display product information to the user using voice commands or gesture recognition, a means for the server to add products to a cart based on the user's voice command or gesture, a means for the user to input payment information using a voice command, a means for the server to process voice payments, and a means for the server to confirm the order. This allows users to easily and quickly complete the process from product selection to purchase procedures by utilizing voice and gesture recognition. By improving the user experience and providing an operation method that does not rely on vision, an e-commerce system that is easy for all users to use is realized.

[1219] "Means for users to select products using voice commands or gesture recognition" refers to an interface that recognizes the voices and hand movements made by users toward the terminal and allows them to select products according to those instructions.

[1220] "Means for the server to manage product information" refers to a function that stores information such as product names, descriptions, prices, and stock status in a database, and allows for manipulation, updating, and retrieval as needed.

[1221] "Means for the terminal to display product information to the user through voice commands or gesture recognition" refers to a function that interprets the user's voice instructions or gestures and displays corresponding product information on the screen.

[1222] "Means for the server to add products to the cart based on the user's voice command or gesture" refers to a function in which, when the user issues an instruction to add a product to the cart using voice or gesture, the server receives that information and registers the product in the cart.

[1223] "A means for users to input payment information using voice commands" refers to a function in which a user verbally inputs a credit card number and payment method, which is then recognized and reflected in the system.

[1224] The "means for the server to process voice payments" is a function that obtains payment information based on the user's voice input and uses that information to complete the payment.

[1225] The "means for the server to confirm the order" is a function for officially registering order information and updating inventory information etc. after successful payment.

[1226] This invention is a system that allows users to conduct e-commerce transactions using voice commands and gesture recognition, and is configured as follows: The main hardware consists of a user terminal such as a smartphone or smart glasses, a server, a voice recognition device, and a gesture recognition device. The software uses Google Cloud Speech-to-Text, TensorFlow & OpenCV, MySQL, React, Node.js, and the Stripe API.

[1227] System Configuration

[1228] User device: A smartphone or smart glasses accepts voice commands and gestures and sends the input to the server, using Google Cloud Speech-to-Text for voice recognition and TensorFlow & OpenCV for gesture recognition.

[1229] Server: The server manages products, adds them to the cart, processes payments, and confirms orders. To achieve this, a web interface built with React and Node.js is used. The database is MySQL.

[1230] Payment system: We use the Stripe API for payments, enabling secure voice payments.

[1231] Product Selection

[1232] When a user types a voice command into their device, such as "Find a smartphone case," the device converts the command into text using voice recognition and sends it to the server. The server retrieves product information from a database based on the received text and sends it to the user's device as a web page generated with React. The user can then select a product using voice or gestures.

[1233] Add to Cart

[1234] When a user issues a voice command such as "Add this to cart," the voice recognition system converts it into text and sends a request from the device to the server. The server updates the cart table based on the received product ID and user ID. If the addition is successful, the server generates a success message and sends it to the device.

[1235] Purchase procedure

[1236] When the user issues the voice command "Proceed to checkout," the device sends a request to the server based on the voice command, including the cart contents and user ID. The server processes the request and generates a checkout page. The page includes fields for entering shipping address and payment information, and is sent to the user's device. The user enters this information using voice commands.

[1237] Payment Processing

[1238] When the user issues the voice command "Complete payment," the terminal sends a request containing the collected payment information to the server. The server then sends it to the Stripe API to process the payment. Depending on whether the payment is successful or unsuccessful, the server performs the appropriate processing and notifies the user of the result.

[1239] Order Confirmed

[1240] If the payment is successful, the server adds new order information to the order table and updates the inventory information. Once the order is confirmed, the server generates an order completion page and sends it to the user's device. The user can view the order completion page to confirm that the purchase was successful.

[1241] Specific examples

[1242] For example, the process for a user purchasing a "smartphone case" is as follows: When the user issues a voice command such as "Find a smartphone case," the device converts the command into text and sends it to the server. The server retrieves product information from a database and sends it to the device. The user then instructs the server to "add this to cart," and the server adds the product to the cart. The user then issues a voice command such as "Proceed to checkout," and a checkout page is generated. When the user enters the necessary information and instructs the server to "Complete payment," the server processes the payment and the order is confirmed.

[1243] Prompt Sentence Examples

[1244] "Looking for a smartphone case"

[1245] "Add this to cart"

[1246] "Proceed to purchase"

[1247] "Complete payment"

[1248] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1249] Step 1:

[1250] The user enters the voice command "Find my phone case" into the device.

[1251] Input: The user's voice command.

[1252] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice commands into text.

[1253] Output: The string "Find a phone case".

[1254] Step 2:

[1255] The terminal sends the string to the server.

[1256] Input: User's voice command converted into a string.

[1257] How it works: The device generates an HTTP request and sends it to the server.

[1258] Output: The request data received on the server side.

[1259] Step 3:

[1260] The server retrieves product information from the database and sends it to the user's terminal.

[1261] Input: Search query for the string "smartphone case".

[1262] How it works: The server searches the MySQL database for product information and retrieves the results.

[1263] Output: A list of product information sent to the user's device.

[1264] Step 4:

[1265] The terminal displays the product information to the user.

[1266] Input: A list of product information received from the server.

[1267] Operation: Generate and display a product list page using React in the device browser.

[1268] Output: A product listing page that can be viewed by the user.

[1269] Step 5:

[1270] The user enters the voice command "Add this to cart."

[1271] Input: The user's voice command.

[1272] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[1273] Output: The string "Add this to cart".

[1274] Step 6:

[1275] The device sends a request including the product ID and user ID to the server.

[1276] Input: Voice command converted into text, product ID, and user ID.

[1277] How it works: The device generates an HTTP request and sends it to the server.

[1278] Output: The request data received on the server side.

[1279] Step 7:

[1280] The server updates the cart table and generates a success message.

[1281] Input: Request data including product ID and user ID.

[1282] What happens: The server adds data to the cart table in the MySQL database and generates a success message.

[1283] Output: Sends a success message to the user's terminal.

[1284] Step 8:

[1285] The user enters the voice command "Proceed to checkout."

[1286] Input: The user's voice command.

[1287] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[1288] Output: The string "Proceed to checkout".

[1289] Step 9:

[1290] The device sends a purchase request to the server.

[1291] Input: Voice command converted into text, cart contents, and user ID.

[1292] How it works: The device generates an HTTP request and sends it to the server.

[1293] Output: The request data received on the server side.

[1294] Step 10:

[1295] The server generates a checkout page and sends it to the user's device.

[1296] Input: Request data including cart contents and user ID.

[1297] How it works: The server generates a checkout page, including fields for shipping address and payment information.

[1298] Output: Sends the completed checkout page to the user's device.

[1299] Step 11:

[1300] The user enters the voice command "Complete payment."

[1301] Input: The user's voice command.

[1302] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[1303] Output: The string "Complete payment".

[1304] Step 12:

[1305] The terminal sends the payment information to the server.

[1306] Input: Voice command converted to text and user payment information.

[1307] How it works: The device generates an HTTP request and sends it to the server.

[1308] Output: The request data received on the server side.

[1309] Step 13:

[1310] The server processes the payment and notifies the user of the result.

[1311] Input: Request data, including payment information.

[1312] What happens: Your server calls the Stripe API to process the payment and checks the result.

[1313] Output: Sends a notification message to the user's device based on the payment result.

[1314] Step 14:

[1315] The server confirms the order and generates an order completion page.

[1316] Input: The result of the payment process.

[1317] What it does: The server updates the orders table in the MySQL database and generates the order completion page.

[1318] Output: Sends the completed order completion page to the user's device.

[1319] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1320] The present invention provides a system that recognizes user emotions in an e-commerce system and efficiently performs product selection, cart addition, purchase procedure, payment, and order confirmation based on that information. An embodiment of the present invention will be described below with a specific example in which an emotion engine is combined.

[1321] System Configuration

[1322] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[1323] Product selection and display

[1324] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of product information.

[1325] Add to Cart

[1326] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[1327] Checkout and Emotion Recording

[1328] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, verifies the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[1329] Payment Processing

[1330] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[1331] Order Confirmed

[1332] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[1333] Specific examples

[1334] For example, let's consider the process of a user purchasing a "smartphone case." The user accesses an e-commerce site and displays the product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or expresses joy, it highlights products that the user is likely to be particularly interested in. When the user clicks on "smartphone case" and displays the detail page, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[1335] This invention uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works in tandem, allowing users to purchase products intuitively and efficiently.

[1336] The processing flow will be explained below.

[1337] Step 1:

[1338] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[1339] Step 2:

[1340] The server generates HTML data for the homepage and sends it to the device. The device receives this HTML data and displays it in the browser. At this time, the emotion engine is activated and analyzes the user's facial expressions and voice using the device's camera and microphone.

[1341] Step 3:

[1342] A user accesses a product list using a product category or search bar. The device sends a request to the server containing the selected category and search keywords.

[1343] Step 4:

[1344] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser. At this time, the emotion engine analyzes the user's reaction and optimizes the display order of the products according to their emotions.

[1345] Step 5:

[1346] A user clicks on a specific product link to view more information about it. The device detects the click event and sends a request including the product ID to the server. The emotion engine analyzes whether the user has a positive reaction to the product.

[1347] Step 6:

[1348] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser. The emotion engine analyzes the user's level of interest and suggests supplementary information or approaches as needed.

[1349] Step 7:

[1350] The user clicks the "Add to Cart" button. The device detects this action and sends a request including the product ID and user ID to the server. The emotion engine records the user's emotion when adding the item to the cart.

[1351] Step 8:

[1352] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser. If the emotion engine detects a positive reaction when the user adds a product to the cart, it displays a message to express joy.

[1353] Step 9:

[1354] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server including the current cart contents and the user ID. The emotion engine records the user's emotions as they proceed to checkout.

[1355] Step 10:

[1356] The server retrieves the user's cart contents from the cart table, generates the HTML for the checkout page, and sends it to the terminal. The terminal receives this HTML and displays the checkout page in the browser. The emotion engine analyzes the user's emotions and adjusts the interface to make the checkout process smoother.

[1357] Step 11:

[1358] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[1359] Step 12:

[1360] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful. The emotion engine records the user's emotion when making a payment.

[1361] Step 13:

[1362] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion page indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser. The emotion engine analyzes whether the user is satisfied when the order is completed, and displays an appropriate feedback message.

[1363] Through these steps, users can select products, add them to their cart, and complete the purchase process. By using the emotion engine, the purchasing experience can be customized to match the user's emotions, improving the purchasing experience.

[1364] Example 2

[1365] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1366] Conventional e-commerce systems lack the ability to recognize user emotions and optimize the purchasing process, which can lead to lower user satisfaction and purchasing motivation. They also fail to provide appropriate information, especially in difficult situations. Furthermore, they lack a mechanism for utilizing user emotion data to recommend future products.

[1367] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing a user's emotion, a means for optimizing product display based on emotion data, and a means for an emotion engine to record the user's emotion. This makes it possible to analyze the user's emotion in real time and optimize the display of product information and the content of messages, thereby improving user satisfaction and purchasing motivation. Furthermore, recording emotion data can improve the accuracy of future product recommendations.

[1368] "User" refers to a person who uses the system to select, view, and purchase products.

[1369] "Server" refers to a computer system that manages product information, processes data, optimizes emotional data, processes payments, and confirms orders.

[1370] A "terminal" refers to a device used by a user, such as a computer or smartphone, that displays product information and recognizes emotions.

[1371] "Product Information" refers to data including information such as product name, price, description, image, and availability.

[1372] A "cart" refers to a virtual shopping basket in which a user temporarily stores products that they wish to purchase.

[1373] "Payment Information" refers to information necessary to make a payment, such as credit card number, payment method, and billing address.

[1374] "Payment" refers to the process of paying for products using payment information entered by a user.

[1375] An "emotion engine" refers to software or hardware that uses the device's camera and microphone to analyze the user's facial expressions and voice and recognize the user's emotions.

[1376] "Emotion data" refers to data relating to a user's emotions recognized by the emotion engine.

[1377] "Product display optimization" refers to the process of dynamically adjusting the selection and order of products to be displayed, the content of messages, etc., based on user emotional data.

[1378] "Emotion recording" refers to the process of storing a user's emotional data in a database or other location and using it for later analysis or product recommendations.

[1379] The present invention is a system for recognizing a user's emotions in an e-commerce system, and based on this information, efficiently performs product selection, adding to cart, purchasing procedures, payment, and order confirmation. Specific embodiments of this system are described below.

[1380] System Configuration

[1381] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[1382] Product selection and display

[1383] When a user browses for a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of the product information.

[1384] Add to Cart

[1385] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[1386] Checkout and Emotion Recording

[1387] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[1388] Payment Processing

[1389] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[1390] Order Confirmed

[1391] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[1392] Specific examples

[1393] For example, let's consider the process of a user purchasing a smartphone case. The user accesses an e-commerce site and displays a product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or happy, it highlights products that the user is likely to be particularly interested in. When the user clicks on a smartphone case and the detail page is displayed, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[1394] This system uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works together, allowing users to purchase products intuitively and efficiently.

[1395] Prompt Sentence Examples

[1396] "We would like you to design a system that recognizes emotions by analyzing facial expressions and voice using a camera and microphone while a user is viewing a specific product, and optimizes the product display."

[1397] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1398] Product selection and display

[1399] Step 1:

[1400] The user accesses the e-commerce site on the device.

[1401] Input: The user enters the URL of the e-commerce site into a web browser.

[1402] How it works: The browser generates and sends a request to the server.

[1403] Output: A request to access the server.

[1404] Step 2:

[1405] The server retrieves product information from the database

[1406] Input: Access request and user request.

[1407] Operation: The server queries the database to retrieve product information.

[1408] Output: Retrieved product information data.

[1409] Step 3:

[1410] The server generates an HTML web page

[1411] Input: Retrieved product information data.

[1412] How it works: The server generates HTML code based on product information.

[1413] Output: The generated HTML code.

[1414] Step 4:

[1415] The server generates a web page and sends it to the user's device.

[1416] Input: The generated HTML code.

[1417] What it does: The server sends an HTTP response containing HTML code.

[1418] Output: HTML response to the user's device.

[1419] Step 5:

[1420] The page received by the user's device is displayed in the browser.

[1421] Input: The received HTML code.

[1422] What happens: The browser renders the HTML code and displays the product listing page.

[1423] Output: A product listing page viewable by the user.

[1424] Step 6:

[1425] The emotion engine uses the device's camera and microphone to recognize the user's emotions.

[1426] Input: Camera video and microphone audio.

[1427] How it works: The emotion engine uses video and audio analysis algorithms to analyze the user's emotions.

[1428] Output: Recognized emotion data.

[1429] Step 7:

[1430] The server optimizes product display based on emotional data

[1431] Input: Recognized emotion data.

[1432] How it works: The server analyzes the emotion data and sorts and highlights products based on the user's interests.

[1433] Output: Optimized product display data.

[1434] Add to Cart

[1435] Step 1:

[1436] User clicks the "Add to Cart" button

[1437] Input: User clicks.

[1438] Behavior: The browser triggers a click event and begins processing.

[1439] Output: The request data including the product ID and user ID.

[1440] Step 2:

[1441] The device sends a request to the server containing the product ID and user ID.

[1442] Input: Request data including product ID and user ID.

[1443] What happens: The browser sends the request data to the server.

[1444] Output: The request to the server.

[1445] Step 3:

[1446] The server receives the request and adds the information to the cart table.

[1447] Input: Request data to the server.

[1448] Operation: The server executes an operation to add the product ID and user ID to the cart table.

[1449] Output: The data added to the cart table.

[1450] Step 4:

[1451] The server generates a success message and sends it to the user's device.

[1452] Input: Result of adding data to the cart table.

[1453] What happens: The server generates a success message and sends it as an HTTP response.

[1454] Output: A success message to the user's terminal.

[1455] Step 5:

[1456] Emotion engine displays encouraging and humorous messages

[1457] Input: User's emotional data and operation details.

[1458] How it works: The emotion engine generates and displays messages based on the user's emotions.

[1459] Output: Encouraging and humorous messages.

[1460] Checkout and Emotion Recording

[1461] Step 1:

[1462] The user checks the contents of the cart and presses the "Proceed to checkout" button

[1463] Input: User clicks.

[1464] Behavior: The browser triggers a click event and begins processing.

[1465] Output: Request data including cart contents and user ID.

[1466] Step 2:

[1467] The device sends a request to the server containing the cart contents and the user ID.

[1468] Input: Request data including cart contents and user ID.

[1469] What happens: The browser sends the request data to the server.

[1470] Output: The request to the server.

[1471] Step 3:

[1472] The server receives the request and generates the checkout page

[1473] Input: Request data to the server.

[1474] How it works: The server parses the request data and generates a checkout page.

[1475] Output: HTML code for the checkout page.

[1476] Step 4:

[1477] Emotion engine records user emotions

[1478] Input: User emotion data during checkout.

[1479] How it works: The emotion engine analyzes the user's emotions and stores the data in a database.

[1480] Output: Saved emotion data.

[1481] Payment Processing

[1482] Step 1:

[1483] The user enters payment information and presses the "Complete payment" button

[1484] Input: When a user enters payment information and clicks a button.

[1485] Behavior: The browser triggers a click event and begins processing.

[1486] Output: The payment information entered.

[1487] Step 2:

[1488] The device sends the entered information to the server.

[1489] Input: Payment information entered.

[1490] How it works: The browser sends payment information to the server.

[1491] Output: Payment information request to server.

[1492] Step 3:

[1493] The server sends the data to the payment service and processes the payment.

[1494] Input: A request to the server for payment information.

[1495] What happens: The server sends the information to a payment service (e.g., a payment gateway) to execute the payment.

[1496] Output: Processing result from payment service.

[1497] Step 4:

[1498] The server checks whether the payment was successful and executes the process.

[1499] Input: Processing result from payment service.

[1500] Operation: The server checks whether the payment was successful or not, and performs the necessary next steps. For example, if successful, the order is confirmed, and if unsuccessful, an error message is generated.

[1501] Output: The processing result of the payment, whether successful or unsuccessful.

[1502] Order Confirmed

[1503] Step 1:

[1504] The server adds the order information to the orders table

[1505] Input: Payment successful processing result and order information.

[1506] Action: The server adds the order information to the orders table.

[1507] Output: The order information added to the orders table.

[1508] Step 2:

[1509] The server updates the inventory information

[1510] Input: Inventory information that needs to be updated.

[1511] Action: The server updates the inventory information in the database.

[1512] Output: Updated inventory information.

[1513] Step 3:

[1514] The server generates an order completion page and sends it to the user's device.

[1515] Input: Order and inventory information.

[1516] How it works: The server generates the HTML code for the order completion page and sends it to the terminal.

[1517] Output: HTML code for order completion page.

[1518] Step 4:

[1519] The user's device displays the order completion page in the browser.

[1520] Input: HTML code for order completion page.

[1521] What happens: The browser renders the HTML code and displays the order completion page.

[1522] Output: A user-viewable order completion page.

[1523] (Application example 2)

[1524] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1525] In modern e-commerce systems, there is a lack of emotionally-driven product recommendations and prompts to increase purchasing motivation when users make purchases. This leads to a suboptimal user experience, which can lead to a decline in purchasing motivation and a decrease in sales.

[1526] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on the emotions, means for collecting the user's facial expressions and voice via a camera and microphone, and means for adjusting the product recommendation method using the collected user emotion data. This makes it possible to recommend products and display messages optimally in accordance with the user's emotions.

[1527] "Means for user product selection" refers to an interface (website or application) that allows a user to select products online and designate specific products for purchase.

[1528] "Means for the server to manage product information" refers to a server or database system for storing, updating, deleting, and searching product data.

[1529] "Means by which a terminal displays product information to a user" refers to software or applications for displaying product information on a user's device (such as a smartphone, tablet, or computer).

[1530] "Means for the server to add products to the cart" refers to the server's function of adding products selected by the user to the shopping cart.

[1531] "Means for users to enter payment information" refers to the interface (form, input field, etc.) through which users enter the payment information required for purchase.

[1532] "Means by which the server processes payments" refers to the server's ability to actually process payments using the provided payment information.

[1533] "Means for the server to confirm the order" refers to the server's function of officially confirming the order after the payment is completed, and recording and processing the information.

[1534] "Means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on those emotions" refers to a system that analyzes a user's facial expressions and voice data collected using a camera and microphone, understands the user's emotional state, and displays product recommendations and customized messages based on that.

[1535] "Means for collecting the user's facial expressions and voice via a camera and microphone" refers to a camera and microphone device for acquiring the user's facial expressions and voice data in real time.

[1536] "Means for adjusting the product recommendation method using collected user emotional data" refers to a system that dynamically adjusts the product recommendation algorithm based on the user's emotional data and presents products that are optimal for the user's emotional state.

[1537] This invention provides a system for recognizing a user's emotions and efficiently selecting products, adding them to a cart, completing purchases, making payments, and finalizing orders within an e-commerce system. The system includes a server, a user terminal, an emotion recognition engine, a camera, and a microphone. A specific embodiment of this system will be described below.

[1538] The server uses an emotion recognition device to analyze the user's emotions and display product recommendations and messages based on those emotions. It also collects the user's facial expressions and voice via a camera and microphone, and adjusts the product recommendation method based on the collected data.

[1539] The device is a device that allows users to browse and select products. It can be a smartphone, tablet, or computer. The device's camera and microphone are used to collect user emotion data, which is then sent to a server. The server then uses this data to run an emotion recognition engine.

[1540] The emotion recognition engine analyzes the user's facial expressions and voice data to identify the user's emotions. This engine uses, for example, the machine learning library Keras and the image processing library OpenCV.

[1541] The server retrieves product information selected by the user from the database and sends it to the terminal. The user uses the terminal to select products and add them to the cart. At this time, the emotion recognition engine analyzes the user's emotions and displays a message to encourage the user to purchase.

[1542] The server then provides a means for the user to enter payment information and processes the payment. If the payment is successful, the server confirms the order information, updates inventory information, and finally generates an order completion page that is displayed on the terminal.

[1543] As a concrete example, imagine a situation where a user is browsing an online shopping site on a smartphone. As the user browses products using the smartphone's camera and microphone, an emotion recognition engine analyzes positive emotions such as joy and excitement from the user's facial expressions. Based on this emotional data, the server recommends the most suitable products for the user and displays messages to further encourage them to purchase.

[1544] An example of a prompt sentence when using a generative AI model is as follows:

[1545] "What products should we recommend if the user's emotions indicate delight or surprise?"

[1546] "If a user is expressing anger or sadness, what message can we show them to encourage them to buy?"

[1547] "Generate a blueprint for a real-time product recommendation system based on sentiment data."

[1548] In this way, by analyzing the user's emotional data in real time and using the results to recommend products and display messages, it is possible to significantly improve the user's purchasing experience.

[1549] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1550] Step 1:

[1551] The device uses a camera and microphone to record the user's facial expressions and voice in real time. The input is video and audio data from the camera and microphone. The output is the recorded video and audio data of the user. This data is used in the subsequent emotion recognition engine.

[1552] Step 2:

[1553] The terminal sends the collected user video and audio data to the server. The input is the user video and audio data recorded in step 1. The output is the video and audio data sent to the server.

[1554] Step 3:

[1555] The server inputs the received video and audio data into the emotion recognition engine to analyze the user's emotions. The input is the video and audio data sent to the server. The emotion recognition engine analyzes the data using OpenCV and Keras to identify the user's emotions (happiness, sadness, surprise, etc.). The output is the analyzed user's emotion data.

[1556] Step 4:

[1557] The server executes a process to recommend products based on the analyzed emotional data. The input is the emotional data obtained in step 3. The server uses a generative AI model to select the most suitable products based on the emotion. The output is a list of recommended products.

[1558] Step 5:

[1559] The server sends the list of recommended products to the terminal. The input is the list of recommended products generated in step 4. The output is the list of products sent to the terminal.

[1560] Step 6:

[1561] The terminal displays the received product list to the user. The input is the list of recommended products sent from the server. The output is the product information displayed to the user. Along with this display, the terminal also displays a message to encourage the user to purchase.

[1562] Step 7:

[1563] The user selects a product and clicks the "Add to Cart" button. The input is the operation data based on the user's selection. The output is the product information "added to cart."

[1564] Step 8:

[1565] The server adds items to the cart and updates their status. The input is the item selection data sent by the user. The server adds the user ID and item ID to the cart table and notifies the user that the cart has been updated. The output is the updated cart information.

[1566] Step 9:

[1567] The user clicks the "Proceed to Checkout" button and enters payment information. The input is the payment information the user enters into the terminal. The output is the payment information sent to the server.

[1568] Step 10:

[1569] The server receives payment information and processes the payment. The input is the payment information entered by the user. The server sends a request to the payment service to check whether the payment is successful. The output is information on whether the payment was successful or not.

[1570] Step 11:

[1571] If the payment is successful, the server confirms the order and updates the inventory information. The input is payment success / failure information and order information. The server adds the information to the order table and updates the inventory information. The output is the confirmed order information.

[1572] Step 12:

[1573] The server generates an order completion page and sends it to the terminal. The input is the confirmed order information. The terminal receives this page and displays it to the user. The output is the order completion page displayed to the user.

[1574] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1575] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1576] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1577] [Fourth embodiment]

[1578] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1579] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1580] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1581] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1582] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1583] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1584] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1585] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1586] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1587] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1588] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1589] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1590] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1591] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. An embodiment of the present invention will be described below with specific examples.

[1592] System Configuration

[1593] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[1594] Product Selection

[1595] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[1596] Add to Cart

[1597] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[1598] Purchase procedure

[1599] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[1600] Payment Processing

[1601] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[1602] Order Confirmed

[1603] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[1604] Specific examples

[1605] For example, the process for a user purchasing a "smartphone case" is described as follows: The user accesses an e-commerce site and displays the product list page. The server obtains product information, generates a list page, and sends it to the user's device. The user clicks "smartphone case" to display detailed information. The server obtains the detailed information, generates a detail page, and sends it to the device. When the user clicks "Add to cart," the server adds the product information to the cart. When the user checks the cart and clicks "Proceed to checkout," the server generates a checkout page. When the user enters payment information and clicks "Complete payment," the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[1606] As described above, the system of the present invention allows users to purchase products efficiently and quickly. Because the entire system operates in cooperation with one another, user operation is simplified and the purchasing experience is improved.

[1607] The processing flow will be explained below.

[1608] Step 1:

[1609] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[1610] Step 2:

[1611] The server generates HTML data for the homepage and sends it to the terminal, which receives the HTML data and displays it in its browser.

[1612] Step 3:

[1613] Users use the categories and search bar on the page to access a list of products they are interested in. The device sends a request to the server containing the selected category and search keywords.

[1614] Step 4:

[1615] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser.

[1616] Step 5:

[1617] A user clicks on a specific product link to view more information about that product. The device detects the click event and sends a request to the server that includes the product ID.

[1618] Step 6:

[1619] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser.

[1620] Step 7:

[1621] The user clicks the "Add to Cart" button. The device detects this action and sends a request to the server that includes the product ID and user ID.

[1622] Step 8:

[1623] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser.

[1624] Step 9:

[1625] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server containing the current cart contents and the user ID.

[1626] Step 10:

[1627] The server retrieves the user's cart contents from the cart table, generates an HTML page for the checkout process, and sends it to the terminal. The terminal receives this HTML and displays the checkout process page in the browser.

[1628] Step 11:

[1629] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[1630] Step 12:

[1631] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful.

[1632] Step 13:

[1633] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser.

[1634] Through these steps, users can select products, add them to their cart, and complete the purchase process. The entire system works together to enable users to purchase products intuitively and efficiently.

[1635] Example 1

[1636] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1637] In conventional e-commerce systems, the process of a user selecting a product, adding it to a cart, completing the purchase procedure, and completing the payment often involves multiple, cumbersome steps. Furthermore, if data exchange with the server during the shopping process is not smooth, the user experience can be poor and purchasing motivation can decrease. The present invention aims to solve these problems and provide a system that allows users to purchase products efficiently and quickly.

[1638] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1639] In this invention, the server includes means for retrieving product information from a database, generating an HTML web page, and sending it to the user's terminal, means for receiving a request including the cart contents and user ID, confirming the product information and user information, and generating a purchase procedure page, and means for sending payment information to the payment service and completing the payment procedure. This allows the user to smoothly and quickly complete operations from product selection to purchase confirmation.

[1640] A "user" is a general user who uses the system to select products and complete the purchase procedure.

[1641] A "server" is a computer system that manages product information, user information, and order information, and performs the necessary data processing in response to various requests.

[1642] "Product information" refers to detailed information stored in the database, such as product name, price, description, and stock quantity.

[1643] A "terminal" is a device used by a user, such as a computer, smartphone, or tablet, that communicates with a server and displays web pages.

[1644] A "cart" is a temporary data storage space that temporarily stores products that a user is considering purchasing.

[1645] A "database" is a collection of data that stores product information and user information in an organized manner and can be searched and retrieved.

[1646] An "HTML web page" is a document format that is generated by a server, sent to a terminal, and can be viewed by a user using a browser.

[1647] A "product ID" is an identifier that uniquely identifies a product.

[1648] A "user ID" is an identifier that uniquely identifies a user.

[1649] A "web browser" is software for viewing HTML web pages on a terminal.

[1650] A "payment service" is an online service that processes actual payments based on payment information provided by the user.

[1651] "Payment success or failure" refers to whether the payment was successful or failed based on the payment information.

[1652] The "order table" is a part of the database that manages user order information.

[1653] "Stock information" is data that manages the current stock quantity for each product.

[1654] A "purchase procedure page" is a web page where a user enters the shipping address and payment information required to make a purchase.

[1655] The "order completion page" is a web page that allows a user to confirm that the purchase procedure has been completed.

[1656] The present invention provides a means for efficiently performing product selection, adding to a cart, purchasing procedures, payment, and order confirmation in an electronic commerce system. The following describes an embodiment of the present invention with specific examples.

[1657] System Configuration

[1658] This system consists of a terminal used by the user, a server that manages product information and databases, and a web interface that the user accesses.

[1659] Product Selection

[1660] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device displays the received page in a browser, allowing the user to view the product list and detailed information.

[1661] For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone case" from the database and displays that information to the user.

[1662] Add to Cart

[1663] When a user wants to add a specific product to the cart, the user clicks the "Add to Cart" button on the terminal. This action causes the terminal to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's terminal.

[1664] For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID and user ID of the "blue smartphone case" to the server, and the server adds it to the cart.

[1665] Purchase procedure

[1666] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the terminal sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering the shipping address and payment information. The generated page is sent to the user's terminal, which displays it, and the user enters the required information.

[1667] For example, when a user clicks "Proceed to Checkout," the server generates a checkout page and displays a form for entering a shipping address and payment method.

[1668] Payment Processing

[1669] When a user fills out the payment form and presses the "Complete Payment" button, the terminal sends a request including the payment information to the server. The server then sends the received payment information to the payment service and processes the payment. Upon receiving a response from the payment service, the server checks whether the payment was successful and takes appropriate action accordingly.

[1670] For example, when a user enters credit card information and clicks "Complete Payment," the server sends that information to the payment service and, if the payment is successful, returns a success message.

[1671] Order Confirmed

[1672] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user confirms the order completion page and realizes that the purchase was successful.

[1673] For example, when a user confirms the purchase of a "blue smartphone case," the server updates the order information, reduces the inventory quantity, and generates and sends an order completion page.

[1674] This allows users to purchase products efficiently and quickly. Because the entire system works in cooperation with one another, user operation is simple and the purchasing experience is improved.

[1675] Examples of prompt statements

[1676] Below is an example of a prompt to input to a generative AI model:

[1677] "Please explain the process from when a user selects a product on an e-commerce site to when they complete the purchase process. Please provide a detailed explanation, including each step: selecting a product, adding it to the cart, paying, and confirming the order. Please also use a 'blue smartphone case' as an example of a specific product."

[1678] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1679] Step 1:

[1680] The user browses products on the device and opens the product selection page.

[1681] Input: The user enters the URL of the e-commerce site in a web browser and accesses it.

[1682] Data processing: The server receives the request, retrieves product information from the database, and generates an HTML web page.

[1683] Output: The server sends the generated HTML page to the user's terminal.

[1684] Specific operation: For example, if a user is looking for a "smartphone case," the server retrieves product information related to "smartphone cases" from the database and generates an HTML page.

[1685] Step 2:

[1686] The terminal displays the received page in a browser, and the user browses the product list.

[1687] Input: The HTML page sent by the server.

[1688] Data processing: The device renders the HTML page in a web browser.

[1689] Output: Product list page displayed in browser.

[1690] Specific operation: The user checks the list of "smartphone cases" in the browser.

[1691] Step 3:

[1692] A user adds a specific product to their cart.

[1693] Input: User clicks the "Add to Cart" button.

[1694] Data processing: The terminal sends a request including the product ID and user ID to the server.

[1695] Output: The request sent from the terminal to the server.

[1696] Specific operation: For example, when a user adds a "blue smartphone case" to the cart, the terminal sends the product ID of the "blue smartphone case" and the user ID to the server.

[1697] Step 4:

[1698] The server updates the cart information.

[1699] Input: A request sent from the device containing the product ID and user ID.

[1700] Data processing: The server receives the request and adds the user ID and product ID to the cart table.

[1701] Output: A success message is sent from the server to the device.

[1702] Specific operation: The server receives the request and adds the product information to the cart. If the addition is successful, the server generates a success message and sends it to the user's device.

[1703] Step 5:

[1704] The user checks the contents of the cart and clicks the "Proceed to checkout" button.

[1705] Input: The user clicks the "Proceed to Checkout" button.

[1706] Data processing: The terminal sends a request including the cart contents and the user ID to the server.

[1707] Output: The request sent from the terminal to the server.

[1708] Specific operation: The user presses the "Proceed to checkout" button on the cart page.

[1709] Step 6:

[1710] The server generates the checkout page.

[1711] Input: A request sent from the device containing the cart contents and user ID.

[1712] Data processing: The server checks the product information and user information and generates a purchase procedure page.

[1713] Output: The generated checkout page is sent from the server to the device.

[1714] Specific operation: Based on the acquired information, the server generates a purchase procedure page where the user can enter their shipping address and payment information, and sends it to the user's terminal.

[1715] Step 7:

[1716] The user enters payment information and clicks the "Complete Payment" button.

[1717] Enter: The user fills out the payment form and clicks the "Complete Payment" button.

[1718] Data processing: The terminal sends a request to the server containing the entered payment information.

[1719] Output: The request sent from the terminal to the server.

[1720] Specific operation: The user enters credit card information and clicks "Complete payment."

[1721] Step 8:

[1722] The server processes the payment.

[1723] Input: The request sent from the terminal containing payment information.

[1724] Data processing: The server sends the payment information received to the payment service and processes the payment.

[1725] Output: Receives a response from the payment service and sends a success / failure message from the server to the terminal.

[1726] Specific operation: The server sends payment information to the payment service, and if the payment is successful, it generates a success message and sends it to the user terminal.

[1727] Step 9:

[1728] The server confirms the order and generates an order completion page.

[1729] Input: Response from the payment service that the payment was successful.

[1730] Data processing: The server adds order information to the order table and updates inventory information.

[1731] Output: Order completion page sent from server to terminal.

[1732] Specific operations: The server updates the order information, reduces the stock quantity, generates an order completion page and sends it to the user terminal.

[1733] (Application example 1)

[1734] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1735] In conventional e-commerce systems, the processes of product selection, adding items to a cart, purchasing procedures, payment, and order confirmation are often time-consuming for users. In particular, when using smart devices, accessibility is limited because only a visual interface is used. Furthermore, voice command and gesture recognition technologies are not fully utilized, creating a need for improved user experience. This creates challenges for all users, including the elderly and people with disabilities, in making product purchases easily and quickly.

[1736] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1737] In this invention, the server includes a means for a user to select a product using a voice command or gesture recognition, a means for the server to manage product information, a means for a terminal to display product information to the user using voice commands or gesture recognition, a means for the server to add products to a cart based on the user's voice command or gesture, a means for the user to input payment information using a voice command, a means for the server to process voice payments, and a means for the server to confirm the order. This allows users to easily and quickly complete the process from product selection to purchase procedures by utilizing voice and gesture recognition. By improving the user experience and providing an operation method that does not rely on vision, an e-commerce system that is easy for all users to use is realized.

[1738] "Means for users to select products using voice commands or gesture recognition" refers to an interface that recognizes the voices and hand movements made by users toward the terminal and allows them to select products according to those instructions.

[1739] "Means for the server to manage product information" refers to a function that stores information such as product names, descriptions, prices, and stock status in a database, and allows for manipulation, updating, and retrieval as needed.

[1740] "Means for the terminal to display product information to the user through voice commands or gesture recognition" refers to a function that interprets the user's voice instructions or gestures and displays corresponding product information on the screen.

[1741] "Means for the server to add products to the cart based on the user's voice command or gesture" refers to a function in which, when the user issues an instruction to add a product to the cart using voice or gesture, the server receives that information and registers the product in the cart.

[1742] "A means for users to input payment information using voice commands" refers to a function in which a user verbally inputs a credit card number and payment method, which is then recognized and reflected in the system.

[1743] The "means for the server to process voice payments" is a function that obtains payment information based on the user's voice input and uses that information to complete the payment.

[1744] The "means for the server to confirm the order" is a function for officially registering order information and updating inventory information etc. after successful payment.

[1745] This invention is a system that allows users to conduct e-commerce transactions using voice commands and gesture recognition, and is configured as follows: The main hardware consists of a user terminal such as a smartphone or smart glasses, a server, a voice recognition device, and a gesture recognition device. The software uses Google Cloud Speech-to-Text, TensorFlow & OpenCV, MySQL, React, Node.js, and the Stripe API.

[1746] System Configuration

[1747] User device: A smartphone or smart glasses accepts voice commands and gestures and sends the input to the server, using Google Cloud Speech-to-Text for voice recognition and TensorFlow & OpenCV for gesture recognition.

[1748] Server: The server manages products, adds them to the cart, processes payments, and confirms orders. To achieve this, a web interface built with React and Node.js is used. The database is MySQL.

[1749] Payment system: We use the Stripe API for payments, enabling secure voice payments.

[1750] Product Selection

[1751] When a user types a voice command into their device, such as "Find a smartphone case," the device converts the command into text using voice recognition and sends it to the server. The server retrieves product information from a database based on the received text and sends it to the user's device as a web page generated with React. The user can then select a product using voice or gestures.

[1752] Add to Cart

[1753] When a user issues a voice command such as "Add this to cart," the voice recognition system converts it into text and sends a request from the device to the server. The server updates the cart table based on the received product ID and user ID. If the addition is successful, the server generates a success message and sends it to the device.

[1754] Purchase procedure

[1755] When the user issues the voice command "Proceed to checkout," the device sends a request to the server based on the voice command, including the cart contents and user ID. The server processes the request and generates a checkout page. The page includes fields for entering shipping address and payment information, and is sent to the user's device. The user enters this information using voice commands.

[1756] Payment Processing

[1757] When the user issues the voice command "Complete payment," the terminal sends a request containing the collected payment information to the server. The server then sends it to the Stripe API to process the payment. Depending on whether the payment is successful or unsuccessful, the server performs the appropriate processing and notifies the user of the result.

[1758] Order Confirmed

[1759] If the payment is successful, the server adds new order information to the order table and updates the inventory information. Once the order is confirmed, the server generates an order completion page and sends it to the user's device. The user can view the order completion page to confirm that the purchase was successful.

[1760] Specific examples

[1761] For example, the process for a user purchasing a "smartphone case" is as follows: When the user issues a voice command such as "Find a smartphone case," the device converts the command into text and sends it to the server. The server retrieves product information from a database and sends it to the device. The user then instructs the server to "add this to cart," and the server adds the product to the cart. The user then issues a voice command such as "Proceed to checkout," and a checkout page is generated. When the user enters the necessary information and instructs the server to "Complete payment," the server processes the payment and the order is confirmed.

[1762] Prompt Sentence Examples

[1763] "Looking for a smartphone case"

[1764] "Add this to cart"

[1765] "Proceed to purchase"

[1766] "Complete payment"

[1767] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1768] Step 1:

[1769] The user enters the voice command "Find my phone case" into the device.

[1770] Input: The user's voice command.

[1771] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice commands into text.

[1772] Output: The string "Find a phone case".

[1773] Step 2:

[1774] The terminal sends the string to the server.

[1775] Input: User's voice command converted into a string.

[1776] How it works: The device generates an HTTP request and sends it to the server.

[1777] Output: The request data received on the server side.

[1778] Step 3:

[1779] The server retrieves product information from the database and sends it to the user's terminal.

[1780] Input: Search query for the string "smartphone case".

[1781] How it works: The server searches the MySQL database for product information and retrieves the results.

[1782] Output: A list of product information sent to the user's device.

[1783] Step 4:

[1784] The terminal displays the product information to the user.

[1785] Input: A list of product information received from the server.

[1786] Operation: Generate and display a product list page using React in the device browser.

[1787] Output: A product listing page that can be viewed by the user.

[1788] Step 5:

[1789] The user enters the voice command "Add this to cart."

[1790] Input: The user's voice command.

[1791] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[1792] Output: The string "Add this to cart".

[1793] Step 6:

[1794] The device sends a request including the product ID and user ID to the server.

[1795] Input: Voice command converted into text, product ID, and user ID.

[1796] How it works: The device generates an HTTP request and sends it to the server.

[1797] Output: The request data received on the server side.

[1798] Step 7:

[1799] The server updates the cart table and generates a success message.

[1800] Input: Request data including product ID and user ID.

[1801] What happens: The server adds data to the cart table in the MySQL database and generates a success message.

[1802] Output: Sends a success message to the user's terminal.

[1803] Step 8:

[1804] The user enters the voice command "Proceed to checkout."

[1805] Input: The user's voice command.

[1806] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[1807] Output: The string "Proceed to checkout".

[1808] Step 9:

[1809] The device sends a purchase request to the server.

[1810] Input: Voice command converted into text, cart contents, and user ID.

[1811] How it works: The device generates an HTTP request and sends it to the server.

[1812] Output: The request data received on the server side.

[1813] Step 10:

[1814] The server generates a checkout page and sends it to the user's device.

[1815] Input: Request data including cart contents and user ID.

[1816] How it works: The server generates a checkout page, including fields for shipping address and payment information.

[1817] Output: Sends the completed checkout page to the user's device.

[1818] Step 11:

[1819] The user enters the voice command "Complete payment."

[1820] Input: The user's voice command.

[1821] How it works: Your device uses Google Cloud Speech-to-Text to convert your voice command into text.

[1822] Output: The string "Complete payment".

[1823] Step 12:

[1824] The terminal sends the payment information to the server.

[1825] Input: Voice command converted to text and user payment information.

[1826] How it works: The device generates an HTTP request and sends it to the server.

[1827] Output: The request data received on the server side.

[1828] Step 13:

[1829] The server processes the payment and notifies the user of the result.

[1830] Input: Request data, including payment information.

[1831] What happens: Your server calls the Stripe API to process the payment and checks the result.

[1832] Output: Sends a notification message to the user's device based on the payment result.

[1833] Step 14:

[1834] The server confirms the order and generates an order completion page.

[1835] Input: The result of the payment process.

[1836] What it does: The server updates the orders table in the MySQL database and generates the order completion page.

[1837] Output: Sends the completed order completion page to the user's device.

[1838] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1839] The present invention provides a system that recognizes user emotions in an e-commerce system and efficiently performs product selection, cart addition, purchase procedure, payment, and order confirmation based on that information. An embodiment of the present invention will be described below with a specific example in which an emotion engine is combined.

[1840] System Configuration

[1841] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[1842] Product selection and display

[1843] When a user browses and selects a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of product information.

[1844] Add to Cart

[1845] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[1846] Checkout and Emotion Recording

[1847] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, verifies the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[1848] Payment Processing

[1849] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[1850] Order Confirmed

[1851] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[1852] Specific examples

[1853] For example, let's consider the process of a user purchasing a "smartphone case." The user accesses an e-commerce site and displays the product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or expresses joy, it highlights products that the user is likely to be particularly interested in. When the user clicks on "smartphone case" and displays the detail page, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[1854] This invention uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works in tandem, allowing users to purchase products intuitively and efficiently.

[1855] The processing flow will be explained below.

[1856] Step 1:

[1857] A user accesses the homepage of an e-commerce site using a terminal. When the terminal receives this request, it requests the homepage data from the server.

[1858] Step 2:

[1859] The server generates HTML data for the homepage and sends it to the device. The device receives this HTML data and displays it in the browser. At this time, the emotion engine is activated and analyzes the user's facial expressions and voice using the device's camera and microphone.

[1860] Step 3:

[1861] A user accesses a product list using a product category or search bar. The device sends a request to the server containing the selected category and search keywords.

[1862] Step 4:

[1863] The server retrieves the relevant product information from the database, generates an HTML page for the product list, and sends it to the terminal. The terminal receives this HTML and displays the product list in the browser. At this time, the emotion engine analyzes the user's reaction and optimizes the display order of the products according to their emotions.

[1864] Step 5:

[1865] A user clicks on a specific product link to view more information about it. The device detects the click event and sends a request including the product ID to the server. The emotion engine analyzes whether the user has a positive reaction to the product.

[1866] Step 6:

[1867] The server retrieves detailed information about the relevant product from the database, generates an HTML page for the product details, and sends it to the terminal. The terminal receives this HTML and displays the detailed product information in the browser. The emotion engine analyzes the user's level of interest and suggests supplementary information or approaches as needed.

[1868] Step 7:

[1869] The user clicks the "Add to Cart" button. The device detects this action and sends a request including the product ID and user ID to the server. The emotion engine records the user's emotion when adding the item to the cart.

[1870] Step 8:

[1871] The server adds the user ID and product ID to the cart table. If the addition process is successful, the server generates a success message and sends it to the terminal. The terminal receives this success message and displays it in the browser. If the emotion engine detects a positive reaction when the user adds a product to the cart, it displays a message to express joy.

[1872] Step 9:

[1873] The user checks the cart and presses the "Proceed to Checkout" button. The device sends a request to the server including the current cart contents and the user ID. The emotion engine records the user's emotions as they proceed to checkout.

[1874] Step 10:

[1875] The server retrieves the user's cart contents from the cart table, generates the HTML for the checkout page, and sends it to the terminal. The terminal receives this HTML and displays the checkout page in the browser. The emotion engine analyzes the user's emotions and adjusts the interface to make the checkout process smoother.

[1876] Step 11:

[1877] The user enters their shipping address and payment information and presses the "Complete Payment" button. The terminal sends the entered information to the server.

[1878] Step 12:

[1879] The server sends the received payment information to the payment service and executes the payment process. When it receives a response from the payment service, it checks whether the payment was successful. The emotion engine records the user's emotion when making a payment.

[1880] Step 13:

[1881] If the payment is successful, the server adds the order information to the order table. At the same time, it updates the inventory information and generates an HTML page for the order completion page indicating that the order has been confirmed, which is sent to the terminal. The user's terminal receives this HTML and displays the order completion page in the browser. The emotion engine analyzes whether the user is satisfied when the order is completed, and displays an appropriate feedback message.

[1882] Through these steps, users can select products, add them to their cart, and complete the purchase process. By using the emotion engine, the purchasing experience can be customized to match the user's emotions, improving the purchasing experience.

[1883] Example 2

[1884] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1885] Conventional e-commerce systems lack the ability to recognize user emotions and optimize the purchasing process, which can lead to lower user satisfaction and purchasing motivation. They also fail to provide appropriate information, especially in difficult situations. Furthermore, they lack a mechanism for utilizing user emotion data to recommend future products.

[1886] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing a user's emotion, a means for optimizing product display based on emotion data, and a means for an emotion engine to record the user's emotion. This makes it possible to analyze the user's emotion in real time and optimize the display of product information and the content of messages, thereby improving user satisfaction and purchasing motivation. Furthermore, recording emotion data can improve the accuracy of future product recommendations.

[1887] "User" refers to a person who uses the system to select, view, and purchase products.

[1888] "Server" refers to a computer system that manages product information, processes data, optimizes emotional data, processes payments, and confirms orders.

[1889] A "terminal" refers to a device used by a user, such as a computer or smartphone, that displays product information and recognizes emotions.

[1890] "Product Information" refers to data including information such as product name, price, description, image, and availability.

[1891] A "cart" refers to a virtual shopping basket in which a user temporarily stores products that they wish to purchase.

[1892] "Payment Information" refers to information necessary to make a payment, such as credit card number, payment method, and billing address.

[1893] "Payment" refers to the process of paying for products using payment information entered by a user.

[1894] An "emotion engine" refers to software or hardware that uses the device's camera and microphone to analyze the user's facial expressions and voice and recognize the user's emotions.

[1895] "Emotion data" refers to data relating to a user's emotions recognized by the emotion engine.

[1896] "Product display optimization" refers to the process of dynamically adjusting the selection and order of products to be displayed, the content of messages, etc., based on user emotional data.

[1897] "Emotion recording" refers to the process of storing a user's emotional data in a database or other location and using it for later analysis or product recommendations.

[1898] The present invention is a system for recognizing a user's emotions in an e-commerce system, and based on this information, efficiently performs product selection, adding to cart, purchasing procedures, payment, and order confirmation. Specific embodiments of this system are described below.

[1899] System Configuration

[1900] This system consists of a terminal used by the user, a server that manages product information and databases, a web interface accessed by the user, and an emotion engine that recognizes the user's emotions.

[1901] Product selection and display

[1902] When a user browses for a product on their device, the server retrieves the product information from the database, generates an HTML web page, and sends it to the user's device. The user's device then displays the received page in a browser. The emotion engine uses the device's camera and microphone to analyze the user's facial expressions and voice, recognizing their emotions. Based on this, the server optimizes the display of the product information.

[1903] Add to Cart

[1904] When a user wants to add a specific product to their cart, they click the "Add to Cart" button on their device. This action causes the device to send a request including the product ID and user ID to the server. The server receives the request and adds the user ID and product ID to the cart table. If the addition is successful, the server generates a success message and sends it to the user's device. The emotion engine simultaneously analyzes the user's emotions and displays humorous messages or messages that encourage purchasing.

[1905] Checkout and Emotion Recording

[1906] When the user checks the contents of the cart and presses the "Proceed to Checkout" button, the device sends a request including the cart contents and user ID to the server. The server receives this request, checks the product information and user information, and generates a checkout page. This page includes fields for entering shipping address and payment information. The emotion engine records the user's emotions in a database and uses that information to recommend products in the future.

[1907] Payment Processing

[1908] When the user fills out the payment form and presses the "Complete Payment" button, the terminal sends the entered information to the server. The server then sends the received payment information to the payment service and executes the payment process. When the server receives a response from the payment service, it checks whether the payment was successful and performs the appropriate process.

[1909] Order Confirmed

[1910] If the payment is successful, the server adds the order information to the order table and updates the inventory information. This confirms the user's purchase. The server generates an order completion page and sends it to the user's device. The user's device receives this page and displays the order completion page in the browser.

[1911] Specific examples

[1912] For example, let's consider the process of a user purchasing a smartphone case. The user accesses an e-commerce site and displays a product list page. The emotion engine analyzes the user's facial expressions and voice, and if the user appears interested or happy, it highlights products that the user is likely to be particularly interested in. When the user clicks on a smartphone case and the detail page is displayed, the emotion engine detects a decrease in excitement and displays a message encouraging action. When the user clicks "Add to Cart," the server adds the product information to the cart, and the emotion engine displays an encouraging message. When the user checks the cart and clicks "Proceed to Checkout," the server facilitates the purchase process. When the user enters payment information and presses the "Complete Payment" button, the server processes the payment and, if successful, confirms the order. Finally, the server generates an order completion page and sends it to the user's device.

[1913] This system uses an emotion engine to optimize the user's purchasing experience and improve user satisfaction. The entire system works together, allowing users to purchase products intuitively and efficiently.

[1914] Prompt Sentence Examples

[1915] "We would like you to design a system that recognizes emotions by analyzing facial expressions and voice using a camera and microphone while a user is viewing a specific product, and optimizes the product display."

[1916] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1917] Product selection and display

[1918] Step 1:

[1919] The user accesses the e-commerce site on the device.

[1920] Input: The user enters the URL of the e-commerce site into a web browser.

[1921] How it works: The browser generates and sends a request to the server.

[1922] Output: A request to access the server.

[1923] Step 2:

[1924] The server retrieves product information from the database

[1925] Input: Access request and user request.

[1926] Operation: The server queries the database to retrieve product information.

[1927] Output: Retrieved product information data.

[1928] Step 3:

[1929] The server generates an HTML web page

[1930] Input: Retrieved product information data.

[1931] How it works: The server generates HTML code based on product information.

[1932] Output: The generated HTML code.

[1933] Step 4:

[1934] The server generates a web page and sends it to the user's device.

[1935] Input: The generated HTML code.

[1936] What it does: The server sends an HTTP response containing HTML code.

[1937] Output: HTML response to the user's device.

[1938] Step 5:

[1939] The page received by the user's device is displayed in the browser.

[1940] Input: The received HTML code.

[1941] What happens: The browser renders the HTML code and displays the product listing page.

[1942] Output: A product listing page viewable by the user.

[1943] Step 6:

[1944] The emotion engine uses the device's camera and microphone to recognize the user's emotions.

[1945] Input: Camera video and microphone audio.

[1946] How it works: The emotion engine uses video and audio analysis algorithms to analyze the user's emotions.

[1947] Output: Recognized emotion data.

[1948] Step 7:

[1949] The server optimizes product display based on emotional data

[1950] Input: Recognized emotion data.

[1951] How it works: The server analyzes the emotion data and sorts and highlights products based on the user's interests.

[1952] Output: Optimized product display data.

[1953] Add to Cart

[1954] Step 1:

[1955] User clicks the "Add to Cart" button

[1956] Input: User clicks.

[1957] Behavior: The browser triggers a click event and begins processing.

[1958] Output: The request data including the product ID and user ID.

[1959] Step 2:

[1960] The device sends a request to the server containing the product ID and user ID.

[1961] Input: Request data including product ID and user ID.

[1962] What happens: The browser sends the request data to the server.

[1963] Output: The request to the server.

[1964] Step 3:

[1965] The server receives the request and adds the information to the cart table.

[1966] Input: Request data to the server.

[1967] Operation: The server executes an operation to add the product ID and user ID to the cart table.

[1968] Output: The data added to the cart table.

[1969] Step 4:

[1970] The server generates a success message and sends it to the user's device.

[1971] Input: Result of adding data to the cart table.

[1972] What happens: The server generates a success message and sends it as an HTTP response.

[1973] Output: A success message to the user's terminal.

[1974] Step 5:

[1975] Emotion engine displays encouraging and humorous messages

[1976] Input: User's emotional data and operation details.

[1977] How it works: The emotion engine generates and displays messages based on the user's emotions.

[1978] Output: Encouraging and humorous messages.

[1979] Checkout and Emotion Recording

[1980] Step 1:

[1981] The user checks the contents of the cart and presses the "Proceed to checkout" button

[1982] Input: User clicks.

[1983] Behavior: The browser triggers a click event and begins processing.

[1984] Output: Request data including cart contents and user ID.

[1985] Step 2:

[1986] The device sends a request to the server containing the cart contents and the user ID.

[1987] Input: Request data including cart contents and user ID.

[1988] What happens: The browser sends the request data to the server.

[1989] Output: The request to the server.

[1990] Step 3:

[1991] The server receives the request and generates the checkout page

[1992] Input: Request data to the server.

[1993] How it works: The server parses the request data and generates a checkout page.

[1994] Output: HTML code for the checkout page.

[1995] Step 4:

[1996] Emotion engine records user emotions

[1997] Input: User emotion data during checkout.

[1998] How it works: The emotion engine analyzes the user's emotions and stores the data in a database.

[1999] Output: Saved emotion data.

[2000] Payment Processing

[2001] Step 1:

[2002] The user enters payment information and presses the "Complete payment" button

[2003] Input: When a user enters payment information and clicks a button.

[2004] Behavior: The browser triggers a click event and begins processing.

[2005] Output: The payment information entered.

[2006] Step 2:

[2007] The device sends the entered information to the server.

[2008] Input: Payment information entered.

[2009] How it works: The browser sends payment information to the server.

[2010] Output: Payment information request to server.

[2011] Step 3:

[2012] The server sends the data to the payment service and processes the payment.

[2013] Input: A request to the server for payment information.

[2014] What happens: The server sends the information to a payment service (e.g., a payment gateway) to execute the payment.

[2015] Output: Processing result from payment service.

[2016] Step 4:

[2017] The server checks whether the payment was successful and executes the process.

[2018] Input: Processing result from payment service.

[2019] Operation: The server checks whether the payment was successful or not, and performs the necessary next steps. For example, if successful, the order is confirmed, and if unsuccessful, an error message is generated.

[2020] Output: The processing result of the payment, whether successful or unsuccessful.

[2021] Order Confirmed

[2022] Step 1:

[2023] The server adds the order information to the orders table

[2024] Input: Payment successful processing result and order information.

[2025] Action: The server adds the order information to the orders table.

[2026] Output: The order information added to the orders table.

[2027] Step 2:

[2028] The server updates the inventory information

[2029] Input: Inventory information that needs to be updated.

[2030] Action: The server updates the inventory information in the database.

[2031] Output: Updated inventory information.

[2032] Step 3:

[2033] The server generates an order completion page and sends it to the user's device.

[2034] Input: Order and inventory information.

[2035] How it works: The server generates the HTML code for the order completion page and sends it to the terminal.

[2036] Output: HTML code for order completion page.

[2037] Step 4:

[2038] The user's device displays the order completion page in the browser.

[2039] Input: HTML code for order completion page.

[2040] What happens: The browser renders the HTML code and displays the order completion page.

[2041] Output: A user-viewable order completion page.

[2042] (Application example 2)

[2043] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2044] In modern e-commerce systems, there is a lack of emotionally-driven product recommendations and prompts to increase purchasing motivation when users make purchases. This leads to a suboptimal user experience, which can lead to a decline in purchasing motivation and a decrease in sales.

[2045] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on the emotions, means for collecting the user's facial expressions and voice via a camera and microphone, and means for adjusting the product recommendation method using the collected user emotion data. This makes it possible to recommend products and display messages optimally in accordance with the user's emotions.

[2046] "Means for user product selection" refers to an interface (website or application) that allows a user to select products online and designate specific products for purchase.

[2047] "Means for the server to manage product information" refers to a server or database system for storing, updating, deleting, and searching product data.

[2048] "Means by which a terminal displays product information to a user" refers to software or applications for displaying product information on a user's device (such as a smartphone, tablet, or computer).

[2049] "Means for the server to add products to the cart" refers to the server's function of adding products selected by the user to the shopping cart.

[2050] "Means for users to enter payment information" refers to the interface (form, input field, etc.) through which users enter the payment information required for purchase.

[2051] "Means by which the server processes payments" refers to the server's ability to actually process payments using the provided payment information.

[2052] "Means for the server to confirm the order" refers to the server's function of officially confirming the order after the payment is completed, and recording and processing the information.

[2053] "Means for analyzing a user's emotions using an emotion recognition device and displaying product recommendations and messages based on those emotions" refers to a system that analyzes a user's facial expressions and voice data collected using a camera and microphone, understands the user's emotional state, and displays product recommendations and customized messages based on that.

[2054] "Means for collecting the user's facial expressions and voice via a camera and microphone" refers to a camera and microphone device for acquiring the user's facial expressions and voice data in real time.

[2055] "Means for adjusting the product recommendation method using collected user emotional data" refers to a system that dynamically adjusts the product recommendation algorithm based on the user's emotional data and presents products that are optimal for the user's emotional state.

[2056] This invention provides a system for recognizing a user's emotions and efficiently selecting products, adding them to a cart, completing purchases, making payments, and finalizing orders within an e-commerce system. The system includes a server, a user terminal, an emotion recognition engine, a camera, and a microphone. A specific embodiment of this system will be described below.

[2057] The server uses an emotion recognition device to analyze the user's emotions and display product recommendations and messages based on those emotions. It also collects the user's facial expressions and voice via a camera and microphone, and adjusts the product recommendation method based on the collected data.

[2058] The device is a device that allows users to browse and select products. It can be a smartphone, tablet, or computer. The device's camera and microphone are used to collect user emotion data, which is then sent to a server. The server then uses this data to run an emotion recognition engine.

[2059] The emotion recognition engine analyzes the user's facial expressions and voice data to identify the user's emotions. This engine uses, for example, the machine learning library Keras and the image processing library OpenCV.

[2060] The server retrieves product information selected by the user from the database and sends it to the terminal. The user uses the terminal to select products and add them to the cart. At this time, the emotion recognition engine analyzes the user's emotions and displays a message to encourage the user to purchase.

[2061] The server then provides a means for the user to enter payment information and processes the payment. If the payment is successful, the server confirms the order information, updates inventory information, and finally generates an order completion page that is displayed on the terminal.

[2062] As a concrete example, imagine a situation where a user is browsing an online shopping site on a smartphone. As the user browses products using the smartphone's camera and microphone, an emotion recognition engine analyzes positive emotions such as joy and excitement from the user's facial expressions. Based on this emotional data, the server recommends the most suitable products for the user and displays messages to further encourage them to purchase.

[2063] An example of a prompt sentence when using a generative AI model is as follows:

[2064] "What products should we recommend if the user's emotions indicate delight or surprise?"

[2065] "If a user is expressing anger or sadness, what message can we show them to encourage them to buy?"

[2066] "Generate a blueprint for a real-time product recommendation system based on sentiment data."

[2067] In this way, by analyzing the user's emotional data in real time and using the results to recommend products and display messages, it is possible to significantly improve the user's purchasing experience.

[2068] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2069] Step 1:

[2070] The device uses a camera and microphone to record the user's facial expressions and voice in real time. The input is video and audio data from the camera and microphone. The output is the recorded video and audio data of the user. This data is used in the subsequent emotion recognition engine.

[2071] Step 2:

[2072] The terminal sends the collected user video and audio data to the server. The input is the user video and audio data recorded in step 1. The output is the video and audio data sent to the server.

[2073] Step 3:

[2074] The server inputs the received video and audio data into the ...

Claims

1. a means for a user to select a product; A means for the server to manage product information; a means for the terminal to display product information to a user; a means for the server to add items to the cart; a means for a user to enter payment information; a means by which the server processes payments; a means by which the server confirms the order; A system including:

2. 2. The system according to claim 1, wherein the terminal through which the user browses the product information communicates with the server via the Internet.

3. 2. The system according to claim 1, wherein the server stores product information in a database, and further comprises means for retrieving product information from the database in response to a user's selection and transmitting the information to the terminal.

Images