System

Abstract

A system is provided.SOLUTION: A system comprising: means for identifying a product from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a user's wink for a particular product; means for presenting the user with an intention confirmation to purchase the product based on the detection of the wink; and means for completing a purchase procedure at a server after obtaining the intention confirmation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many consumers today often become interested in products that are featured in television programs while watching them, but traditional methods require a lot of time and effort to identify and purchase the products. For this reason, there is a need for a system that allows viewers to easily purchase products instantly while enjoying the program. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system including means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for presenting the user with a confirmation of intent to purchase the product based on the detection of the wink, and means for completing the purchase procedure on a server after obtaining the confirmation of intent.

[0006] A "generative model" is a type of machine learning model used to identify products from image data.

[0007] "Video data" refers to data that contains visual information such as television programs.

[0008] "Means for identifying a product" refers to a method or technology for identifying a specific product shown in the video data.

[0009] An "eye tracking device" is a device for detecting the movement of a user's eye line and point of gaze.

[0010] "Means for detecting winks" refers to algorithms and devices for detecting when a user blinks.

[0011] The "means for presenting confirmation of intention to purchase a product" refers to display and operation means for confirming with the user whether or not he / she intends to purchase a particular product.

[0012] A "server" is a computer system that processes, stores, transmits, etc. data.

[0013] The "means of completing the purchase procedure" refers to a series of processes for the user to go through the purchase procedure for the product selected by the user and finally confirm the purchase. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[0036] System configuration

[0037] The system of the present invention mainly comprises the following components:

[0038] 1. Server

[0039] 2. Devices (TV, eye tracking devices)

[0040] 3. Users

[0041] server

[0042] The server has the following roles:

[0043] Receiving video data: Receive real-time video data sent from the TV terminal.

[0044] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[0045] Recommendation generation: Using the analysis results, product recommendations for the user are generated and sent to the device.

[0046] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[0047] Terminal

[0048] The device consists of a television and an eye-tracking device.

[0049] Capture of video data: Captures television images and sends them to the server.

[0050] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[0051] Eye tracking: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[0052] Confirmation of purchase intent: When a wink is detected, a confirmation message for purchasing the product is displayed to the user.

[0053] User

[0054] The user performs the following actions:

[0055] Watching TV: Watch TV programs in the usual way.

[0056] Product selection: When a product of interest appears on the screen, look at it and wink.

[0057] Confirm your intention to purchase: After the wink, a confirmation message appears, and you can indicate your intention to purchase by winking again or pressing a specific key.

[0058] Explanation of program processing

[0059] Server Processing

[0060] 1. Receiving image data: The server receives video data transmitted in real time from the TV terminal.

[0061] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data.

[0062] 3. Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the terminal.

[0063] Terminal handling

[0064] 1. Displaying recommended information: The TV terminal receives the recommended information sent from the server and displays it as an overlay on the TV screen.

[0065] 2. Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[0066] 3. Display purchase confirmation message: When a wink is detected, a purchase confirmation message will be displayed on the TV screen.

[0067] User Actions

[0068] 1. TV viewing: Users watch TV and wait for products of interest to appear.

[0069] 2. Product selection: Wink at a specific product on the screen.

[0070] 3. Confirm your intention to purchase: In response to the confirmation message that appears, wink or press a key again to indicate your intention to purchase.

[0071] Specific examples

[0072] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0073] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[0074] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[0075] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[0076] 4. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[0077] 5. Purchase confirmation: A confirmation message will appear on the TV screen asking, "Are you sure you want to purchase this item?"

[0078] 6. Intent to buy: The user winks again to confirm their intent to buy.

[0079] 7. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[0080] As described above, the present invention provides a system that allows users to easily purchase products displayed on a television screen.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] Terminal

[0084] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[0085] Step 2:

[0086] server

[0087] The server receives the video data sent from the TV terminal and saves it in high resolution.

[0088] Step 3:

[0089] server

[0090] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[0091] Step 4:

[0092] server

[0093] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[0094] Step 5:

[0095] Terminal

[0096] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[0097] Step 6:

[0098] Terminal

[0099] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[0100] Step 7:

[0101] Terminal

[0102] If a wink is detected, a purchase confirmation message will be displayed on the TV screen, asking "Are you sure you want to purchase this item?" and asking for a wink or keystroke for confirmation.

[0103] Step 8:

[0104] User

[0105] When a product of interest appears on the TV screen, the user turns their gaze towards the product and winks. After that, in response to the purchase confirmation message displayed on the TV screen, the user can wink again or use a keystroke to indicate their intention to purchase.

[0106] Step 9:

[0107] Terminal

[0108] The information confirming the user's intention to purchase is sent to the server, which is immediately notified and the purchase process begins.

[0109] Step 10:

[0110] server

[0111] Once the purchase confirmation is received, the system completes the purchase process, including verifying the user's address and payment method, and finally notifies the user that the purchase is complete.

[0112] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching television.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] While watching TV, users have no way to visually select and easily purchase products displayed, which makes the process of purchasing complicated and time-consuming. Furthermore, conventional methods make it difficult to track and quickly suggest products that interest users.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for presenting the user with a confirmation of their intention to purchase the product based on the detection of the wink, means for completing the sales procedure after obtaining the confirmation of their intention, means for generating and displaying product recommendations for the user based on the results of image analysis by the generative model, means for displaying a confirmation message for a product selected by a wink and indicating their intention to purchase again by a wink or key operation, and means for sending a notification to the user after the purchase procedure is completed. This allows the user to visually easily select a product and quickly complete the procedure up to the purchase.

[0118] A "generative model" is a machine learning algorithm used to identify and extract specific objects and information from video data.

[0119] "Video data" refers to digital signals containing visual information transmitted by televisions and other visual devices.

[0120] An "eye-tracking device" is a device that detects the direction and movement of a user's gaze in real time.

[0121] "Wink detection" is a function that detects the user closing one eye and recognizes this action as a specific instruction.

[0122] "Confirming intention" is an act in which a user explicitly indicates their intention through a specific operation or action.

[0123] "Sales procedure" refers to the series of operations and processes required to purchase a product, including payment and delivery procedures.

[0124] "Recommendation information" is information that includes suggestions for specific products based on the user's preferences and purchase history.

[0125] A "visual display device" is a device, such as a television or monitor, that provides visual information to a user.

[0126] A "notification" is a message or alert sent by the system to convey information or results to a user.

[0127] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[0128] System configuration

[0129] The system of the present invention mainly comprises the following components:

[0130] 1. Server

[0131] 2. Devices (TV, eye tracking devices)

[0132] 3. Users

[0133] server

[0134] The server has the following roles:

[0135] Receiving video data: The server receives real-time video data sent from the TV terminal, for example, by streaming the video data using an HTTP request.

[0136] Analysis using a generative model: The server uses a generative model to identify the location, type, brand, and model number of products contained in the video data. Specifically, it runs a product recognition model using TENSORFLOW (registered trademark) or PyTorch. An example of a prompt statement is, "Please run a model to identify the type, brand, and model number of the product displayed in this video."

[0137] Recommendation generation: Based on the analysis results, the server generates product recommendations for the user and sends them to the device. During this process, a list of recommended products is generated that takes into account the user's purchase history and preferences.

[0138] Terminal

[0139] The device consists of a television and an eye-tracking device.

[0140] Capture of video data: Captures television images and sends them to the server.

[0141] Recommendation display: The TV device receives the recommendation information sent from the server and displays it as an overlay on the TV screen. For example, it displays the product image and detailed information along with the message "Do you like this product?"

[0142] Eye tracking and wink detection: An eye tracking device (e.g., Tobii Eye Tracker) tracks the user's gaze and detects winks toward specific products. Wink detection detects the action of closing one eye after the gaze has stopped on the product image for a certain period of time.

[0143] Displaying purchase intent confirmation: When a wink is detected, a confirmation message will be displayed on the TV screen asking, "Do you really want to purchase this product?" Specific purchase procedures and payment information will also be displayed.

[0144] User

[0145] The user performs the following actions:

[0146] TV viewing: Users watch TV and wait for products of interest to appear.

[0147] Product selection: Wink at a specific product on the screen.

[0148] Confirm your intention to purchase: In response to the displayed confirmation message, you can wink or press a key again to indicate your intention to purchase. For example, you can press the "OK" button on the remote control or wink again to indicate your intention to purchase.

[0149] Specific examples

[0150] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0151] 1. The server receives real-time video data sent from the TV terminal via an HTTP request. For example, the server retrieves video data from http: / / example.com / streaming-url.

[0152] 2. The server uses TensorFlow to identify the coffee maker brand, model number, etc. from the video data. For example, load the model with tf.keras.models.load_model('product_recognition_model.h5') and analyze the video data.

[0153] 3. The server generates recommendations for new coffee makers based on the user's past purchase history.

[0154] 4. The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, "The latest coffee makers" will pop up in the bottom right corner of the screen.

[0155] 5. The device uses the Tobii Eye Tracker to track the user's gaze and winks. After the user's gaze is focused on the image of the coffee maker, the device detects the closing of one eye.

[0156] 6. A message will appear on your TV screen asking, "Are you sure you want to purchase this item?" The message will be displayed large and in the center of the screen, along with payment options.

[0157] 7. The user confirms their intention to purchase by winking again or pressing the "OK" button on the remote control.

[0158] 8. The server calls the online store's API to complete the purchase. For example, it sends the user and product information to http: / / example-store.com / api / purchase.

[0159] 9. The server sends a purchase completion notification to the user. The message "Purchase Completed" is displayed on the TV screen, and a purchase completion notification is sent to the user's email address.

[0160] The above is an embodiment of the present invention. This system allows users to visually select products easily and quickly complete the procedures up to purchase.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1:

[0163] Receiving video data

[0164] The server receives video data sent in real time from a TV terminal using an HTTP request. Specifically, the terminal captures the TV video and sends it to the server. The TV video data is input, and the received TV video data is saved on the server as output.

[0165] Step 2:

[0166] Image analysis using generative models

[0167] The server analyzes the received video data using a generative model (for example, a model using TensorFlow or PyTorch). This analysis identifies the location, type, brand, and model number of the product within the video data. The received video data is the input, and specific product information is obtained as the output.

[0168] An example of a prompt sentence is "Please run a model to identify the type, brand, and model number of the product shown in this video."

[0169] Step 3:

[0170] Recommendation generation

[0171] The server generates product recommendations for the user based on the product information identified by the generative model. The generated recommendation information is sent to the terminal. In this step, the user's preferences and purchase history are also taken into consideration. The input is product identification information and the user's purchase history, and the output is recommendation information.

[0172] Step 4:

[0173] Displaying recommendation information

[0174] The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, a product image and detailed information are displayed along with the message "Do you like this product?" The input is the recommendation information, and the output is a screen display that the user can visually confirm.

[0175] Step 5:

[0176] Eye Tracking and Wink Detection

[0177] An eye tracking device (for example, Tobii Eye Tracker) connected to the device tracks the user's gaze and detects winks at a specific product. Specifically, it detects when the gaze is focused on the product image and then the user closes one eye. The input is the user's gaze data, and the output is wink detection data.

[0178] Step 6:

[0179] Display of purchase intent confirmation

[0180] When the device detects a wink, it displays a confirmation message on the TV screen asking, "Do you really want to purchase this product?". Specific purchase procedures and payment information are also displayed. The input is the wink detection data, and the output is the confirmation message and purchase procedure information displayed on the screen.

[0181] Step 7:

[0182] Purchase Intention

[0183] The user indicates their intention to purchase by winking again or pressing the "OK" button on the remote control. Specifically, the user's operation is the input, and the data on their intention to purchase is obtained as the output.

[0184] Step 8:

[0185] Completing the purchase process

[0186] The server confirms the user's intent to purchase and begins the purchase process. Specifically, it calls the online store's API using the product information, as well as the user's address and payment information, to confirm the order. The inputs are data indicating the user's intent to purchase and user information, and the output is data indicating the purchase process has been completed.

[0187] Step 9:

[0188] Sending a purchase completion notification

[0189] After the purchase procedure is completed, the server sends a notification of purchase completion to the user. Specifically, it displays a message on the TV screen saying "Purchase completed" and sends a notification by email. The input is data indicating the purchase procedure has been completed, and the output is a completion notification sent to the user.

[0190] (Application example 1)

[0191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] Conventional systems that identify products from video data and recommend them to users are primarily limited to television footage, with few examples of application in physical stores. Furthermore, systems that efficiently confirm a user's purchase intentions by using eye tracking and winks are also lacking. Furthermore, recommendation displays that are limited to television footage are not suitable for in-store environments, and a flexible system that can be used in a variety of store situations is desired. Given this background, the challenge is to provide a system that can be used in physical stores and improves the efficiency of the purchasing process by using eye tracking and wink detection.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0194] In this invention, the server includes: means for identifying a product from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a confirmation of the user's intention to purchase the product based on the detection of the wink; means for completing the purchase procedure on the server after obtaining the confirmation of intention; means for converting the video data into in-store video data and performing eye-tracking through smart glasses or a head-mounted display; means for displaying a purchase confirmation message within the user's field of view after detecting a wink; and means for reconfirming the user's intention to purchase using voice recognition. This enables efficient product selection through eye-tracking and automated purchase procedures even in physical stores. Furthermore, the use of voice recognition allows the user to confirm their intention to purchase more intuitively and naturally.

[0195] A "generative model" is an algorithm that uses machine learning technology to automatically identify products from video data.

[0196] "Video data" refers to digital data of visual information captured by a camera or the like, including television footage and footage of inside a store.

[0197] An "eye tracking device" is a device that detects the movement of a user's eyes in real time and identifies the direction in which the user is looking.

[0198] "Wink detection" is a technology that uses an eye-tracking device or the like to detect when a user makes a specific action, i.e., a wink.

[0199] The "server" is a central computer system that analyzes video data, runs generative models, manages purchase procedures, etc.

[0200] "Smart glasses" are high-performance eyeglass devices that can be worn by users and are capable of eye tracking and augmented reality display.

[0201] A "head-mounted display" is a device worn on the user's head that displays visual information and tracks the user's gaze.

[0202] A "purchase confirmation message" is a message that is displayed when a user selects a product, to reconfirm the user's intention to purchase the product.

[0203] "Speech recognition" is a technology that recognizes a user's voice and converts it into digital information.

[0204] "Product recommendation" is a function that presents products suitable for the user based on product information analyzed by the generative model.

[0205] A system for implementing the present invention mainly comprises the following components: a server, a terminal (smart glasses or head-mounted display, eye-tracking device), and a user.

[0206] The server uses the generative model to identify products from the video data, tracks the user's gaze using an eye-tracking device, detects winks toward the specific products, and, based on the detected winks, prompts the user to confirm their intention to purchase the product, after which the purchase process is completed.

[0207] Specifically, the server has the following roles:

[0208] Receiving video data: Receive real-time video data transmitted from smart glasses or head-mounted displays.

[0209] Generative model analysis: Run a generative model to identify the location, type, and brand of products in the video data.

[0210] Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the device.

[0211] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[0212] The smart glasses or head-mounted display as a terminal plays the following roles:

[0213] Video data capture: Video from inside the physical store is captured in real time and sent to the server.

[0214] Recommendation display: Recommendation information sent from the server is displayed within the field of view.

[0215] Eye Tracking: The eye tracking device tracks the user's gaze in real time and detects winks towards specific products.

[0216] Confirmation of purchase intent: When a wink is detected, a purchase confirmation message is displayed within the field of view and voice recognition is used to prompt the user to reconfirm their intention to purchase.

[0217] The user performs the following actions:

[0218] Store Walking: Walking around the store and exploring products that interest you.

[0219] Product selection: Look at the product you are interested in and wink.

[0220] Confirm your intention to purchase: When the purchase confirmation message appears, wink or speak again to indicate your intention to purchase.

[0221] As a concrete example, consider a situation where a user is walking through a brick-and-mortar store and looking at a coffee maker on a shelf. The smart glasses capture video data and send it to a server. A generative model analyzes the coffee maker and identifies detailed information. An eye-tracking device tracks the user's gaze and detects that the user is focusing on the coffee maker. When the user winks, a confirmation message appears on the smart glasses asking, "Do you want to purchase this item?" The user can wink again or respond "yes" verbally to proceed with the purchase.

[0222] An example of a prompt for the generative AI model is as follows:

[0223] "Identify products from the following store video data and determine their location, type, and brand."

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1:

[0226] The server receives real-time video data transmitted from smart glasses or head-mounted displays. This input data is video data captured by cameras inside the store, and after being sent to the server, analysis begins.

[0227] Step 2:

[0228] The server inputs the received video data into a generative model, which analyzes the data and identifies the product's location, type, and brand. The specific processing of this data involves the generative model's algorithm identifying the product and generating metadata such as location information and brand name.

[0229] Step 3:

[0230] The server then generates product recommendations based on the generated metadata. For example, if a particular product has a high rating, it will be recommended first. This information is then sent to the smart glasses or head-mounted display.

[0231] Step 4:

[0232] The device (smart glasses or head-mounted display) displays the received recommendation information within the user's field of view. The displayed recommendation information is provided as an overlay on the video data.

[0233] Step 5:

[0234] The device tracks the user's gaze in real time using an eye-tracking device, which detects the user's eye movements and identifies which products the user's gaze is focused on.

[0235] Step 6:

[0236] The device detects when a user winks at a specific product. The wink detection algorithm analyzes the gaze detection data and recognizes the wink. This information is sent to the server.

[0237] Step 7:

[0238] The server receives the wink detection information, generates a purchase confirmation message for the specific product, and sends it to the terminal. The message includes a confirmation message such as "Do you want to purchase this product?"

[0239] Step 8:

[0240] The device displays a purchase confirmation message within the user's field of vision, and the user can respond to this message by winking again or saying "yes" or similar to indicate their intention to purchase.

[0241] Step 9:

[0242] The device then sends another wink or voice recognition result to the server, which is used to confirm the user's final purchase decision.

[0243] Step 10:

[0244] The server receives the final purchase confirmation data and executes the purchase procedure, including retrieving product information from the product database and completing the purchase. Finally, it sends a purchase completion notification to the user.

[0245] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0246] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[0247] System configuration

[0248] The system of the present invention consists of the following major components:

[0249] 1. Server

[0250] 2. Devices (TVs, eye tracking devices, emotion engines)

[0251] 3. Users

[0252] server

[0253] The server has the following roles:

[0254] Receiving video data: Receive real-time video data sent from the TV terminal.

[0255] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[0256] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device.

[0257] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[0258] Terminal

[0259] The device consists of a television, an eye-tracking device, and an emotion engine.

[0260] Capture of video data: Captures television images and sends them to the server.

[0261] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[0262] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[0263] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotions.

[0264] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[0265] User

[0266] The user performs the following actions:

[0267] Watching TV: Watch TV programs in the usual way.

[0268] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[0269] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[0270] Explanation of program processing

[0271] Server Processing

[0272] Receiving image data: The server receives the video data transmitted in real time from the television terminal.

[0273] Image analysis with generative models: Using generative models to identify the location, type, brand, and model number of products in video data.

[0274] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the device.

[0275] Terminal handling

[0276] Display of recommendation information: The TV terminal receives the recommendation information sent from the server and displays it as an overlay on the TV screen.

[0277] Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[0278] Emotion recognition: The emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[0279] Displaying purchase intent confirmation: A purchase confirmation message is displayed on the TV screen based on winks and emotion data.

[0280] User Actions

[0281] TV viewing: Users watch TV and wait for products of interest to appear.

[0282] Product Selection: Winks at specific products on the screen and expresses emotional state through emotion analysis by the emotion engine.

[0283] Confirm your intention to purchase: In response to the displayed confirmation message, wink or press a key again to indicate your intention to purchase.

[0284] Specific examples

[0285] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0286] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[0287] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[0288] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[0289] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions.

[0290] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[0291] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is favorable, the confirmation message will be highlighted.

[0292] 7. Intent to buy: The user winks again to confirm their intent to buy.

[0293] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[0294] As described above, the system of the present invention takes into account the user's emotions and provides a more personalized and advanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[0295] The processing flow will be explained below.

[0296] Step 1:

[0297] Terminal

[0298] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[0299] Step 2:

[0300] server

[0301] The server receives the video data sent from the TV terminal and saves it in high resolution.

[0302] Step 3:

[0303] server

[0304] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[0305] Step 4:

[0306] server

[0307] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[0308] Step 5:

[0309] Terminal

[0310] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[0311] Step 6:

[0312] Terminal

[0313] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[0314] Step 7:

[0315] Terminal

[0316] At the same time as detecting the wink, the emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotional state. If the user is interested and expressing positive emotions, the information is sent to the server.

[0317] Step 8:

[0318] server

[0319] The server receives the emotion data from the emotion engine and further optimizes the recommendation information based on the user's emotions. Based on the wink detection and emotion data, a purchase confirmation message is generated and sent to the device.

[0320] Step 9:

[0321] Terminal

[0322] The device displays a purchase confirmation message on the TV screen. The message asks the user to confirm whether they really want to purchase this product. Depending on the emotion data, the message may be highlighted.

[0323] Step 10:

[0324] User

[0325] Based on the confirmation message, the user can then wink or press a key again to indicate their intention to purchase. If their emotional state is appropriate for purchasing, they can wink again to indicate their intention to purchase.

[0326] Step 11:

[0327] Terminal

[0328] Once the user's intention to purchase is confirmed, the terminal transmits this information to the server and begins the purchase procedure.

[0329] Step 12:

[0330] server

[0331] The server receives the confirmation of intent to purchase and processes the product purchase, which may include verifying the user's address and payment method, connecting with back-end systems, etc. Finally, it notifies the user that the purchase is complete.

[0332] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching TV, and by utilizing the emotion engine, provides a more personalized purchasing experience.

[0333] Example 2

[0334] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0335] Conventional product purchasing systems that rely on eye gaze tracking and wink detection have the problem of not being able to fully consider the user's emotional state and therefore fail to optimize the user experience. Furthermore, false detection due to users unintentionally winking is also a problem. Therefore, there is a need to provide a more advanced product purchasing process that reflects the user's emotional state, rather than simply detecting eye gaze and winks.

[0336] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0337] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for analyzing the user's emotions using an emotion analysis device, means for presenting a purchase confirmation to the user based on the wink and the results of the emotion analysis, and means for completing the purchase procedure on the server after receiving the confirmation, thereby providing a product purchase process that takes the user's emotional state into account, reducing false positives and enabling a more personalized purchasing experience.

[0338] A "generative model" is an algorithm used to analyze video data to identify or classify specific items.

[0339] An "eye tracking device" is a device that detects the movement of a user's eyes and analyzes where the eyes are looking.

[0340] An "emotion analysis device" is a device that analyzes a user's facial expressions and voice patterns to identify the user's emotional state.

[0341] A "wink" is a user action in which one or both eyes are temporarily closed for a specific purpose.

[0342] "Product purchase intent confirmation" is the process of confirming whether a user intends to purchase a particular product based on a wink or other confirmation means.

[0343] A "server" is a computer system that receives, transmits, analyzes, and processes various types of data on a network.

[0344] "Video data" refers to real-time or existing video content such as television programs and videos.

[0345] "Recommendation information" is information that suggests optimal products and services to users based on the analysis results and the user's emotional data.

[0346] MODE FOR CARRYING OUT THE INVENTION

[0347] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[0348] System configuration

[0349] The system of the present invention consists of the following major components:

[0350] 1. Server

[0351] 2. Devices (TVs, eye tracking devices, emotion engines)

[0352] 3. Users

[0353] server

[0354] The server has the following roles:

[0355] Receiving video data: Receive real-time video data sent from the TV terminal.

[0356] Analysis using a generative model: A generative model is run to identify the location, type, brand, and model number of products contained in the video data. The generative model used here is "YOLO (You Only Look Once)."

[0357] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device. Microsoft® Azure® Cognitive Services is used to analyze the emotional data.

[0358] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[0359] Terminal

[0360] The device consists of a television, an eye-tracking device, and an emotion engine.

[0361] Capture of video data: Captures television images and sends them to the server.

[0362] Recommendation display: Recommendation information sent from the server is displayed on the TV screen. Game engines such as "Unity" are used for display.

[0363] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products. The device used here is the "Tobii Eye Tracker."

[0364] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to generate emotion data. The "Affectiva SDK" is used to analyze the user's emotions.

[0365] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[0366] User

[0367] The user performs the following actions:

[0368] Watching TV: Watch TV programs in the usual way.

[0369] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[0370] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[0371] Specific examples

[0372] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0373] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[0374] 2. Image analysis: The server uses the generative model to identify the brand, model number, etc. of the coffee maker. In this process, the prompt statement is used: "Analyze a video of a cooking show and identify the brand and model number of the product shown."

[0375] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[0376] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions, using the prompt "Please recognize the user's emotional state and prioritize recommendations."

[0377] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[0378] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is positive, a prompt will be set saying, "If the user's emotion is positive, highlight the confirmation message."

[0379] 7. Intent to buy: The user winks again to confirm their intent to buy.

[0380] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[0381] In this way, the system of the present invention takes into account the user's emotions and provides a more personalized and enhanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[0382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0383] Program processing flow

[0384] Server Processing

[0385] Step 1: Receiving video data

[0386] The server receives video data transmitted in real time from the television terminal.

[0387] Input: Real-time video data from a TV terminal

[0388] Output: Received video data

[0389] Specifically, the server receives a video data stream via a network.

[0390] Step 2: Image analysis

[0391] The server analyzes the received video data and uses the generative model "YOLO (You Only Look Once)" to identify the product's location, type, brand, and model number.

[0392] Input: Video data

[0393] Output: Product location, type, brand, and model number information

[0394] After the video data is sent to the server, a generative model is run to extract and classify features of objects in the image.

[0395] Step 3: Integrating Emotional Data

[0396] The server receives the user's emotion data obtained from the emotion engine and integrates it with the image analysis results. Emotion analysis is performed using Microsoft Azure Cognitive Services.

[0397] Input: Image analysis results, emotion data

[0398] Output: Consolidated data

[0399] The server combines the analyzed product information with the user's emotional data and integrates the information necessary to generate recommendations.

[0400] Step 4: Recommendation generation

[0401] The server generates product recommendations for the user based on the image analysis results and emotional data and sends them to the terminal.

[0402] Input: Integrated data

[0403] Output: Recommendation information

[0404] The server selects the most suitable products based on the integrated data and generates specific recommendation information.

[0405] Step 5: Checkout

[0406] The server confirms the user's intention to purchase and completes the purchase procedure. The server notifies the user that the purchase procedure is complete.

[0407] Input: Confirm purchase intention

[0408] Output: Purchase completion notification

[0409] The server receives confirmation of the user's intention to purchase, processes the purchase, and notifies the user of the result.

[0410] Terminal handling

[0411] Step 1: Capture video data

[0412] The device captures the TV image and sends it to the server.

[0413] Input: TV program video

[0414] Output: Captured video data

[0415] Specifically, video data is acquired from a television screen and sent to a server as a stream.

[0416] Step 2: Displaying recommendation information

[0417] The terminal receives the recommendation information sent from the server and displays it as an overlay on the television screen.

[0418] Input: Recommendation information

[0419] Output: Displayed recommendation information

[0420] The received information is analyzed, and a user interface is generated and displayed on the screen.

[0421] Step 3: Eye tracking and wink detection

[0422] The eye tracking device, Tobii Eye Tracker, tracks the user's gaze and detects winks at specific products.

[0423] Input: User gaze data

[0424] Output: Wink detection result

[0425] The device tracks the user's gaze in real time, detects winks, and notifies the server.

[0426] Step 4: Emotion Recognition

[0427] The emotion engine analyzes the user's facial expressions and voice to generate emotion data. The user's emotions are analyzed using the "Affectiva SDK."

[0428] Input: User's facial expression and voice data

[0429] Output: Emotion data

[0430] Analyzed emotion data is generated and this information is sent to a server.

[0431] Step 5: Confirm your intention to purchase

[0432] A purchase confirmation message is displayed on the TV screen based on wink and emotion data.

[0433] Input: Wink detection results, emotion data

[0434] Output: Purchase confirmation message

[0435] Based on the analysis results, a confirmation message is overlaid on the screen.

[0436] User Actions

[0437] Step 1: Watch TV

[0438] The user watches the television program in the usual way.

[0439] Input: TV show

[0440] Output: User viewing behavior

[0441] Operate and watch the TV to view the program on the screen.

[0442] Step 2: Product selection

[0443] When a product that interests the user appears on the screen, the user turns their gaze and winks. The emotion engine analyzes the user's interest from their reaction.

[0444] Input: On-screen item

[0445] Output: Gaze data, wink, emotion data

[0446] The system detects gazes and winks towards the product and sends them to the server along with emotional data.

[0447] Step 3: Confirm your intention to purchase

[0448] In response to the purchase confirmation message, the user indicates their intention to purchase by winking again or pressing a specific key.

[0449] Input: Purchase confirmation message

[0450] Output: Re-wink, key operation

[0451] View the confirmation message and take the appropriate action to indicate your intent to purchase.

[0452] The above is a specific explanation of each processing step.

[0453] (Application example 2)

[0454] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0455] In online shopping, conventional methods have limited ability to quickly and intuitively confirm a user's purchasing intention, and they are unable to recommend products that take into account the user's emotional state. This can lead to a poor user experience and a decrease in purchasing motivation. To solve this problem, an advanced recommendation system that combines eye tracking and emotion recognition is needed.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0457] In this invention, the server includes: means for identifying products from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a product purchase confirmation to the user based on the wink detection; means for completing the purchase procedure on the server after receiving the confirmation; means for analyzing the user's emotions using an emotion recognition engine and adjusting / emphasizing the product recommendation and confirmation message based on the emotion data; means for performing eye-tracking, emotion recognition, and wink detection using the smartphone's front camera and microphone; and means for overlaying recommendation information on the online shop screen and displaying a purchase confirmation message based on the emotion data, thereby enabling an intuitive and personalized purchasing experience that takes into account the user's gaze and emotional state.

[0458] A "generative model" is a machine learning algorithm for extracting and analyzing specific information from video data.

[0459] "Video data" refers to image and video data acquired from a photographing device such as a camera.

[0460] "Product" is a term that generally refers to products and services sold in online shops, etc.

[0461] An "eye tracking device" is a device that uses a camera or sensor to detect and track the position of a user's gaze.

[0462] "User" refers to the consumer or user of the System.

[0463] "Wink detection" is the process of using a sensor or camera to recognize when a user closes one of their eyes.

[0464] "Confirming intention" is an act in which a user confirms his or her own intentions or wishes through a specific operation or action.

[0465] A "server" is a computer system that provides services to multiple clients.

[0466] An "emotion recognition engine" is a technology for analyzing and estimating a user's emotional state from facial expressions, voice, etc.

[0467] "Recommendation" is the act of the system presenting products or information that are appropriate and likely to interest the user.

[0468] A "smartphone" is a multi-functional mobile device with advanced computing capabilities.

[0469] A "front camera" is a camera installed on the screen side of a smartphone or tablet.

[0470] A "microphone" is a device that picks up sound and converts it into an electrical signal.

[0471] "Overlay display" is a method of displaying additional information over an existing image or video.

[0472] An "online shop" is a website or application that sells products or services over the Internet.

[0473] The present invention provides a system that provides an advanced product recommendation and purchasing process that takes into account the user's gaze and emotional state. This system consists of the following main components: a server, a terminal (the user's smartphone), and a user. Next, we will explain the role of each component and its specific implementation.

[0474] server

[0475] The server has the following features:

[0476] 1. Receiving image data: The server receives real-time video data sent from the smartphone terminal.

[0477] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data. The generative model can be based on TensorFlow or Google Cloud Vision API.

[0478] 3. Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the smartphone device.

[0479] 4. Completing the purchase process: Based on the user's purchase intent and emotional state, the purchase process is completed using the Stripe API or similar.

[0480] Device (smartphone)

[0481] The smartphone has the following features:

[0482] 1. Capture video data: Capture video data from the smartphone's front camera and send it to the server.

[0483] 2. Displaying recommendation information: The recommendation information sent from the server is overlaid on the online shop screen.

[0484] 3. Eye Tracking and Wink Detection: Use the front camera to track the user's gaze and detect winks towards specific products. This is done using OpenCV.

[0485] 4. Emotion Recognition: Using the front camera and microphone, the system analyzes the user's facial expressions and voice patterns to generate emotion data. The emotion recognition engine is DeepFace.

[0486] 5. Confirmation of purchase intention: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[0487] User

[0488] The user performs the following actions:

[0489] 1. Online Shopping Browsing: Browse the online shop page in the usual way.

[0490] 2. Product Selection: When a product of interest appears on the screen, the robot turns its gaze and winks, and also expresses its emotions through facial expressions and voice.

[0491] 3. Confirming purchase intent: In response to a confirmation message displayed based on a wink or emotional state, the user can indicate their intention to purchase by winking again or using a specific finger gesture.

[0492] Specific examples

[0493] For example, consider a user browsing the latest smartphone models in an online shop. The server receives video data sent from the smartphone and uses a generative model to identify product information. The server generates recommendation information and overlays it on the smartphone screen. Meanwhile, the front camera and microphone are used to track the user's gaze and emotional state, and an emotion recognition engine recognizes that the user is pleased. If the user winks at the product, a purchase confirmation message is displayed, and the user confirms their intention to purchase with another wink, and payment is completed via the Stripe API.

[0494] Prompt Sentence Examples

[0495] Capture video from the front camera

[0496] Face detection and eye tracking initialization

[0497] Eye tracking and wink detection

[0498] Emotion Recognition and Recommendation Generation

[0499] Display purchase confirmation message

[0500] Wink again to confirm your intention to purchase

[0501] Completing the purchase process

[0502] This allows users to enjoy an intuitive and emotional purchasing experience.

[0503] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0504] Step 1:

[0505] Video data capture

[0506] The device uses the smartphone's front camera to capture video data of the online shop page the user is viewing. This video data is sent to the server in real time. The video data is the input, and the data sent to the server is the output.

[0507] Step 2:

[0508] Image analysis using generative models

[0509] The server runs a generative model based on the video data received from the device to identify the location, type, brand, and model number of the product in the video data. Generative models such as TensorFlow and Google Cloud Vision API are used. The input is the received video data, and the output is detailed product data such as the product's location and type.

[0510] Step 3:

[0511] Eye tracking and wink detection

[0512] The device uses the front camera to track the user's gaze and OpenCV to determine whether the user is looking at a specific product. It also runs a wink detection algorithm to detect winks toward a specific product. The input is camera image data, and the output is gaze information and wink detection results.

[0513] Step 4:

[0514] emotion recognition

[0515] The device uses the front camera and microphone to capture the user's facial expressions and voice, which are then analyzed by an emotion recognition engine such as DeepFace. This generates user emotion data. The input is facial expression data and voice data, and the output is the emotion recognition result.

[0516] Step 5:

[0517] Generating and displaying recommendation information

[0518] The server combines the gaze tracking results, wink detection results, and emotion data to generate product recommendation information. This information is then sent to the device, which then displays the recommendation information as an overlay on the online shop screen. The inputs are gaze information, wink detection results, and emotion data, and the output is recommendation information.

[0519] Step 6:

[0520] Confirmation of purchase intention

[0521] The device displays a product purchase confirmation message to the user based on the wink detection and emotion recognition results. The user confirms their intention to purchase by winking again or performing a specific finger gesture. The inputs are the wink detection results and emotion recognition results, and the output is the user's intention to purchase.

[0522] Step 7:

[0523] Completing the purchase process

[0524] The server receives confirmation of the user's intent to purchase and completes the purchase procedure using the Stripe API or similar. This confirms the product purchase and sends a purchase completion notification to the user. The input is the user's confirmation of intent to purchase, and the output is purchase confirmation and a purchase completion notification.

[0525] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0526] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0527] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0528] [Second embodiment]

[0529] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0530] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0531] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0532] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0533] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0534] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0535] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0536] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0537] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0538] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0539] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0540] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0541] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[0542] System configuration

[0543] The system of the present invention mainly comprises the following components:

[0544] 1. Server

[0545] 2. Devices (TV, eye tracking devices)

[0546] 3. Users

[0547] server

[0548] The server has the following roles:

[0549] Receiving video data: Receive real-time video data sent from the TV terminal.

[0550] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[0551] Recommendation generation: Using the analysis results, product recommendations for the user are generated and sent to the device.

[0552] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[0553] Terminal

[0554] The device consists of a television and an eye-tracking device.

[0555] Capture of video data: Captures television images and sends them to the server.

[0556] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[0557] Eye tracking: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[0558] Confirmation of purchase intent: When a wink is detected, a confirmation message for purchasing the product is displayed to the user.

[0559] User

[0560] The user performs the following actions:

[0561] Watching TV: Watch TV programs in the usual way.

[0562] Product selection: When a product of interest appears on the screen, look at it and wink.

[0563] Confirm your intention to purchase: After the wink, a confirmation message appears, and you can indicate your intention to purchase by winking again or pressing a specific key.

[0564] Explanation of program processing

[0565] Server Processing

[0566] 1. Receiving image data: The server receives video data transmitted in real time from the TV terminal.

[0567] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data.

[0568] 3. Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the terminal.

[0569] Terminal handling

[0570] 1. Displaying recommended information: The TV terminal receives the recommended information sent from the server and displays it as an overlay on the TV screen.

[0571] 2. Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[0572] 3. Display purchase confirmation message: When a wink is detected, a purchase confirmation message will be displayed on the TV screen.

[0573] User Actions

[0574] 1. TV viewing: Users watch TV and wait for products of interest to appear.

[0575] 2. Product selection: Wink at a specific product on the screen.

[0576] 3. Confirm your intention to purchase: In response to the confirmation message that appears, wink or press a key again to indicate your intention to purchase.

[0577] Specific examples

[0578] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0579] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[0580] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[0581] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[0582] 4. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[0583] 5. Purchase confirmation: A confirmation message will appear on the TV screen asking, "Are you sure you want to purchase this item?"

[0584] 6. Intent to buy: The user winks again to confirm their intent to buy.

[0585] 7. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[0586] As described above, the present invention provides a system that allows users to easily purchase products displayed on a television screen.

[0587] The processing flow will be explained below.

[0588] Step 1:

[0589] Terminal

[0590] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[0591] Step 2:

[0592] server

[0593] The server receives the video data sent from the TV terminal and saves it in high resolution.

[0594] Step 3:

[0595] server

[0596] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[0597] Step 4:

[0598] server

[0599] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[0600] Step 5:

[0601] Terminal

[0602] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[0603] Step 6:

[0604] Terminal

[0605] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[0606] Step 7:

[0607] Terminal

[0608] If a wink is detected, a purchase confirmation message will be displayed on the TV screen, asking "Are you sure you want to purchase this item?" and asking for a wink or keystroke for confirmation.

[0609] Step 8:

[0610] User

[0611] When a product of interest appears on the TV screen, the user turns their gaze towards the product and winks. After that, in response to the purchase confirmation message displayed on the TV screen, the user can wink again or use a keystroke to indicate their intention to purchase.

[0612] Step 9:

[0613] Terminal

[0614] The information confirming the user's intention to purchase is sent to the server, which is immediately notified and the purchase process begins.

[0615] Step 10:

[0616] server

[0617] Once the purchase confirmation is received, the system completes the purchase process, including verifying the user's address and payment method, and finally notifies the user that the purchase is complete.

[0618] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching television.

[0619] Example 1

[0620] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0621] While watching TV, users have no way to visually select and easily purchase products displayed, which makes the process of purchasing complicated and time-consuming. Furthermore, conventional methods make it difficult to track and quickly suggest products that interest users.

[0622] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0623] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for presenting the user with a confirmation of their intention to purchase the product based on the detection of the wink, means for completing the sales procedure after obtaining the confirmation of their intention, means for generating and displaying product recommendations for the user based on the results of image analysis by the generative model, means for displaying a confirmation message for a product selected by a wink and indicating their intention to purchase again by a wink or key operation, and means for sending a notification to the user after the purchase procedure is completed. This allows the user to visually easily select a product and quickly complete the procedure up to the purchase.

[0624] A "generative model" is a machine learning algorithm used to identify and extract specific objects and information from video data.

[0625] "Video data" refers to digital signals containing visual information transmitted by televisions and other visual devices.

[0626] An "eye-tracking device" is a device that detects the direction and movement of a user's gaze in real time.

[0627] "Wink detection" is a function that detects the user closing one eye and recognizes this action as a specific instruction.

[0628] "Confirming intention" is an act in which a user explicitly indicates their intention through a specific operation or action.

[0629] "Sales procedure" refers to the series of operations and processes required to purchase a product, including payment and delivery procedures.

[0630] "Recommendation information" is information that includes suggestions for specific products based on the user's preferences and purchase history.

[0631] A "visual display device" is a device, such as a television or monitor, that provides visual information to a user.

[0632] A "notification" is a message or alert sent by the system to convey information or results to a user.

[0633] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[0634] System configuration

[0635] The system of the present invention mainly comprises the following components:

[0636] 1. Server

[0637] 2. Devices (TV, eye tracking devices)

[0638] 3. Users

[0639] server

[0640] The server has the following roles:

[0641] Receiving video data: The server receives real-time video data sent from the TV terminal, for example, by streaming the video data using an HTTP request.

[0642] Analysis using a generative model: The server uses a generative model to identify the location, type, brand, and model number of products contained in the video data. Specifically, it runs a product recognition model using TensorFlow or PyTorch. An example of a prompt statement is, "Please run a model to identify the type, brand, and model number of the product shown in this video."

[0643] Recommendation generation: Based on the analysis results, the server generates product recommendations for the user and sends them to the device. During this process, a list of recommended products is generated that takes into account the user's purchase history and preferences.

[0644] Terminal

[0645] The device consists of a television and an eye-tracking device.

[0646] Capture of video data: Captures television images and sends them to the server.

[0647] Recommendation display: The TV device receives the recommendation information sent from the server and displays it as an overlay on the TV screen. For example, it displays the product image and detailed information along with the message "Do you like this product?"

[0648] Eye tracking and wink detection: An eye tracking device (e.g., Tobii Eye Tracker) tracks the user's gaze and detects winks toward specific products. Wink detection detects the action of closing one eye after the gaze has stopped on the product image for a certain period of time.

[0649] Displaying purchase intent confirmation: When a wink is detected, a confirmation message will be displayed on the TV screen asking, "Do you really want to purchase this product?" Specific purchase procedures and payment information will also be displayed.

[0650] User

[0651] The user performs the following actions:

[0652] TV viewing: Users watch TV and wait for products of interest to appear.

[0653] Product selection: Wink at a specific product on the screen.

[0654] Confirm your intention to purchase: In response to the displayed confirmation message, you can wink or press a key again to indicate your intention to purchase. For example, you can press the "OK" button on the remote control or wink again to indicate your intention to purchase.

[0655] Specific examples

[0656] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0657] 1. The server receives real-time video data sent from the TV terminal via an HTTP request. For example, the server retrieves video data from http: / / example.com / streaming-url.

[0658] 2. The server uses TensorFlow to identify the coffee maker brand, model number, etc. from the video data. For example, load the model with tf.keras.models.load_model('product_recognition_model.h5') and analyze the video data.

[0659] 3. The server generates recommendations for new coffee makers based on the user's past purchase history.

[0660] 4. The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, "The latest coffee makers" will pop up in the bottom right corner of the screen.

[0661] 5. The device uses the Tobii Eye Tracker to track the user's gaze and winks. After the user's gaze is focused on the image of the coffee maker, the device detects the closing of one eye.

[0662] 6. A message will appear on your TV screen asking, "Are you sure you want to purchase this item?" The message will be displayed large and in the center of the screen, along with payment options.

[0663] 7. The user confirms their intention to purchase by winking again or pressing the "OK" button on the remote control.

[0664] 8. The server calls the online store's API to complete the purchase. For example, it sends the user and product information to http: / / example-store.com / api / purchase.

[0665] 9. The server sends a purchase completion notification to the user. The message "Purchase Completed" is displayed on the TV screen, and a purchase completion notification is sent to the user's email address.

[0666] The above is an embodiment of the present invention. This system allows users to visually select products easily and quickly complete the procedures up to purchase.

[0667] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0668] Step 1:

[0669] Receiving video data

[0670] The server receives video data sent in real time from a TV terminal using an HTTP request. Specifically, the terminal captures the TV video and sends it to the server. The TV video data is input, and the received TV video data is saved on the server as output.

[0671] Step 2:

[0672] Image analysis using generative models

[0673] The server analyzes the received video data using a generative model (for example, a model using TensorFlow or PyTorch). This analysis identifies the location, type, brand, and model number of the product within the video data. The received video data is the input, and specific product information is obtained as the output.

[0674] An example of a prompt sentence is "Please run a model to identify the type, brand, and model number of the product shown in this video."

[0675] Step 3:

[0676] Recommendation generation

[0677] The server generates product recommendations for the user based on the product information identified by the generative model. The generated recommendation information is sent to the terminal. In this step, the user's preferences and purchase history are also taken into consideration. The input is product identification information and the user's purchase history, and the output is recommendation information.

[0678] Step 4:

[0679] Displaying recommendation information

[0680] The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, a product image and detailed information are displayed along with the message "Do you like this product?" The input is the recommendation information, and the output is a screen display that the user can visually confirm.

[0681] Step 5:

[0682] Eye Tracking and Wink Detection

[0683] An eye tracking device (for example, Tobii Eye Tracker) connected to the device tracks the user's gaze and detects winks at a specific product. Specifically, it detects when the gaze is focused on the product image and then the user closes one eye. The input is the user's gaze data, and the output is wink detection data.

[0684] Step 6:

[0685] Display of purchase intent confirmation

[0686] When the device detects a wink, it displays a confirmation message on the TV screen asking, "Do you really want to purchase this product?". Specific purchase procedures and payment information are also displayed. The input is the wink detection data, and the output is the confirmation message and purchase procedure information displayed on the screen.

[0687] Step 7:

[0688] Purchase Intention

[0689] The user indicates their intention to purchase by winking again or pressing the "OK" button on the remote control. Specifically, the user's operation is the input, and the data on their intention to purchase is obtained as the output.

[0690] Step 8:

[0691] Completing the purchase process

[0692] The server confirms the user's intent to purchase and begins the purchase process. Specifically, it calls the online store's API using the product information, as well as the user's address and payment information, to confirm the order. The inputs are data indicating the user's intent to purchase and user information, and the output is data indicating the purchase process has been completed.

[0693] Step 9:

[0694] Sending a purchase completion notification

[0695] After the purchase procedure is completed, the server sends a notification of purchase completion to the user. Specifically, it displays a message on the TV screen saying "Purchase completed" and sends a notification by email. The input is data indicating the purchase procedure has been completed, and the output is a completion notification sent to the user.

[0696] (Application example 1)

[0697] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0698] Conventional systems that identify products from video data and recommend them to users are primarily limited to television footage, with few examples of application in physical stores. Furthermore, systems that efficiently confirm a user's purchase intentions by using eye tracking and winks are also lacking. Furthermore, recommendation displays that are limited to television footage are not suitable for in-store environments, and a flexible system that can be used in a variety of store situations is desired. Given this background, the challenge is to provide a system that can be used in physical stores and improves the efficiency of the purchasing process by using eye tracking and wink detection.

[0699] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0700] In this invention, the server includes: means for identifying a product from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a confirmation of the user's intention to purchase the product based on the detection of the wink; means for completing the purchase procedure on the server after obtaining the confirmation of intention; means for converting the video data into in-store video data and performing eye-tracking through smart glasses or a head-mounted display; means for displaying a purchase confirmation message within the user's field of view after detecting a wink; and means for reconfirming the user's intention to purchase using voice recognition. This enables efficient product selection through eye-tracking and automated purchase procedures even in physical stores. Furthermore, the use of voice recognition allows the user to confirm their intention to purchase more intuitively and naturally.

[0701] A "generative model" is an algorithm that uses machine learning technology to automatically identify products from video data.

[0702] "Video data" refers to digital data of visual information captured by a camera or the like, including television footage and footage of inside a store.

[0703] An "eye tracking device" is a device that detects the movement of a user's eyes in real time and identifies the direction in which the user is looking.

[0704] "Wink detection" is a technology that uses an eye-tracking device or the like to detect when a user makes a specific action, i.e., a wink.

[0705] The "server" is a central computer system that analyzes video data, runs generative models, manages purchase procedures, etc.

[0706] "Smart glasses" are high-performance eyeglass devices that can be worn by users and are capable of eye tracking and augmented reality display.

[0707] A "head-mounted display" is a device worn on the user's head that displays visual information and tracks the user's gaze.

[0708] A "purchase confirmation message" is a message that is displayed when a user selects a product, to reconfirm the user's intention to purchase the product.

[0709] "Speech recognition" is a technology that recognizes a user's voice and converts it into digital information.

[0710] "Product recommendation" is a function that presents products suitable for the user based on product information analyzed by the generative model.

[0711] A system for implementing the present invention mainly comprises the following components: a server, a terminal (smart glasses or head-mounted display, eye-tracking device), and a user.

[0712] The server uses the generative model to identify products from the video data, tracks the user's gaze using an eye-tracking device, detects winks toward the specific products, and, based on the detected winks, prompts the user to confirm their intention to purchase the product, after which the purchase process is completed.

[0713] Specifically, the server has the following roles:

[0714] Receiving video data: Receive real-time video data transmitted from smart glasses or head-mounted displays.

[0715] Generative model analysis: Run a generative model to identify the location, type, and brand of products in the video data.

[0716] Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the device.

[0717] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[0718] The smart glasses or head-mounted display as a terminal plays the following roles:

[0719] Video data capture: Video from inside the physical store is captured in real time and sent to the server.

[0720] Recommendation display: Recommendation information sent from the server is displayed within the field of view.

[0721] Eye Tracking: The eye tracking device tracks the user's gaze in real time and detects winks towards specific products.

[0722] Confirmation of purchase intent: When a wink is detected, a purchase confirmation message is displayed within the field of view and voice recognition is used to prompt the user to reconfirm their intention to purchase.

[0723] The user performs the following actions:

[0724] Store Walking: Walking around the store and exploring products that interest you.

[0725] Product selection: Look at the product you are interested in and wink.

[0726] Confirm your intention to purchase: When the purchase confirmation message appears, wink or speak again to indicate your intention to purchase.

[0727] As a concrete example, consider a situation where a user is walking through a brick-and-mortar store and looking at a coffee maker on a shelf. The smart glasses capture video data and send it to a server. A generative model analyzes the coffee maker and identifies detailed information. An eye-tracking device tracks the user's gaze and detects that the user is focusing on the coffee maker. When the user winks, a confirmation message appears on the smart glasses asking, "Do you want to purchase this item?" The user can wink again or respond "yes" verbally to proceed with the purchase.

[0728] An example of a prompt for the generative AI model is as follows:

[0729] "Identify products from the following store video data and determine their location, type, and brand."

[0730] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0731] Step 1:

[0732] The server receives real-time video data transmitted from smart glasses or head-mounted displays. This input data is video data captured by cameras inside the store, and after being sent to the server, analysis begins.

[0733] Step 2:

[0734] The server inputs the received video data into a generative model, which analyzes the data and identifies the product's location, type, and brand. The specific processing of this data involves the generative model's algorithm identifying the product and generating metadata such as location information and brand name.

[0735] Step 3:

[0736] The server then generates product recommendations based on the generated metadata. For example, if a particular product has a high rating, it will be recommended first. This information is then sent to the smart glasses or head-mounted display.

[0737] Step 4:

[0738] The device (smart glasses or head-mounted display) displays the received recommendation information within the user's field of view. The displayed recommendation information is provided as an overlay on the video data.

[0739] Step 5:

[0740] The device tracks the user's gaze in real time using an eye-tracking device, which detects the user's eye movements and identifies which products the user's gaze is focused on.

[0741] Step 6:

[0742] The device detects when a user winks at a specific product. The wink detection algorithm analyzes the gaze detection data and recognizes the wink. This information is sent to the server.

[0743] Step 7:

[0744] The server receives the wink detection information, generates a purchase confirmation message for the specific product, and sends it to the terminal. The message includes a confirmation message such as "Do you want to purchase this product?"

[0745] Step 8:

[0746] The device displays a purchase confirmation message within the user's field of vision, and the user can respond to this message by winking again or saying "yes" or similar to indicate their intention to purchase.

[0747] Step 9:

[0748] The device then sends another wink or voice recognition result to the server, which is used to confirm the user's final purchase decision.

[0749] Step 10:

[0750] The server receives the final purchase confirmation data and executes the purchase procedure, including retrieving product information from the product database and completing the purchase. Finally, it sends a purchase completion notification to the user.

[0751] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0752] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[0753] System configuration

[0754] The system of the present invention consists of the following major components:

[0755] 1. Server

[0756] 2. Devices (TVs, eye tracking devices, emotion engines)

[0757] 3. Users

[0758] server

[0759] The server has the following roles:

[0760] Receiving video data: Receive real-time video data sent from the TV terminal.

[0761] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[0762] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device.

[0763] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[0764] Terminal

[0765] The device consists of a television, an eye-tracking device, and an emotion engine.

[0766] Capture of video data: Captures television images and sends them to the server.

[0767] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[0768] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[0769] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotions.

[0770] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[0771] User

[0772] The user performs the following actions:

[0773] Watching TV: Watch TV programs in the usual way.

[0774] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[0775] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[0776] Explanation of program processing

[0777] Server Processing

[0778] Receiving image data: The server receives the video data transmitted in real time from the television terminal.

[0779] Image analysis with generative models: Using generative models to identify the location, type, brand, and model number of products in video data.

[0780] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the device.

[0781] Terminal handling

[0782] Display of recommendation information: The TV terminal receives the recommendation information sent from the server and displays it as an overlay on the TV screen.

[0783] Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[0784] Emotion recognition: The emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[0785] Displaying purchase intent confirmation: A purchase confirmation message is displayed on the TV screen based on winks and emotion data.

[0786] User Actions

[0787] TV viewing: Users watch TV and wait for products of interest to appear.

[0788] Product Selection: Winks at specific products on the screen and expresses emotional state through emotion analysis by the emotion engine.

[0789] Confirm your intention to purchase: In response to the displayed confirmation message, wink or press a key again to indicate your intention to purchase.

[0790] Specific examples

[0791] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0792] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[0793] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[0794] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[0795] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions.

[0796] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[0797] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is favorable, the confirmation message will be highlighted.

[0798] 7. Intent to buy: The user winks again to confirm their intent to buy.

[0799] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[0800] As described above, the system of the present invention takes into account the user's emotions and provides a more personalized and advanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[0801] The processing flow will be explained below.

[0802] Step 1:

[0803] Terminal

[0804] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[0805] Step 2:

[0806] server

[0807] The server receives the video data sent from the TV terminal and saves it in high resolution.

[0808] Step 3:

[0809] server

[0810] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[0811] Step 4:

[0812] server

[0813] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[0814] Step 5:

[0815] Terminal

[0816] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[0817] Step 6:

[0818] Terminal

[0819] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[0820] Step 7:

[0821] Terminal

[0822] At the same time as detecting the wink, the emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotional state. If the user is interested and expressing positive emotions, the information is sent to the server.

[0823] Step 8:

[0824] server

[0825] The server receives the emotion data from the emotion engine and further optimizes the recommendation information based on the user's emotions. Based on the wink detection and emotion data, a purchase confirmation message is generated and sent to the device.

[0826] Step 9:

[0827] Terminal

[0828] The device displays a purchase confirmation message on the TV screen. The message asks the user to confirm whether they really want to purchase this product. Depending on the emotion data, the message may be highlighted.

[0829] Step 10:

[0830] User

[0831] Based on the confirmation message, the user can then wink or press a key again to indicate their intention to purchase. If their emotional state is appropriate for purchasing, they can wink again to indicate their intention to purchase.

[0832] Step 11:

[0833] Terminal

[0834] Once the user's intention to purchase is confirmed, the terminal transmits this information to the server and begins the purchase procedure.

[0835] Step 12:

[0836] server

[0837] The server receives the confirmation of intent to purchase and processes the product purchase, which may include verifying the user's address and payment method, connecting with back-end systems, etc. Finally, it notifies the user that the purchase is complete.

[0838] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching TV, and by utilizing the emotion engine, provides a more personalized purchasing experience.

[0839] Example 2

[0840] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0841] Conventional product purchasing systems that rely on eye gaze tracking and wink detection have the problem of not being able to fully consider the user's emotional state and therefore fail to optimize the user experience. Furthermore, false detection due to users unintentionally winking is also a problem. Therefore, there is a need to provide a more advanced product purchasing process that reflects the user's emotional state, rather than simply detecting eye gaze and winks.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0843] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for analyzing the user's emotions using an emotion analysis device, means for presenting a purchase confirmation to the user based on the wink and the results of the emotion analysis, and means for completing the purchase procedure on the server after receiving the confirmation, thereby providing a product purchase process that takes the user's emotional state into account, reducing false positives and enabling a more personalized purchasing experience.

[0844] A "generative model" is an algorithm used to analyze video data to identify or classify specific items.

[0845] An "eye tracking device" is a device that detects the movement of a user's eyes and analyzes where the eyes are looking.

[0846] An "emotion analysis device" is a device that analyzes a user's facial expressions and voice patterns to identify the user's emotional state.

[0847] A "wink" is a user action in which one or both eyes are temporarily closed for a specific purpose.

[0848] "Product purchase intent confirmation" is the process of confirming whether a user intends to purchase a particular product based on a wink or other confirmation means.

[0849] A "server" is a computer system that receives, transmits, analyzes, and processes various types of data on a network.

[0850] "Video data" refers to real-time or existing video content such as television programs and videos.

[0851] "Recommendation information" is information that suggests optimal products and services to users based on the analysis results and the user's emotional data.

[0852] MODE FOR CARRYING OUT THE INVENTION

[0853] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[0854] System configuration

[0855] The system of the present invention consists of the following major components:

[0856] 1. Server

[0857] 2. Devices (TVs, eye tracking devices, emotion engines)

[0858] 3. Users

[0859] server

[0860] The server has the following roles:

[0861] Receiving video data: Receive real-time video data sent from the TV terminal.

[0862] Analysis using a generative model: A generative model is run to identify the location, type, brand, and model number of products contained in the video data. The generative model used here is "YOLO (You Only Look Once)."

[0863] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device. Microsoft Azure Cognitive Services is used to analyze the emotional data.

[0864] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[0865] Terminal

[0866] The device consists of a television, an eye-tracking device, and an emotion engine.

[0867] Capture of video data: Captures television images and sends them to the server.

[0868] Recommendation display: Recommendation information sent from the server is displayed on the TV screen. Game engines such as "Unity" are used for display.

[0869] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products. The device used here is the "Tobii Eye Tracker."

[0870] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to generate emotion data. The "Affectiva SDK" is used to analyze the user's emotions.

[0871] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[0872] User

[0873] The user performs the following actions:

[0874] Watching TV: Watch TV programs in the usual way.

[0875] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[0876] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[0877] Specific examples

[0878] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[0879] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[0880] 2. Image analysis: The server uses the generative model to identify the brand, model number, etc. of the coffee maker. In this process, the prompt statement is used: "Analyze a video of a cooking show and identify the brand and model number of the product shown."

[0881] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[0882] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions, using the prompt "Please recognize the user's emotional state and prioritize recommendations."

[0883] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[0884] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is positive, a prompt will be set saying, "If the user's emotion is positive, highlight the confirmation message."

[0885] 7. Intent to buy: The user winks again to confirm their intent to buy.

[0886] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[0887] In this way, the system of the present invention takes into account the user's emotions and provides a more personalized and enhanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[0888] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0889] Program processing flow

[0890] Server Processing

[0891] Step 1: Receiving video data

[0892] The server receives video data transmitted in real time from the television terminal.

[0893] Input: Real-time video data from a TV terminal

[0894] Output: Received video data

[0895] Specifically, the server receives a video data stream via a network.

[0896] Step 2: Image analysis

[0897] The server analyzes the received video data and uses the generative model "YOLO (You Only Look Once)" to identify the product's location, type, brand, and model number.

[0898] Input: Video data

[0899] Output: Product location, type, brand, and model number information

[0900] After the video data is sent to the server, a generative model is run to extract and classify features of objects in the image.

[0901] Step 3: Integrating Emotional Data

[0902] The server receives the user's emotion data obtained from the emotion engine and integrates it with the image analysis results. Emotion analysis is performed using Microsoft Azure Cognitive Services.

[0903] Input: Image analysis results, emotion data

[0904] Output: Consolidated data

[0905] The server combines the analyzed product information with the user's emotional data and integrates the information necessary to generate recommendations.

[0906] Step 4: Recommendation generation

[0907] The server generates product recommendations for the user based on the image analysis results and emotional data and sends them to the terminal.

[0908] Input: Integrated data

[0909] Output: Recommendation information

[0910] The server selects the most suitable products based on the integrated data and generates specific recommendation information.

[0911] Step 5: Checkout

[0912] The server confirms the user's intention to purchase and completes the purchase procedure. The server notifies the user that the purchase procedure is complete.

[0913] Input: Confirm purchase intention

[0914] Output: Purchase completion notification

[0915] The server receives confirmation of the user's intention to purchase, processes the purchase, and notifies the user of the result.

[0916] Terminal handling

[0917] Step 1: Capture video data

[0918] The device captures the TV image and sends it to the server.

[0919] Input: TV program video

[0920] Output: Captured video data

[0921] Specifically, video data is acquired from a television screen and sent to a server as a stream.

[0922] Step 2: Displaying recommendation information

[0923] The terminal receives the recommendation information sent from the server and displays it as an overlay on the television screen.

[0924] Input: Recommendation information

[0925] Output: Displayed recommendation information

[0926] The received information is analyzed, and a user interface is generated and displayed on the screen.

[0927] Step 3: Eye tracking and wink detection

[0928] The eye tracking device, Tobii Eye Tracker, tracks the user's gaze and detects winks at specific products.

[0929] Input: User gaze data

[0930] Output: Wink detection result

[0931] The device tracks the user's gaze in real time, detects winks, and notifies the server.

[0932] Step 4: Emotion Recognition

[0933] The emotion engine analyzes the user's facial expressions and voice to generate emotion data. The user's emotions are analyzed using the "Affectiva SDK."

[0934] Input: User's facial expression and voice data

[0935] Output: Emotion data

[0936] Analyzed emotion data is generated and this information is sent to a server.

[0937] Step 5: Confirm your intention to purchase

[0938] A purchase confirmation message is displayed on the TV screen based on wink and emotion data.

[0939] Input: Wink detection results, emotion data

[0940] Output: Purchase confirmation message

[0941] Based on the analysis results, a confirmation message is overlaid on the screen.

[0942] User Actions

[0943] Step 1: Watch TV

[0944] The user watches the television program in the usual way.

[0945] Input: TV show

[0946] Output: User viewing behavior

[0947] Operate and watch the TV to view the program on the screen.

[0948] Step 2: Product selection

[0949] When a product that interests the user appears on the screen, the user turns their gaze and winks. The emotion engine analyzes the user's interest from their reaction.

[0950] Input: On-screen item

[0951] Output: Gaze data, wink, emotion data

[0952] The system detects gazes and winks towards the product and sends them to the server along with emotional data.

[0953] Step 3: Confirm your intention to purchase

[0954] In response to the purchase confirmation message, the user indicates their intention to purchase by winking again or pressing a specific key.

[0955] Input: Purchase confirmation message

[0956] Output: Re-wink, key operation

[0957] View the confirmation message and take the appropriate action to indicate your intent to purchase.

[0958] The above is a specific explanation of each processing step.

[0959] (Application example 2)

[0960] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0961] In online shopping, conventional methods have limited ability to quickly and intuitively confirm a user's purchasing intention, and they are unable to recommend products that take into account the user's emotional state. This can lead to a poor user experience and a decrease in purchasing motivation. To solve this problem, an advanced recommendation system that combines eye tracking and emotion recognition is needed.

[0962] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0963] In this invention, the server includes: means for identifying products from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a product purchase confirmation to the user based on the wink detection; means for completing the purchase procedure on the server after receiving the confirmation; means for analyzing the user's emotions using an emotion recognition engine and adjusting / emphasizing the product recommendation and confirmation message based on the emotion data; means for performing eye-tracking, emotion recognition, and wink detection using the smartphone's front camera and microphone; and means for overlaying recommendation information on the online shop screen and displaying a purchase confirmation message based on the emotion data, thereby enabling an intuitive and personalized purchasing experience that takes into account the user's gaze and emotional state.

[0964] A "generative model" is a machine learning algorithm for extracting and analyzing specific information from video data.

[0965] "Video data" refers to image and video data acquired from a photographing device such as a camera.

[0966] "Product" is a term that generally refers to products and services sold in online shops, etc.

[0967] An "eye tracking device" is a device that uses a camera or sensor to detect and track the position of a user's gaze.

[0968] "User" refers to the consumer or user of the System.

[0969] "Wink detection" is the process of using a sensor or camera to recognize when a user closes one of their eyes.

[0970] "Confirming intention" is an act in which a user confirms his or her own intentions or wishes through a specific operation or action.

[0971] A "server" is a computer system that provides services to multiple clients.

[0972] An "emotion recognition engine" is a technology for analyzing and estimating a user's emotional state from facial expressions, voice, etc.

[0973] "Recommendation" is the act of the system presenting products or information that are appropriate and likely to interest the user.

[0974] A "smartphone" is a multi-functional mobile device with advanced computing capabilities.

[0975] A "front camera" is a camera installed on the screen side of a smartphone or tablet.

[0976] A "microphone" is a device that picks up sound and converts it into an electrical signal.

[0977] "Overlay display" is a method of displaying additional information over an existing image or video.

[0978] An "online shop" is a website or application that sells products or services over the Internet.

[0979] The present invention provides a system that provides an advanced product recommendation and purchasing process that takes into account the user's gaze and emotional state. This system consists of the following main components: a server, a terminal (the user's smartphone), and a user. Next, we will explain the role of each component and its specific implementation.

[0980] server

[0981] The server has the following features:

[0982] 1. Receiving image data: The server receives real-time video data sent from the smartphone terminal.

[0983] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data. Generative models can be created using TensorFlow or Google Cloud Vision API.

[0984] 3. Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the smartphone device.

[0985] 4. Completing the purchase process: Based on the user's purchase intent and emotional state, the purchase process is completed using the Stripe API or similar.

[0986] Device (smartphone)

[0987] The smartphone has the following features:

[0988] 1. Capture video data: Capture video data from the smartphone's front camera and send it to the server.

[0989] 2. Displaying recommendation information: The recommendation information sent from the server is overlaid on the online shop screen.

[0990] 3. Eye Tracking and Wink Detection: Use the front camera to track the user's gaze and detect winks towards specific products. This is done using OpenCV.

[0991] 4. Emotion Recognition: Using the front camera and microphone, the system analyzes the user's facial expressions and voice patterns to generate emotion data. The emotion recognition engine is DeepFace.

[0992] 5. Confirmation of purchase intention: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[0993] User

[0994] The user performs the following actions:

[0995] 1. Online Shopping Browsing: Browse the online shop page in the usual way.

[0996] 2. Product Selection: When a product of interest appears on the screen, the robot turns its gaze and winks, and also expresses its emotions through facial expressions and voice.

[0997] 3. Confirming purchase intent: In response to a confirmation message displayed based on a wink or emotional state, the user can indicate their intention to purchase by winking again or using a specific finger gesture.

[0998] Specific examples

[0999] For example, consider a user browsing the latest smartphone models in an online shop. The server receives video data sent from the smartphone and uses a generative model to identify product information. The server generates recommendation information and overlays it on the smartphone screen. Meanwhile, the front camera and microphone are used to track the user's gaze and emotional state, and an emotion recognition engine recognizes that the user is pleased. If the user winks at the product, a purchase confirmation message is displayed, and the user confirms their intention to purchase with another wink, and payment is completed via the Stripe API.

[1000] Prompt Sentence Examples

[1001] Capture video from the front camera

[1002] Face detection and eye tracking initialization

[1003] Eye tracking and wink detection

[1004] Emotion Recognition and Recommendation Generation

[1005] Display purchase confirmation message

[1006] Wink again to confirm your intention to purchase

[1007] Completing the purchase process

[1008] This allows users to enjoy an intuitive and emotional purchasing experience.

[1009] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1010] Step 1:

[1011] Video data capture

[1012] The device uses the smartphone's front camera to capture video data of the online shop page the user is viewing. This video data is sent to the server in real time. The video data is the input, and the data sent to the server is the output.

[1013] Step 2:

[1014] Image analysis using generative models

[1015] The server runs a generative model based on the video data received from the device to identify the location, type, brand, and model number of the product in the video data. Generative models such as TensorFlow and Google Cloud Vision API are used. The input is the received video data, and the output is detailed product data such as the product's location and type.

[1016] Step 3:

[1017] Eye tracking and wink detection

[1018] The device uses the front camera to track the user's gaze and OpenCV to determine whether the user is looking at a specific product. It also runs a wink detection algorithm to detect winks toward a specific product. The input is camera image data, and the output is gaze information and wink detection results.

[1019] Step 4:

[1020] emotion recognition

[1021] The device uses the front camera and microphone to capture the user's facial expressions and voice, which are then analyzed by an emotion recognition engine such as DeepFace. This generates user emotion data. The input is facial expression data and voice data, and the output is the emotion recognition result.

[1022] Step 5:

[1023] Generating and displaying recommendation information

[1024] The server combines the gaze tracking results, wink detection results, and emotion data to generate product recommendation information. This information is then sent to the device, which then displays the recommendation information as an overlay on the online shop screen. The inputs are gaze information, wink detection results, and emotion data, and the output is recommendation information.

[1025] Step 6:

[1026] Confirmation of purchase intention

[1027] The device displays a product purchase confirmation message to the user based on the wink detection and emotion recognition results. The user confirms their intention to purchase by winking again or performing a specific finger gesture. The inputs are the wink detection results and emotion recognition results, and the output is the user's intention to purchase.

[1028] Step 7:

[1029] Completing the purchase process

[1030] The server receives confirmation of the user's intent to purchase and completes the purchase procedure using the Stripe API or similar. This confirms the product purchase and sends a purchase completion notification to the user. The input is the user's confirmation of intent to purchase, and the output is purchase confirmation and a purchase completion notification.

[1031] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1032] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1033] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1034] [Third embodiment]

[1035] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1036] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1037] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1038] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1039] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1040] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1041] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1042] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1043] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1044] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1045] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1046] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1047] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[1048] System configuration

[1049] The system of the present invention mainly comprises the following components:

[1050] 1. Server

[1051] 2. Devices (TV, eye tracking devices)

[1052] 3. Users

[1053] server

[1054] The server has the following roles:

[1055] Receiving video data: Receive real-time video data sent from the TV terminal.

[1056] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[1057] Recommendation generation: Using the analysis results, product recommendations for the user are generated and sent to the device.

[1058] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[1059] Terminal

[1060] The device consists of a television and an eye-tracking device.

[1061] Capture of video data: Captures television images and sends them to the server.

[1062] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[1063] Eye tracking: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[1064] Confirmation of purchase intent: When a wink is detected, a confirmation message for purchasing the product is displayed to the user.

[1065] User

[1066] The user performs the following actions:

[1067] Watching TV: Watch TV programs in the usual way.

[1068] Product selection: When a product of interest appears on the screen, look at it and wink.

[1069] Confirm your intention to purchase: After the wink, a confirmation message appears, and you can indicate your intention to purchase by winking again or pressing a specific key.

[1070] Explanation of program processing

[1071] Server Processing

[1072] 1. Receiving image data: The server receives video data transmitted in real time from the TV terminal.

[1073] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data.

[1074] 3. Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the terminal.

[1075] Terminal handling

[1076] 1. Displaying recommended information: The TV terminal receives the recommended information sent from the server and displays it as an overlay on the TV screen.

[1077] 2. Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[1078] 3. Display purchase confirmation message: When a wink is detected, a purchase confirmation message will be displayed on the TV screen.

[1079] User Actions

[1080] 1. TV viewing: Users watch TV and wait for products of interest to appear.

[1081] 2. Product selection: Wink at a specific product on the screen.

[1082] 3. Confirm your intention to purchase: In response to the confirmation message that appears, wink or press a key again to indicate your intention to purchase.

[1083] Specific examples

[1084] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1085] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[1086] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[1087] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[1088] 4. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[1089] 5. Purchase confirmation: A confirmation message will appear on the TV screen asking, "Are you sure you want to purchase this item?"

[1090] 6. Intent to buy: The user winks again to confirm their intent to buy.

[1091] 7. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[1092] As described above, the present invention provides a system that allows users to easily purchase products displayed on a television screen.

[1093] The processing flow will be explained below.

[1094] Step 1:

[1095] Terminal

[1096] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[1097] Step 2:

[1098] server

[1099] The server receives the video data sent from the TV terminal and saves it in high resolution.

[1100] Step 3:

[1101] server

[1102] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[1103] Step 4:

[1104] server

[1105] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[1106] Step 5:

[1107] Terminal

[1108] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[1109] Step 6:

[1110] Terminal

[1111] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[1112] Step 7:

[1113] Terminal

[1114] If a wink is detected, a purchase confirmation message will be displayed on the TV screen, asking "Are you sure you want to purchase this item?" and asking for a wink or keystroke for confirmation.

[1115] Step 8:

[1116] User

[1117] When a product of interest appears on the TV screen, the user turns their gaze towards the product and winks. After that, in response to the purchase confirmation message displayed on the TV screen, the user can wink again or use a keystroke to indicate their intention to purchase.

[1118] Step 9:

[1119] Terminal

[1120] The information confirming the user's intention to purchase is sent to the server, which is immediately notified and the purchase process begins.

[1121] Step 10:

[1122] server

[1123] Once the purchase confirmation is received, the system completes the purchase process, including verifying the user's address and payment method, and finally notifies the user that the purchase is complete.

[1124] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching television.

[1125] Example 1

[1126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1127] While watching TV, users have no way to visually select and easily purchase products displayed, which makes the process of purchasing complicated and time-consuming. Furthermore, conventional methods make it difficult to track and quickly suggest products that interest users.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1129] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for presenting the user with a confirmation of their intention to purchase the product based on the detection of the wink, means for completing the sales procedure after obtaining the confirmation of their intention, means for generating and displaying product recommendations for the user based on the results of image analysis by the generative model, means for displaying a confirmation message for a product selected by a wink and indicating their intention to purchase again by a wink or key operation, and means for sending a notification to the user after the purchase procedure is completed. This allows the user to visually easily select a product and quickly complete the procedure up to the purchase.

[1130] A "generative model" is a machine learning algorithm used to identify and extract specific objects and information from video data.

[1131] "Video data" refers to digital signals containing visual information transmitted by televisions and other visual devices.

[1132] An "eye-tracking device" is a device that detects the direction and movement of a user's gaze in real time.

[1133] "Wink detection" is a function that detects the user closing one eye and recognizes this action as a specific instruction.

[1134] "Confirming intention" is an act in which a user explicitly indicates their intention through a specific operation or action.

[1135] "Sales procedure" refers to the series of operations and processes required to purchase a product, including payment and delivery procedures.

[1136] "Recommendation information" is information that includes suggestions for specific products based on the user's preferences and purchase history.

[1137] A "visual display device" is a device, such as a television or monitor, that provides visual information to a user.

[1138] A "notification" is a message or alert sent by the system to convey information or results to a user.

[1139] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[1140] System configuration

[1141] The system of the present invention mainly comprises the following components:

[1142] 1. Server

[1143] 2. Devices (TV, eye tracking devices)

[1144] 3. Users

[1145] server

[1146] The server has the following roles:

[1147] Receiving video data: The server receives real-time video data sent from the TV terminal, for example, by streaming the video data using an HTTP request.

[1148] Analysis using a generative model: The server uses a generative model to identify the location, type, brand, and model number of products contained in the video data. Specifically, it runs a product recognition model using TensorFlow or PyTorch. An example of a prompt statement is, "Please run a model to identify the type, brand, and model number of the product shown in this video."

[1149] Recommendation generation: Based on the analysis results, the server generates product recommendations for the user and sends them to the device. During this process, a list of recommended products is generated that takes into account the user's purchase history and preferences.

[1150] Terminal

[1151] The device consists of a television and an eye-tracking device.

[1152] Capture of video data: Captures television images and sends them to the server.

[1153] Recommendation display: The TV device receives the recommendation information sent from the server and displays it as an overlay on the TV screen. For example, it displays the product image and detailed information along with the message "Do you like this product?"

[1154] Eye tracking and wink detection: An eye tracking device (e.g., Tobii Eye Tracker) tracks the user's gaze and detects winks toward specific products. Wink detection detects the action of closing one eye after the gaze has stopped on the product image for a certain period of time.

[1155] Displaying purchase intent confirmation: When a wink is detected, a confirmation message will be displayed on the TV screen asking, "Do you really want to purchase this product?" Specific purchase procedures and payment information will also be displayed.

[1156] User

[1157] The user performs the following actions:

[1158] TV viewing: Users watch TV and wait for products of interest to appear.

[1159] Product selection: Wink at a specific product on the screen.

[1160] Confirm your intention to purchase: In response to the displayed confirmation message, you can wink or press a key again to indicate your intention to purchase. For example, you can press the "OK" button on the remote control or wink again to indicate your intention to purchase.

[1161] Specific examples

[1162] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1163] 1. The server receives real-time video data sent from the TV terminal via an HTTP request. For example, the server retrieves video data from http: / / example.com / streaming-url.

[1164] 2. The server uses TensorFlow to identify the coffee maker brand, model number, etc. from the video data. For example, load the model with tf.keras.models.load_model('product_recognition_model.h5') and analyze the video data.

[1165] 3. The server generates recommendations for new coffee makers based on the user's past purchase history.

[1166] 4. The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, "The latest coffee makers" will pop up in the bottom right corner of the screen.

[1167] 5. The device uses the Tobii Eye Tracker to track the user's gaze and winks. After the user's gaze is focused on the image of the coffee maker, the device detects the closing of one eye.

[1168] 6. A message will appear on your TV screen asking, "Are you sure you want to purchase this item?" The message will be displayed large and in the center of the screen, along with payment options.

[1169] 7. The user confirms their intention to purchase by winking again or pressing the "OK" button on the remote control.

[1170] 8. The server calls the online store's API to complete the purchase. For example, it sends the user and product information to http: / / example-store.com / api / purchase.

[1171] 9. The server sends a purchase completion notification to the user. The message "Purchase Completed" is displayed on the TV screen, and a purchase completion notification is sent to the user's email address.

[1172] The above is an embodiment of the present invention. This system allows users to visually select products easily and quickly complete the procedures up to purchase.

[1173] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1174] Step 1:

[1175] Receiving video data

[1176] The server receives video data sent in real time from a TV terminal using an HTTP request. Specifically, the terminal captures the TV video and sends it to the server. The TV video data is input, and the received TV video data is saved on the server as output.

[1177] Step 2:

[1178] Image analysis using generative models

[1179] The server analyzes the received video data using a generative model (for example, a model using TensorFlow or PyTorch). This analysis identifies the location, type, brand, and model number of the product within the video data. The received video data is the input, and specific product information is obtained as the output.

[1180] An example of a prompt sentence is "Please run a model to identify the type, brand, and model number of the product shown in this video."

[1181] Step 3:

[1182] Recommendation generation

[1183] The server generates product recommendations for the user based on the product information identified by the generative model. The generated recommendation information is sent to the terminal. In this step, the user's preferences and purchase history are also taken into consideration. The input is product identification information and the user's purchase history, and the output is recommendation information.

[1184] Step 4:

[1185] Displaying recommendation information

[1186] The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, a product image and detailed information are displayed along with the message "Do you like this product?" The input is the recommendation information, and the output is a screen display that the user can visually confirm.

[1187] Step 5:

[1188] Eye Tracking and Wink Detection

[1189] An eye tracking device (for example, Tobii Eye Tracker) connected to the device tracks the user's gaze and detects winks at a specific product. Specifically, it detects when the gaze is focused on the product image and then the user closes one eye. The input is the user's gaze data, and the output is wink detection data.

[1190] Step 6:

[1191] Display of purchase intent confirmation

[1192] When the device detects a wink, it displays a confirmation message on the TV screen asking, "Do you really want to purchase this product?". Specific purchase procedures and payment information are also displayed. The input is the wink detection data, and the output is the confirmation message and purchase procedure information displayed on the screen.

[1193] Step 7:

[1194] Purchase Intention

[1195] The user indicates their intention to purchase by winking again or pressing the "OK" button on the remote control. Specifically, the user's operation is the input, and the data on their intention to purchase is obtained as the output.

[1196] Step 8:

[1197] Completing the purchase process

[1198] The server confirms the user's intent to purchase and begins the purchase process. Specifically, it calls the online store's API using the product information, as well as the user's address and payment information, to confirm the order. The inputs are data indicating the user's intent to purchase and user information, and the output is data indicating the purchase process has been completed.

[1199] Step 9:

[1200] Sending a purchase completion notification

[1201] After the purchase procedure is completed, the server sends a notification of purchase completion to the user. Specifically, it displays a message on the TV screen saying "Purchase completed" and sends a notification by email. The input is data indicating the purchase procedure has been completed, and the output is a completion notification sent to the user.

[1202] (Application example 1)

[1203] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1204] Conventional systems that identify products from video data and recommend them to users are primarily limited to television footage, with few examples of application in physical stores. Furthermore, systems that efficiently confirm a user's purchase intentions by using eye tracking and winks are also lacking. Furthermore, recommendation displays that are limited to television footage are not suitable for in-store environments, and a flexible system that can be used in a variety of store situations is desired. Given this background, the challenge is to provide a system that can be used in physical stores and improves the efficiency of the purchasing process by using eye tracking and wink detection.

[1205] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1206] In this invention, the server includes: means for identifying a product from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a confirmation of the user's intention to purchase the product based on the detection of the wink; means for completing the purchase procedure on the server after obtaining the confirmation of intention; means for converting the video data into in-store video data and performing eye-tracking through smart glasses or a head-mounted display; means for displaying a purchase confirmation message within the user's field of view after detecting a wink; and means for reconfirming the user's intention to purchase using voice recognition. This enables efficient product selection through eye-tracking and automated purchase procedures even in physical stores. Furthermore, the use of voice recognition allows the user to confirm their intention to purchase more intuitively and naturally.

[1207] A "generative model" is an algorithm that uses machine learning technology to automatically identify products from video data.

[1208] "Video data" refers to digital data of visual information captured by a camera or the like, including television footage and footage of inside a store.

[1209] An "eye tracking device" is a device that detects the movement of a user's eyes in real time and identifies the direction in which the user is looking.

[1210] "Wink detection" is a technology that uses an eye-tracking device or the like to detect when a user makes a specific action, i.e., a wink.

[1211] The "server" is a central computer system that analyzes video data, runs generative models, manages purchase procedures, etc.

[1212] "Smart glasses" are high-performance eyeglass devices that can be worn by users and are capable of eye tracking and augmented reality display.

[1213] A "head-mounted display" is a device worn on the user's head that displays visual information and tracks the user's gaze.

[1214] A "purchase confirmation message" is a message that is displayed when a user selects a product, to reconfirm the user's intention to purchase the product.

[1215] "Speech recognition" is a technology that recognizes a user's voice and converts it into digital information.

[1216] "Product recommendation" is a function that presents products suitable for the user based on product information analyzed by the generative model.

[1217] A system for implementing the present invention mainly comprises the following components: a server, a terminal (smart glasses or head-mounted display, eye-tracking device), and a user.

[1218] The server uses the generative model to identify products from the video data, tracks the user's gaze using an eye-tracking device, detects winks toward the specific products, and, based on the detected winks, prompts the user to confirm their intention to purchase the product, after which the purchase process is completed.

[1219] Specifically, the server has the following roles:

[1220] Receiving video data: Receive real-time video data transmitted from smart glasses or head-mounted displays.

[1221] Generative model analysis: Run a generative model to identify the location, type, and brand of products in the video data.

[1222] Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the device.

[1223] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[1224] The smart glasses or head-mounted display as a terminal plays the following roles:

[1225] Video data capture: Video from inside the physical store is captured in real time and sent to the server.

[1226] Recommendation display: Recommendation information sent from the server is displayed within the field of view.

[1227] Eye Tracking: The eye tracking device tracks the user's gaze in real time and detects winks towards specific products.

[1228] Confirmation of purchase intent: When a wink is detected, a purchase confirmation message is displayed within the field of view and voice recognition is used to prompt the user to reconfirm their intention to purchase.

[1229] The user performs the following actions:

[1230] Store Walking: Walking around the store and exploring products that interest you.

[1231] Product selection: Look at the product you are interested in and wink.

[1232] Confirm your intention to purchase: When the purchase confirmation message appears, wink or speak again to indicate your intention to purchase.

[1233] As a concrete example, consider a situation where a user is walking through a brick-and-mortar store and looking at a coffee maker on a shelf. The smart glasses capture video data and send it to a server. A generative model analyzes the coffee maker and identifies detailed information. An eye-tracking device tracks the user's gaze and detects that the user is focusing on the coffee maker. When the user winks, a confirmation message appears on the smart glasses asking, "Do you want to purchase this item?" The user can wink again or respond "yes" verbally to proceed with the purchase.

[1234] An example of a prompt for the generative AI model is as follows:

[1235] "Identify products from the following store video data and determine their location, type, and brand."

[1236] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1237] Step 1:

[1238] The server receives real-time video data transmitted from smart glasses or head-mounted displays. This input data is video data captured by cameras inside the store, and after being sent to the server, analysis begins.

[1239] Step 2:

[1240] The server inputs the received video data into a generative model, which analyzes the data and identifies the product's location, type, and brand. The specific processing of this data involves the generative model's algorithm identifying the product and generating metadata such as location information and brand name.

[1241] Step 3:

[1242] The server then generates product recommendations based on the generated metadata. For example, if a particular product has a high rating, it will be recommended first. This information is then sent to the smart glasses or head-mounted display.

[1243] Step 4:

[1244] The device (smart glasses or head-mounted display) displays the received recommendation information within the user's field of view. The displayed recommendation information is provided as an overlay on the video data.

[1245] Step 5:

[1246] The device tracks the user's gaze in real time using an eye-tracking device, which detects the user's eye movements and identifies which products the user's gaze is focused on.

[1247] Step 6:

[1248] The device detects when a user winks at a specific product. The wink detection algorithm analyzes the gaze detection data and recognizes the wink. This information is sent to the server.

[1249] Step 7:

[1250] The server receives the wink detection information, generates a purchase confirmation message for the specific product, and sends it to the terminal. The message includes a confirmation message such as "Do you want to purchase this product?"

[1251] Step 8:

[1252] The device displays a purchase confirmation message within the user's field of vision, and the user can respond to this message by winking again or saying "yes" or similar to indicate their intention to purchase.

[1253] Step 9:

[1254] The device then sends another wink or voice recognition result to the server, which is used to confirm the user's final purchase decision.

[1255] Step 10:

[1256] The server receives the final purchase confirmation data and executes the purchase procedure, including retrieving product information from the product database and completing the purchase. Finally, it sends a purchase completion notification to the user.

[1257] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1258] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[1259] System configuration

[1260] The system of the present invention consists of the following major components:

[1261] 1. Server

[1262] 2. Devices (TVs, eye tracking devices, emotion engines)

[1263] 3. Users

[1264] server

[1265] The server has the following roles:

[1266] Receiving video data: Receive real-time video data sent from the TV terminal.

[1267] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[1268] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device.

[1269] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[1270] Terminal

[1271] The device consists of a television, an eye-tracking device, and an emotion engine.

[1272] Capture of video data: Captures television images and sends them to the server.

[1273] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[1274] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[1275] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotions.

[1276] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[1277] User

[1278] The user performs the following actions:

[1279] Watching TV: Watch TV programs in the usual way.

[1280] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[1281] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[1282] Explanation of program processing

[1283] Server Processing

[1284] Receiving image data: The server receives the video data transmitted in real time from the television terminal.

[1285] Image analysis with generative models: Using generative models to identify the location, type, brand, and model number of products in video data.

[1286] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the device.

[1287] Terminal handling

[1288] Display of recommendation information: The TV terminal receives the recommendation information sent from the server and displays it as an overlay on the TV screen.

[1289] Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[1290] Emotion recognition: The emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[1291] Displaying purchase intent confirmation: A purchase confirmation message is displayed on the TV screen based on winks and emotion data.

[1292] User Actions

[1293] TV viewing: Users watch TV and wait for products of interest to appear.

[1294] Product Selection: Winks at specific products on the screen and expresses emotional state through emotion analysis by the emotion engine.

[1295] Confirm your intention to purchase: In response to the displayed confirmation message, wink or press a key again to indicate your intention to purchase.

[1296] Specific examples

[1297] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1298] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[1299] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[1300] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[1301] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions.

[1302] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[1303] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is favorable, the confirmation message will be highlighted.

[1304] 7. Intent to buy: The user winks again to confirm their intent to buy.

[1305] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[1306] As described above, the system of the present invention takes into account the user's emotions and provides a more personalized and advanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[1307] The processing flow will be explained below.

[1308] Step 1:

[1309] Terminal

[1310] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[1311] Step 2:

[1312] server

[1313] The server receives the video data sent from the TV terminal and saves it in high resolution.

[1314] Step 3:

[1315] server

[1316] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[1317] Step 4:

[1318] server

[1319] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[1320] Step 5:

[1321] Terminal

[1322] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[1323] Step 6:

[1324] Terminal

[1325] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[1326] Step 7:

[1327] Terminal

[1328] At the same time as detecting the wink, the emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotional state. If the user is interested and expressing positive emotions, the information is sent to the server.

[1329] Step 8:

[1330] server

[1331] The server receives the emotion data from the emotion engine and further optimizes the recommendation information based on the user's emotions. Based on the wink detection and emotion data, a purchase confirmation message is generated and sent to the device.

[1332] Step 9:

[1333] Terminal

[1334] The device displays a purchase confirmation message on the TV screen. The message asks the user to confirm whether they really want to purchase this product. Depending on the emotion data, the message may be highlighted.

[1335] Step 10:

[1336] User

[1337] Based on the confirmation message, the user can then wink or press a key again to indicate their intention to purchase. If their emotional state is appropriate for purchasing, they can wink again to indicate their intention to purchase.

[1338] Step 11:

[1339] Terminal

[1340] Once the user's intention to purchase is confirmed, the terminal transmits this information to the server and begins the purchase procedure.

[1341] Step 12:

[1342] server

[1343] The server receives the confirmation of intent to purchase and processes the product purchase, which may include verifying the user's address and payment method, connecting with back-end systems, etc. Finally, it notifies the user that the purchase is complete.

[1344] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching TV, and by utilizing the emotion engine, provides a more personalized purchasing experience.

[1345] Example 2

[1346] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1347] Conventional product purchasing systems that rely on eye gaze tracking and wink detection have the problem of not being able to fully consider the user's emotional state and therefore fail to optimize the user experience. Furthermore, false detection due to users unintentionally winking is also a problem. Therefore, there is a need to provide a more advanced product purchasing process that reflects the user's emotional state, rather than simply detecting eye gaze and winks.

[1348] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1349] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for analyzing the user's emotions using an emotion analysis device, means for presenting a purchase confirmation to the user based on the wink and the results of the emotion analysis, and means for completing the purchase procedure on the server after receiving the confirmation, thereby providing a product purchase process that takes the user's emotional state into account, reducing false positives and enabling a more personalized purchasing experience.

[1350] A "generative model" is an algorithm used to analyze video data to identify or classify specific items.

[1351] An "eye tracking device" is a device that detects the movement of a user's eyes and analyzes where the eyes are looking.

[1352] An "emotion analysis device" is a device that analyzes a user's facial expressions and voice patterns to identify the user's emotional state.

[1353] A "wink" is a user action in which one or both eyes are temporarily closed for a specific purpose.

[1354] "Product purchase intent confirmation" is the process of confirming whether a user intends to purchase a particular product based on a wink or other confirmation means.

[1355] A "server" is a computer system that receives, transmits, analyzes, and processes various types of data on a network.

[1356] "Video data" refers to real-time or existing video content such as television programs and videos.

[1357] "Recommendation information" is information that suggests optimal products and services to users based on the analysis results and the user's emotional data.

[1358] MODE FOR CARRYING OUT THE INVENTION

[1359] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[1360] System configuration

[1361] The system of the present invention consists of the following major components:

[1362] 1. Server

[1363] 2. Devices (TVs, eye tracking devices, emotion engines)

[1364] 3. Users

[1365] server

[1366] The server has the following roles:

[1367] Receiving video data: Receive real-time video data sent from the TV terminal.

[1368] Analysis using a generative model: A generative model is run to identify the location, type, brand, and model number of products contained in the video data. The generative model used here is "YOLO (You Only Look Once)."

[1369] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device. Microsoft Azure Cognitive Services is used to analyze the emotional data.

[1370] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[1371] Terminal

[1372] The device consists of a television, an eye-tracking device, and an emotion engine.

[1373] Capture of video data: Captures television images and sends them to the server.

[1374] Recommendation display: Recommendation information sent from the server is displayed on the TV screen. Game engines such as "Unity" are used for display.

[1375] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products. The device used here is the "Tobii Eye Tracker."

[1376] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to generate emotion data. The "Affectiva SDK" is used to analyze the user's emotions.

[1377] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[1378] User

[1379] The user performs the following actions:

[1380] Watching TV: Watch TV programs in the usual way.

[1381] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[1382] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[1383] Specific examples

[1384] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1385] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[1386] 2. Image analysis: The server uses the generative model to identify the brand, model number, etc. of the coffee maker. In this process, the prompt statement is used: "Analyze a video of a cooking show and identify the brand and model number of the product shown."

[1387] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[1388] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions, using the prompt "Please recognize the user's emotional state and prioritize recommendations."

[1389] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[1390] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is positive, a prompt will be set saying, "If the user's emotion is positive, highlight the confirmation message."

[1391] 7. Intent to buy: The user winks again to confirm their intent to buy.

[1392] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[1393] In this way, the system of the present invention takes into account the user's emotions and provides a more personalized and enhanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[1394] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1395] Program processing flow

[1396] Server Processing

[1397] Step 1: Receiving video data

[1398] The server receives video data transmitted in real time from the television terminal.

[1399] Input: Real-time video data from a TV terminal

[1400] Output: Received video data

[1401] Specifically, the server receives a video data stream via a network.

[1402] Step 2: Image analysis

[1403] The server analyzes the received video data and uses the generative model "YOLO (You Only Look Once)" to identify the product's location, type, brand, and model number.

[1404] Input: Video data

[1405] Output: Product location, type, brand, and model number information

[1406] After the video data is sent to the server, a generative model is run to extract and classify features of objects in the image.

[1407] Step 3: Integrating Emotional Data

[1408] The server receives the user's emotion data obtained from the emotion engine and integrates it with the image analysis results. Emotion analysis is performed using Microsoft Azure Cognitive Services.

[1409] Input: Image analysis results, emotion data

[1410] Output: Consolidated data

[1411] The server combines the analyzed product information with the user's emotional data and integrates the information necessary to generate recommendations.

[1412] Step 4: Recommendation generation

[1413] The server generates product recommendations for the user based on the image analysis results and emotional data and sends them to the terminal.

[1414] Input: Integrated data

[1415] Output: Recommendation information

[1416] The server selects the most suitable products based on the integrated data and generates specific recommendation information.

[1417] Step 5: Checkout

[1418] The server confirms the user's intention to purchase and completes the purchase procedure. The server notifies the user that the purchase procedure is complete.

[1419] Input: Confirm purchase intention

[1420] Output: Purchase completion notification

[1421] The server receives confirmation of the user's intention to purchase, processes the purchase, and notifies the user of the result.

[1422] Terminal handling

[1423] Step 1: Capture video data

[1424] The device captures the TV image and sends it to the server.

[1425] Input: TV program video

[1426] Output: Captured video data

[1427] Specifically, video data is acquired from a television screen and sent to a server as a stream.

[1428] Step 2: Displaying recommendation information

[1429] The terminal receives the recommendation information sent from the server and displays it as an overlay on the television screen.

[1430] Input: Recommendation information

[1431] Output: Displayed recommendation information

[1432] The received information is analyzed, and a user interface is generated and displayed on the screen.

[1433] Step 3: Eye tracking and wink detection

[1434] The eye tracking device, Tobii Eye Tracker, tracks the user's gaze and detects winks at specific products.

[1435] Input: User gaze data

[1436] Output: Wink detection result

[1437] The device tracks the user's gaze in real time, detects winks, and notifies the server.

[1438] Step 4: Emotion Recognition

[1439] The emotion engine analyzes the user's facial expressions and voice to generate emotion data. The user's emotions are analyzed using the "Affectiva SDK."

[1440] Input: User's facial expression and voice data

[1441] Output: Emotion data

[1442] Analyzed emotion data is generated and this information is sent to a server.

[1443] Step 5: Confirm your intention to purchase

[1444] A purchase confirmation message is displayed on the TV screen based on wink and emotion data.

[1445] Input: Wink detection results, emotion data

[1446] Output: Purchase confirmation message

[1447] Based on the analysis results, a confirmation message is overlaid on the screen.

[1448] User Actions

[1449] Step 1: Watch TV

[1450] The user watches the television program in the usual way.

[1451] Input: TV show

[1452] Output: User viewing behavior

[1453] Operate and watch the TV to view the program on the screen.

[1454] Step 2: Product selection

[1455] When a product that interests the user appears on the screen, the user turns their gaze and winks. The emotion engine analyzes the user's interest from their reaction.

[1456] Input: On-screen item

[1457] Output: Gaze data, wink, emotion data

[1458] The system detects gazes and winks towards the product and sends them to the server along with emotional data.

[1459] Step 3: Confirm your intention to purchase

[1460] In response to the purchase confirmation message, the user indicates their intention to purchase by winking again or pressing a specific key.

[1461] Input: Purchase confirmation message

[1462] Output: Re-wink, key operation

[1463] View the confirmation message and take the appropriate action to indicate your intent to purchase.

[1464] The above is a specific explanation of each processing step.

[1465] (Application example 2)

[1466] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1467] In online shopping, conventional methods have limited ability to quickly and intuitively confirm a user's purchasing intention, and they are unable to recommend products that take into account the user's emotional state. This can lead to a poor user experience and a decrease in purchasing motivation. To solve this problem, an advanced recommendation system that combines eye tracking and emotion recognition is needed.

[1468] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1469] In this invention, the server includes: means for identifying products from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a product purchase confirmation to the user based on the wink detection; means for completing the purchase procedure on the server after receiving the confirmation; means for analyzing the user's emotions using an emotion recognition engine and adjusting / emphasizing the product recommendation and confirmation message based on the emotion data; means for performing eye-tracking, emotion recognition, and wink detection using the smartphone's front camera and microphone; and means for overlaying recommendation information on the online shop screen and displaying a purchase confirmation message based on the emotion data, thereby enabling an intuitive and personalized purchasing experience that takes into account the user's gaze and emotional state.

[1470] A "generative model" is a machine learning algorithm for extracting and analyzing specific information from video data.

[1471] "Video data" refers to image and video data acquired from a photographing device such as a camera.

[1472] "Product" is a term that generally refers to products and services sold in online shops, etc.

[1473] An "eye tracking device" is a device that uses a camera or sensor to detect and track the position of a user's gaze.

[1474] "User" refers to the consumer or user of the System.

[1475] "Wink detection" is the process of using a sensor or camera to recognize when a user closes one of their eyes.

[1476] "Confirming intention" is an act in which a user confirms his or her own intentions or wishes through a specific operation or action.

[1477] A "server" is a computer system that provides services to multiple clients.

[1478] An "emotion recognition engine" is a technology for analyzing and estimating a user's emotional state from facial expressions, voice, etc.

[1479] "Recommendation" is the act of the system presenting products or information that are appropriate and likely to interest the user.

[1480] A "smartphone" is a multi-functional mobile device with advanced computing capabilities.

[1481] A "front camera" is a camera installed on the screen side of a smartphone or tablet.

[1482] A "microphone" is a device that picks up sound and converts it into an electrical signal.

[1483] "Overlay display" is a method of displaying additional information over an existing image or video.

[1484] An "online shop" is a website or application that sells products or services over the Internet.

[1485] The present invention provides a system that provides an advanced product recommendation and purchasing process that takes into account the user's gaze and emotional state. This system consists of the following main components: a server, a terminal (the user's smartphone), and a user. Next, we will explain the role of each component and its specific implementation.

[1486] server

[1487] The server has the following features:

[1488] 1. Receiving image data: The server receives real-time video data sent from the smartphone terminal.

[1489] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data. Generative models can be created using TensorFlow or Google Cloud Vision API.

[1490] 3. Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the smartphone device.

[1491] 4. Completing the purchase process: Based on the user's purchase intent and emotional state, the purchase process is completed using the Stripe API or similar.

[1492] Device (smartphone)

[1493] The smartphone has the following features:

[1494] 1. Capture video data: Capture video data from the smartphone's front camera and send it to the server.

[1495] 2. Displaying recommendation information: The recommendation information sent from the server is overlaid on the online shop screen.

[1496] 3. Eye Tracking and Wink Detection: Use the front camera to track the user's gaze and detect winks towards specific products. This is done using OpenCV.

[1497] 4. Emotion Recognition: Using the front camera and microphone, the system analyzes the user's facial expressions and voice patterns to generate emotion data. The emotion recognition engine is DeepFace.

[1498] 5. Confirmation of purchase intention: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[1499] User

[1500] The user performs the following actions:

[1501] 1. Online Shopping Browsing: Browse the online shop page in the usual way.

[1502] 2. Product Selection: When a product of interest appears on the screen, the robot turns its gaze and winks, and also expresses its emotions through facial expressions and voice.

[1503] 3. Confirming purchase intent: In response to a confirmation message displayed based on a wink or emotional state, the user can indicate their intention to purchase by winking again or using a specific finger gesture.

[1504] Specific examples

[1505] For example, consider a user browsing the latest smartphone models in an online shop. The server receives video data sent from the smartphone and uses a generative model to identify product information. The server generates recommendation information and overlays it on the smartphone screen. Meanwhile, the front camera and microphone are used to track the user's gaze and emotional state, and an emotion recognition engine recognizes that the user is pleased. If the user winks at the product, a purchase confirmation message is displayed, and the user confirms their intention to purchase with another wink, and payment is completed via the Stripe API.

[1506] Prompt Sentence Examples

[1507] Capture video from the front camera

[1508] Face detection and eye tracking initialization

[1509] Eye tracking and wink detection

[1510] Emotion Recognition and Recommendation Generation

[1511] Display purchase confirmation message

[1512] Wink again to confirm your intention to purchase

[1513] Completing the purchase process

[1514] This allows users to enjoy an intuitive and emotional purchasing experience.

[1515] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1516] Step 1:

[1517] Video data capture

[1518] The device uses the smartphone's front camera to capture video data of the online shop page the user is viewing. This video data is sent to the server in real time. The video data is the input, and the data sent to the server is the output.

[1519] Step 2:

[1520] Image analysis using generative models

[1521] The server runs a generative model based on the video data received from the device to identify the location, type, brand, and model number of the product in the video data. Generative models such as TensorFlow and Google Cloud Vision API are used. The input is the received video data, and the output is detailed product data such as the product's location and type.

[1522] Step 3:

[1523] Eye tracking and wink detection

[1524] The device uses the front camera to track the user's gaze and OpenCV to determine whether the user is looking at a specific product. It also runs a wink detection algorithm to detect winks toward a specific product. The input is camera image data, and the output is gaze information and wink detection results.

[1525] Step 4:

[1526] emotion recognition

[1527] The device uses the front camera and microphone to capture the user's facial expressions and voice, which are then analyzed by an emotion recognition engine such as DeepFace. This generates user emotion data. The input is facial expression data and voice data, and the output is the emotion recognition result.

[1528] Step 5:

[1529] Generating and displaying recommendation information

[1530] The server combines the gaze tracking results, wink detection results, and emotion data to generate product recommendation information. This information is then sent to the device, which then displays the recommendation information as an overlay on the online shop screen. The inputs are gaze information, wink detection results, and emotion data, and the output is recommendation information.

[1531] Step 6:

[1532] Confirmation of purchase intention

[1533] The device displays a product purchase confirmation message to the user based on the wink detection and emotion recognition results. The user confirms their intention to purchase by winking again or performing a specific finger gesture. The inputs are the wink detection results and emotion recognition results, and the output is the user's intention to purchase.

[1534] Step 7:

[1535] Completing the purchase process

[1536] The server receives confirmation of the user's intent to purchase and completes the purchase procedure using the Stripe API or similar. This confirms the product purchase and sends a purchase completion notification to the user. The input is the user's confirmation of intent to purchase, and the output is purchase confirmation and a purchase completion notification.

[1537] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1538] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1539] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1540] [Fourth embodiment]

[1541] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1542] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1543] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1544] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1545] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1546] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1547] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1548] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1549] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1550] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1551] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1552] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1553] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1554] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[1555] System configuration

[1556] The system of the present invention mainly comprises the following components:

[1557] 1. Server

[1558] 2. Devices (TV, eye tracking devices)

[1559] 3. Users

[1560] server

[1561] The server has the following roles:

[1562] Receiving video data: Receive real-time video data sent from the TV terminal.

[1563] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[1564] Recommendation generation: Using the analysis results, product recommendations for the user are generated and sent to the device.

[1565] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[1566] Terminal

[1567] The device consists of a television and an eye-tracking device.

[1568] Capture of video data: Captures television images and sends them to the server.

[1569] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[1570] Eye tracking: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[1571] Confirmation of purchase intent: When a wink is detected, a confirmation message for purchasing the product is displayed to the user.

[1572] User

[1573] The user performs the following actions:

[1574] Watching TV: Watch TV programs in the usual way.

[1575] Product selection: When a product of interest appears on the screen, look at it and wink.

[1576] Confirm your intention to purchase: After the wink, a confirmation message appears, and you can indicate your intention to purchase by winking again or pressing a specific key.

[1577] Explanation of program processing

[1578] Server Processing

[1579] 1. Receiving image data: The server receives video data transmitted in real time from the TV terminal.

[1580] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data.

[1581] 3. Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the terminal.

[1582] Terminal handling

[1583] 1. Displaying recommended information: The TV terminal receives the recommended information sent from the server and displays it as an overlay on the TV screen.

[1584] 2. Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[1585] 3. Display purchase confirmation message: When a wink is detected, a purchase confirmation message will be displayed on the TV screen.

[1586] User Actions

[1587] 1. TV viewing: Users watch TV and wait for products of interest to appear.

[1588] 2. Product selection: Wink at a specific product on the screen.

[1589] 3. Confirm your intention to purchase: In response to the confirmation message that appears, wink or press a key again to indicate your intention to purchase.

[1590] Specific examples

[1591] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1592] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[1593] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[1594] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[1595] 4. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[1596] 5. Purchase confirmation: A confirmation message will appear on the TV screen asking, "Are you sure you want to purchase this item?"

[1597] 6. Intent to buy: The user winks again to confirm their intent to buy.

[1598] 7. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[1599] As described above, the present invention provides a system that allows users to easily purchase products displayed on a television screen.

[1600] The processing flow will be explained below.

[1601] Step 1:

[1602] Terminal

[1603] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[1604] Step 2:

[1605] server

[1606] The server receives the video data sent from the TV terminal and saves it in high resolution.

[1607] Step 3:

[1608] server

[1609] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[1610] Step 4:

[1611] server

[1612] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[1613] Step 5:

[1614] Terminal

[1615] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[1616] Step 6:

[1617] Terminal

[1618] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[1619] Step 7:

[1620] Terminal

[1621] If a wink is detected, a purchase confirmation message will be displayed on the TV screen, asking "Are you sure you want to purchase this item?" and asking for a wink or keystroke for confirmation.

[1622] Step 8:

[1623] User

[1624] When a product of interest appears on the TV screen, the user turns their gaze towards the product and winks. After that, in response to the purchase confirmation message displayed on the TV screen, the user can wink again or use a keystroke to indicate their intention to purchase.

[1625] Step 9:

[1626] Terminal

[1627] The information confirming the user's intention to purchase is sent to the server, which is immediately notified and the purchase process begins.

[1628] Step 10:

[1629] server

[1630] Once the purchase confirmation is received, the system completes the purchase process, including verifying the user's address and payment method, and finally notifies the user that the purchase is complete.

[1631] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching television.

[1632] Example 1

[1633] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1634] While watching TV, users have no way to visually select and easily purchase products displayed, which makes the process of purchasing complicated and time-consuming. Furthermore, conventional methods make it difficult to track and quickly suggest products that interest users.

[1635] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1636] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for presenting the user with a confirmation of their intention to purchase the product based on the detection of the wink, means for completing the sales procedure after obtaining the confirmation of their intention, means for generating and displaying product recommendations for the user based on the results of image analysis by the generative model, means for displaying a confirmation message for a product selected by a wink and indicating their intention to purchase again by a wink or key operation, and means for sending a notification to the user after the purchase procedure is completed. This allows the user to visually easily select a product and quickly complete the procedure up to the purchase.

[1637] A "generative model" is a machine learning algorithm used to identify and extract specific objects and information from video data.

[1638] "Video data" refers to digital signals containing visual information transmitted by televisions and other visual devices.

[1639] An "eye-tracking device" is a device that detects the direction and movement of a user's gaze in real time.

[1640] "Wink detection" is a function that detects the user closing one eye and recognizes this action as a specific instruction.

[1641] "Confirming intention" is an act in which a user explicitly indicates their intention through a specific operation or action.

[1642] "Sales procedure" refers to the series of operations and processes required to purchase a product, including payment and delivery procedures.

[1643] "Recommendation information" is information that includes suggestions for specific products based on the user's preferences and purchase history.

[1644] A "visual display device" is a device, such as a television or monitor, that provides visual information to a user.

[1645] A "notification" is a message or alert sent by the system to convey information or results to a user.

[1646] The present invention is a system that identifies products from video data using a generative model, detects the user's wink toward a specific product by tracking the user's gaze using an eye-tracking device, and based on this, presents the user with a confirmation of their intention to purchase the product, after which the purchase procedure is completed on a server. The main aspects of the present invention are described below.

[1647] System configuration

[1648] The system of the present invention mainly comprises the following components:

[1649] 1. Server

[1650] 2. Devices (TV, eye tracking devices)

[1651] 3. Users

[1652] server

[1653] The server has the following roles:

[1654] Receiving video data: The server receives real-time video data sent from the TV terminal, for example, by streaming the video data using an HTTP request.

[1655] Analysis using a generative model: The server uses a generative model to identify the location, type, brand, and model number of products contained in the video data. Specifically, it runs a product recognition model using TensorFlow or PyTorch. An example of a prompt statement is, "Please run a model to identify the type, brand, and model number of the product shown in this video."

[1656] Recommendation generation: Based on the analysis results, the server generates product recommendations for the user and sends them to the device. During this process, a list of recommended products is generated that takes into account the user's purchase history and preferences.

[1657] Terminal

[1658] The device consists of a television and an eye-tracking device.

[1659] Capture of video data: Captures television images and sends them to the server.

[1660] Recommendation display: The TV device receives the recommendation information sent from the server and displays it as an overlay on the TV screen. For example, it displays the product image and detailed information along with the message "Do you like this product?"

[1661] Eye tracking and wink detection: An eye tracking device (e.g., Tobii Eye Tracker) tracks the user's gaze and detects winks toward specific products. Wink detection detects the action of closing one eye after the gaze has stopped on the product image for a certain period of time.

[1662] Displaying purchase intent confirmation: When a wink is detected, a confirmation message will be displayed on the TV screen asking, "Do you really want to purchase this product?" Specific purchase procedures and payment information will also be displayed.

[1663] User

[1664] The user performs the following actions:

[1665] TV viewing: Users watch TV and wait for products of interest to appear.

[1666] Product selection: Wink at a specific product on the screen.

[1667] Confirm your intention to purchase: In response to the displayed confirmation message, you can wink or press a key again to indicate your intention to purchase. For example, you can press the "OK" button on the remote control or wink again to indicate your intention to purchase.

[1668] Specific examples

[1669] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1670] 1. The server receives real-time video data sent from the TV terminal via an HTTP request. For example, the server retrieves video data from http: / / example.com / streaming-url.

[1671] 2. The server uses TensorFlow to identify the coffee maker brand, model number, etc. from the video data. For example, load the model with tf.keras.models.load_model('product_recognition_model.h5') and analyze the video data.

[1672] 3. The server generates recommendations for new coffee makers based on the user's past purchase history.

[1673] 4. The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, "The latest coffee makers" will pop up in the bottom right corner of the screen.

[1674] 5. The device uses the Tobii Eye Tracker to track the user's gaze and winks. After the user's gaze is focused on the image of the coffee maker, the device detects the closing of one eye.

[1675] 6. A message will appear on your TV screen asking, "Are you sure you want to purchase this item?" The message will be displayed large and in the center of the screen, along with payment options.

[1676] 7. The user confirms their intention to purchase by winking again or pressing the "OK" button on the remote control.

[1677] 8. The server calls the online store's API to complete the purchase. For example, it sends the user and product information to http: / / example-store.com / api / purchase.

[1678] 9. The server sends a purchase completion notification to the user. The message "Purchase Completed" is displayed on the TV screen, and a purchase completion notification is sent to the user's email address.

[1679] The above is an embodiment of the present invention. This system allows users to visually select products easily and quickly complete the procedures up to purchase.

[1680] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1681] Step 1:

[1682] Receiving video data

[1683] The server receives video data sent in real time from a TV terminal using an HTTP request. Specifically, the terminal captures the TV video and sends it to the server. The TV video data is input, and the received TV video data is saved on the server as output.

[1684] Step 2:

[1685] Image analysis using generative models

[1686] The server analyzes the received video data using a generative model (for example, a model using TensorFlow or PyTorch). This analysis identifies the location, type, brand, and model number of the product within the video data. The received video data is the input, and specific product information is obtained as the output.

[1687] An example of a prompt sentence is "Please run a model to identify the type, brand, and model number of the product shown in this video."

[1688] Step 3:

[1689] Recommendation generation

[1690] The server generates product recommendations for the user based on the product information identified by the generative model. The generated recommendation information is sent to the terminal. In this step, the user's preferences and purchase history are also taken into consideration. The input is product identification information and the user's purchase history, and the output is recommendation information.

[1691] Step 4:

[1692] Displaying recommendation information

[1693] The device receives the recommendation information from the server and displays it as an overlay on the TV screen. For example, a product image and detailed information are displayed along with the message "Do you like this product?" The input is the recommendation information, and the output is a screen display that the user can visually confirm.

[1694] Step 5:

[1695] Eye Tracking and Wink Detection

[1696] An eye tracking device (for example, Tobii Eye Tracker) connected to the device tracks the user's gaze and detects winks at a specific product. Specifically, it detects when the gaze is focused on the product image and then the user closes one eye. The input is the user's gaze data, and the output is wink detection data.

[1697] Step 6:

[1698] Display of purchase intent confirmation

[1699] When the device detects a wink, it displays a confirmation message on the TV screen asking, "Do you really want to purchase this product?". Specific purchase procedures and payment information are also displayed. The input is the wink detection data, and the output is the confirmation message and purchase procedure information displayed on the screen.

[1700] Step 7:

[1701] Purchase Intention

[1702] The user indicates their intention to purchase by winking again or pressing the "OK" button on the remote control. Specifically, the user's operation is the input, and the data on their intention to purchase is obtained as the output.

[1703] Step 8:

[1704] Completing the purchase process

[1705] The server confirms the user's intent to purchase and begins the purchase process. Specifically, it calls the online store's API using the product information, as well as the user's address and payment information, to confirm the order. The inputs are data indicating the user's intent to purchase and user information, and the output is data indicating the purchase process has been completed.

[1706] Step 9:

[1707] Sending a purchase completion notification

[1708] After the purchase procedure is completed, the server sends a notification of purchase completion to the user. Specifically, it displays a message on the TV screen saying "Purchase completed" and sends a notification by email. The input is data indicating the purchase procedure has been completed, and the output is a completion notification sent to the user.

[1709] (Application example 1)

[1710] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1711] Conventional systems that identify products from video data and recommend them to users are primarily limited to television footage, with few examples of application in physical stores. Furthermore, systems that efficiently confirm a user's purchase intentions by using eye tracking and winks are also lacking. Furthermore, recommendation displays that are limited to television footage are not suitable for in-store environments, and a flexible system that can be used in a variety of store situations is desired. Given this background, the challenge is to provide a system that can be used in physical stores and improves the efficiency of the purchasing process by using eye tracking and wink detection.

[1712] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1713] In this invention, the server includes: means for identifying a product from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a confirmation of the user's intention to purchase the product based on the detection of the wink; means for completing the purchase procedure on the server after obtaining the confirmation of intention; means for converting the video data into in-store video data and performing eye-tracking through smart glasses or a head-mounted display; means for displaying a purchase confirmation message within the user's field of view after detecting a wink; and means for reconfirming the user's intention to purchase using voice recognition. This enables efficient product selection through eye-tracking and automated purchase procedures even in physical stores. Furthermore, the use of voice recognition allows the user to confirm their intention to purchase more intuitively and naturally.

[1714] A "generative model" is an algorithm that uses machine learning technology to automatically identify products from video data.

[1715] "Video data" refers to digital data of visual information captured by a camera or the like, including television footage and footage of inside a store.

[1716] An "eye tracking device" is a device that detects the movement of a user's eyes in real time and identifies the direction in which the user is looking.

[1717] "Wink detection" is a technology that uses an eye-tracking device or the like to detect when a user makes a specific action, i.e., a wink.

[1718] The "server" is a central computer system that analyzes video data, runs generative models, manages purchase procedures, etc.

[1719] "Smart glasses" are high-performance eyeglass devices that can be worn by users and are capable of eye tracking and augmented reality display.

[1720] A "head-mounted display" is a device worn on the user's head that displays visual information and tracks the user's gaze.

[1721] A "purchase confirmation message" is a message that is displayed when a user selects a product, to reconfirm the user's intention to purchase the product.

[1722] "Speech recognition" is a technology that recognizes a user's voice and converts it into digital information.

[1723] "Product recommendation" is a function that presents products suitable for the user based on product information analyzed by the generative model.

[1724] A system for implementing the present invention mainly comprises the following components: a server, a terminal (smart glasses or head-mounted display, eye-tracking device), and a user.

[1725] The server uses the generative model to identify products from the video data, tracks the user's gaze using an eye-tracking device, detects winks toward the specific products, and, based on the detected winks, prompts the user to confirm their intention to purchase the product, after which the purchase process is completed.

[1726] Specifically, the server has the following roles:

[1727] Receiving video data: Receive real-time video data transmitted from smart glasses or head-mounted displays.

[1728] Generative model analysis: Run a generative model to identify the location, type, and brand of products in the video data.

[1729] Recommendation generation: Based on the analysis results, product recommendations for the user are generated and sent to the device.

[1730] Completion of purchase procedure: After the user confirms their intention to purchase, the product purchase procedure is carried out.

[1731] The smart glasses or head-mounted display as a terminal plays the following roles:

[1732] Video data capture: Video from inside the physical store is captured in real time and sent to the server.

[1733] Recommendation display: Recommendation information sent from the server is displayed within the field of view.

[1734] Eye Tracking: The eye tracking device tracks the user's gaze in real time and detects winks towards specific products.

[1735] Confirmation of purchase intent: When a wink is detected, a purchase confirmation message is displayed within the field of view and voice recognition is used to prompt the user to reconfirm their intention to purchase.

[1736] The user performs the following actions:

[1737] Store Walking: Walking around the store and exploring products that interest you.

[1738] Product selection: Look at the product you are interested in and wink.

[1739] Confirm your intention to purchase: When the purchase confirmation message appears, wink or speak again to indicate your intention to purchase.

[1740] As a concrete example, consider a situation where a user is walking through a brick-and-mortar store and looking at a coffee maker on a shelf. The smart glasses capture video data and send it to a server. A generative model analyzes the coffee maker and identifies detailed information. An eye-tracking device tracks the user's gaze and detects that the user is focusing on the coffee maker. When the user winks, a confirmation message appears on the smart glasses asking, "Do you want to purchase this item?" The user can wink again or respond "yes" verbally to proceed with the purchase.

[1741] An example of a prompt for the generative AI model is as follows:

[1742] "Identify products from the following store video data and determine their location, type, and brand."

[1743] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1744] Step 1:

[1745] The server receives real-time video data transmitted from smart glasses or head-mounted displays. This input data is video data captured by cameras inside the store, and after being sent to the server, analysis begins.

[1746] Step 2:

[1747] The server inputs the received video data into a generative model, which analyzes the data and identifies the product's location, type, and brand. The specific processing of this data involves the generative model's algorithm identifying the product and generating metadata such as location information and brand name.

[1748] Step 3:

[1749] The server then generates product recommendations based on the generated metadata. For example, if a particular product has a high rating, it will be recommended first. This information is then sent to the smart glasses or head-mounted display.

[1750] Step 4:

[1751] The device (smart glasses or head-mounted display) displays the received recommendation information within the user's field of view. The displayed recommendation information is provided as an overlay on the video data.

[1752] Step 5:

[1753] The device tracks the user's gaze in real time using an eye-tracking device, which detects the user's eye movements and identifies which products the user's gaze is focused on.

[1754] Step 6:

[1755] The device detects when a user winks at a specific product. The wink detection algorithm analyzes the gaze detection data and recognizes the wink. This information is sent to the server.

[1756] Step 7:

[1757] The server receives the wink detection information, generates a purchase confirmation message for the specific product, and sends it to the terminal. The message includes a confirmation message such as "Do you want to purchase this product?"

[1758] Step 8:

[1759] The device displays a purchase confirmation message within the user's field of vision, and the user can respond to this message by winking again or saying "yes" or similar to indicate their intention to purchase.

[1760] Step 9:

[1761] The device then sends another wink or voice recognition result to the server, which is used to confirm the user's final purchase decision.

[1762] Step 10:

[1763] The server receives the final purchase confirmation data and executes the purchase procedure, including retrieving product information from the product database and completing the purchase. Finally, it sends a purchase completion notification to the user.

[1764] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1765] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[1766] System configuration

[1767] The system of the present invention consists of the following major components:

[1768] 1. Server

[1769] 2. Devices (TVs, eye tracking devices, emotion engines)

[1770] 3. Users

[1771] server

[1772] The server has the following roles:

[1773] Receiving video data: Receive real-time video data sent from the TV terminal.

[1774] Generative model analysis: Run a generative model to identify the location, type, brand, and model number of products in the video data.

[1775] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device.

[1776] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[1777] Terminal

[1778] The device consists of a television, an eye-tracking device, and an emotion engine.

[1779] Capture of video data: Captures television images and sends them to the server.

[1780] Recommendation display: Recommendation information sent from the server is displayed on the TV screen.

[1781] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products.

[1782] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotions.

[1783] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[1784] User

[1785] The user performs the following actions:

[1786] Watching TV: Watch TV programs in the usual way.

[1787] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[1788] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[1789] Explanation of program processing

[1790] Server Processing

[1791] Receiving image data: The server receives the video data transmitted in real time from the television terminal.

[1792] Image analysis with generative models: Using generative models to identify the location, type, brand, and model number of products in video data.

[1793] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the device.

[1794] Terminal handling

[1795] Display of recommendation information: The TV terminal receives the recommendation information sent from the server and displays it as an overlay on the TV screen.

[1796] Eye tracking and wink detection: An eye tracking device follows the user's gaze and detects winks towards specific products.

[1797] Emotion recognition: The emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[1798] Displaying purchase intent confirmation: A purchase confirmation message is displayed on the TV screen based on winks and emotion data.

[1799] User Actions

[1800] TV viewing: Users watch TV and wait for products of interest to appear.

[1801] Product Selection: Winks at specific products on the screen and expresses emotional state through emotion analysis by the emotion engine.

[1802] Confirm your intention to purchase: In response to the displayed confirmation message, wink or press a key again to indicate your intention to purchase.

[1803] Specific examples

[1804] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1805] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[1806] 2. Image analysis: The server uses the generative model to identify the coffee maker brand, model number, etc.

[1807] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[1808] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions.

[1809] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[1810] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is favorable, the confirmation message will be highlighted.

[1811] 7. Intent to buy: The user winks again to confirm their intent to buy.

[1812] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[1813] As described above, the system of the present invention takes into account the user's emotions and provides a more personalized and advanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[1814] The processing flow will be explained below.

[1815] Step 1:

[1816] Terminal

[1817] The TV terminal captures the video data being broadcast in real time. It also acquires the video time and channel information at the same time. The acquired data is sent to the server.

[1818] Step 2:

[1819] server

[1820] The server receives the video data sent from the TV terminal and saves it in high resolution.

[1821] Step 3:

[1822] server

[1823] The received video data is input into a generative model to identify the location, type, brand, and model number of the product in the image. Deep learning technology is used for analysis, and the data is compared with a commercial database to accurately identify the product information.

[1824] Step 4:

[1825] server

[1826] Based on the generated product information, the system generates recommendations for the most suitable products for the viewer. This process also takes into account the viewer's purchase history and market trends. The generated recommendation information is then sent to the device.

[1827] Step 5:

[1828] Terminal

[1829] Receives recommendation information sent from the server and overlays it in the appropriate position on the TV screen to ensure the viewing experience is not disrupted.

[1830] Step 6:

[1831] Terminal

[1832] An eye-tracking device tracks the viewer's gaze in real time, and an appropriate algorithm is used to detect whether the viewer winks at a particular product.

[1833] Step 7:

[1834] Terminal

[1835] At the same time as detecting the wink, the emotion engine analyzes the user's facial expressions and voice patterns to recognize their emotional state. If the user is interested and expressing positive emotions, the information is sent to the server.

[1836] Step 8:

[1837] server

[1838] The server receives the emotion data from the emotion engine and further optimizes the recommendation information based on the user's emotions. Based on the wink detection and emotion data, a purchase confirmation message is generated and sent to the device.

[1839] Step 9:

[1840] Terminal

[1841] The device displays a purchase confirmation message on the TV screen. The message asks the user to confirm whether they really want to purchase this product. Depending on the emotion data, the message may be highlighted.

[1842] Step 10:

[1843] User

[1844] Based on the confirmation message, the user can then wink or press a key again to indicate their intention to purchase. If their emotional state is appropriate for purchasing, they can wink again to indicate their intention to purchase.

[1845] Step 11:

[1846] Terminal

[1847] Once the user's intention to purchase is confirmed, the terminal transmits this information to the server and begins the purchase procedure.

[1848] Step 12:

[1849] server

[1850] The server receives the confirmation of intent to purchase and processes the product purchase, which may include verifying the user's address and payment method, connecting with back-end systems, etc. Finally, it notifies the user that the purchase is complete.

[1851] The above is the specific program processing flow of the system of the present invention, which allows users to easily purchase products while watching TV, and by utilizing the emotion engine, provides a more personalized purchasing experience.

[1852] Example 2

[1853] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1854] Conventional product purchasing systems that rely on eye gaze tracking and wink detection have the problem of not being able to fully consider the user's emotional state and therefore fail to optimize the user experience. Furthermore, false detection due to users unintentionally winking is also a problem. Therefore, there is a need to provide a more advanced product purchasing process that reflects the user's emotional state, rather than simply detecting eye gaze and winks.

[1855] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1856] In this invention, the server includes means for identifying products from video data using a generative model, means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product, means for analyzing the user's emotions using an emotion analysis device, means for presenting a purchase confirmation to the user based on the wink and the results of the emotion analysis, and means for completing the purchase procedure on the server after receiving the confirmation, thereby providing a product purchase process that takes the user's emotional state into account, reducing false positives and enabling a more personalized purchasing experience.

[1857] A "generative model" is an algorithm used to analyze video data to identify or classify specific items.

[1858] An "eye tracking device" is a device that detects the movement of a user's eyes and analyzes where the eyes are looking.

[1859] An "emotion analysis device" is a device that analyzes a user's facial expressions and voice patterns to identify the user's emotional state.

[1860] A "wink" is a user action in which one or both eyes are temporarily closed for a specific purpose.

[1861] "Product purchase intent confirmation" is the process of confirming whether a user intends to purchase a particular product based on a wink or other confirmation means.

[1862] A "server" is a computer system that receives, transmits, analyzes, and processes various types of data on a network.

[1863] "Video data" refers to real-time or existing video content such as television programs and videos.

[1864] "Recommendation information" is information that suggests optimal products and services to users based on the analysis results and the user's emotional data.

[1865] MODE FOR CARRYING OUT THE INVENTION

[1866] The present invention is a system that provides an advanced product recommendation and purchasing process that takes into account not only eye tracking and wink detection but also the user's emotional state by combining an emotion engine that recognizes the user's emotions. The following describes embodiments of the present invention.

[1867] System configuration

[1868] The system of the present invention consists of the following major components:

[1869] 1. Server

[1870] 2. Devices (TVs, eye tracking devices, emotion engines)

[1871] 3. Users

[1872] server

[1873] The server has the following roles:

[1874] Receiving video data: Receive real-time video data sent from the TV terminal.

[1875] Analysis using a generative model: A generative model is run to identify the location, type, brand, and model number of products contained in the video data. The generative model used here is "YOLO (You Only Look Once)."

[1876] Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, product recommendations for the user are generated and sent to the device. Microsoft Azure Cognitive Services is used to analyze the emotional data.

[1877] Completing the purchase process: The product purchase process is completed based on the user's purchase intent and emotional state.

[1878] Terminal

[1879] The device consists of a television, an eye-tracking device, and an emotion engine.

[1880] Capture of video data: Captures television images and sends them to the server.

[1881] Recommendation display: Recommendation information sent from the server is displayed on the TV screen. Game engines such as "Unity" are used for display.

[1882] Eye tracking and wink detection: An eye tracking device tracks the user's gaze and detects winks towards specific products. The device used here is the "Tobii Eye Tracker."

[1883] Emotion Recognition: The emotion engine analyzes the user's facial expressions and voice patterns to generate emotion data. The "Affectiva SDK" is used to analyze the user's emotions.

[1884] Purchase Intention Confirmation: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[1885] User

[1886] The user performs the following actions:

[1887] Watching TV: Watch TV programs in the usual way.

[1888] Product selection: When a product of interest appears on the screen, the robot turns its gaze and winks. It also expresses its emotions through facial expressions and voice.

[1889] Confirming purchase intent: A confirmation message is displayed based on a wink or emotional state, and the user can indicate their intention to purchase by winking again or pressing a specific key.

[1890] Specific examples

[1891] For example, consider a case where a user is watching a cooking show and the latest coffee maker is introduced.

[1892] 1. Transmission of video data: The TV terminal transmits the program video to the server.

[1893] 2. Image analysis: The server uses the generative model to identify the brand, model number, etc. of the coffee maker. In this process, the prompt statement is used: "Analyze a video of a cooking show and identify the brand and model number of the product shown."

[1894] 3. Recommendation display: Recommendation information is displayed on the TV screen the user is watching as the "latest coffee maker."

[1895] 4. Gaze tracking and emotion recognition: The eye tracking device tracks the user's gaze, and the emotion engine analyzes the user's facial expressions and voice to recognize emotions, using the prompt "Please recognize the user's emotional state and prioritize recommendations."

[1896] 5. Wink detection: The user becomes interested in the coffee maker and winks to select it.

[1897] 6. Purchase confirmation: A confirmation message will be displayed on the TV screen asking, "Are you sure you want to purchase this product?" Furthermore, if the emotion engine determines that the user's emotion is positive, a prompt will be set saying, "If the user's emotion is positive, highlight the confirmation message."

[1898] 7. Intent to buy: The user winks again to confirm their intent to buy.

[1899] 8. Purchase completed: The server executes the purchase and sends a purchase completion notification to the user.

[1900] In this way, the system of the present invention takes into account the user's emotions and provides a more personalized and enhanced product purchasing experience by combining gaze tracking and wink detection with an emotion engine.

[1901] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1902] Program processing flow

[1903] Server Processing

[1904] Step 1: Receiving video data

[1905] The server receives video data transmitted in real time from the television terminal.

[1906] Input: Real-time video data from a TV terminal

[1907] Output: Received video data

[1908] Specifically, the server receives a video data stream via a network.

[1909] Step 2: Image analysis

[1910] The server analyzes the received video data and uses the generative model "YOLO (You Only Look Once)" to identify the product's location, type, brand, and model number.

[1911] Input: Video data

[1912] Output: Product location, type, brand, and model number information

[1913] After the video data is sent to the server, a generative model is run to extract and classify features of objects in the image.

[1914] Step 3: Integrating Emotional Data

[1915] The server receives the user's emotion data obtained from the emotion engine and integrates it with the image analysis results. Emotion analysis is performed using Microsoft Azure Cognitive Services.

[1916] Input: Image analysis results, emotion data

[1917] Output: Consolidated data

[1918] The server combines the analyzed product information with the user's emotional data and integrates the information necessary to generate recommendations.

[1919] Step 4: Recommendation generation

[1920] The server generates product recommendations for the user based on the image analysis results and emotional data and sends them to the terminal.

[1921] Input: Integrated data

[1922] Output: Recommendation information

[1923] The server selects the most suitable products based on the integrated data and generates specific recommendation information.

[1924] Step 5: Checkout

[1925] The server confirms the user's intention to purchase and completes the purchase procedure. The server notifies the user that the purchase procedure is complete.

[1926] Input: Confirm purchase intention

[1927] Output: Purchase completion notification

[1928] The server receives confirmation of the user's intention to purchase, processes the purchase, and notifies the user of the result.

[1929] Terminal handling

[1930] Step 1: Capture video data

[1931] The device captures the TV image and sends it to the server.

[1932] Input: TV program video

[1933] Output: Captured video data

[1934] Specifically, video data is acquired from a television screen and sent to a server as a stream.

[1935] Step 2: Displaying recommendation information

[1936] The terminal receives the recommendation information sent from the server and displays it as an overlay on the television screen.

[1937] Input: Recommendation information

[1938] Output: Displayed recommendation information

[1939] The received information is analyzed, and a user interface is generated and displayed on the screen.

[1940] Step 3: Eye tracking and wink detection

[1941] The eye tracking device, Tobii Eye Tracker, tracks the user's gaze and detects winks at specific products.

[1942] Input: User gaze data

[1943] Output: Wink detection result

[1944] The device tracks the user's gaze in real time, detects winks, and notifies the server.

[1945] Step 4: Emotion Recognition

[1946] The emotion engine analyzes the user's facial expressions and voice to generate emotion data. The user's emotions are analyzed using the "Affectiva SDK."

[1947] Input: User's facial expression and voice data

[1948] Output: Emotion data

[1949] Analyzed emotion data is generated and this information is sent to a server.

[1950] Step 5: Confirm your intention to purchase

[1951] A purchase confirmation message is displayed on the TV screen based on wink and emotion data.

[1952] Input: Wink detection results, emotion data

[1953] Output: Purchase confirmation message

[1954] Based on the analysis results, a confirmation message is overlaid on the screen.

[1955] User Actions

[1956] Step 1: Watch TV

[1957] The user watches the television program in the usual way.

[1958] Input: TV show

[1959] Output: User viewing behavior

[1960] Operate and watch the TV to view the program on the screen.

[1961] Step 2: Product selection

[1962] When a product that interests the user appears on the screen, the user turns their gaze and winks. The emotion engine analyzes the user's interest from their reaction.

[1963] Input: On-screen item

[1964] Output: Gaze data, wink, emotion data

[1965] The system detects gazes and winks towards the product and sends them to the server along with emotional data.

[1966] Step 3: Confirm your intention to purchase

[1967] In response to the purchase confirmation message, the user indicates their intention to purchase by winking again or pressing a specific key.

[1968] Input: Purchase confirmation message

[1969] Output: Re-wink, key operation

[1970] View the confirmation message and take the appropriate action to indicate your intent to purchase.

[1971] The above is a specific explanation of each processing step.

[1972] (Application example 2)

[1973] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1974] In online shopping, conventional methods have limited ability to quickly and intuitively confirm a user's purchasing intention, and they are unable to recommend products that take into account the user's emotional state. This can lead to a poor user experience and a decrease in purchasing motivation. To solve this problem, an advanced recommendation system that combines eye tracking and emotion recognition is needed.

[1975] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1976] In this invention, the server includes: means for identifying products from video data using a generative model; means for tracking a user's gaze using an eye-tracking device and detecting a wink from the user toward a specific product; means for presenting a product purchase confirmation to the user based on the wink detection; means for completing the purchase procedure on the server after receiving the confirmation; means for analyzing the user's emotions using an emotion recognition engine and adjusting / emphasizing the product recommendation and confirmation message based on the emotion data; means for performing eye-tracking, emotion recognition, and wink detection using the smartphone's front camera and microphone; and means for overlaying recommendation information on the online shop screen and displaying a purchase confirmation message based on the emotion data, thereby enabling an intuitive and personalized purchasing experience that takes into account the user's gaze and emotional state.

[1977] A "generative model" is a machine learning algorithm for extracting and analyzing specific information from video data.

[1978] "Video data" refers to image and video data acquired from a photographing device such as a camera.

[1979] "Product" is a term that generally refers to products and services sold in online shops, etc.

[1980] An "eye tracking device" is a device that uses a camera or sensor to detect and track the position of a user's gaze.

[1981] "User" refers to the consumer or user of the System.

[1982] "Wink detection" is the process of using a sensor or camera to recognize when a user closes one of their eyes.

[1983] "Confirming intention" is an act in which a user confirms his or her own intentions or wishes through a specific operation or action.

[1984] A "server" is a computer system that provides services to multiple clients.

[1985] An "emotion recognition engine" is a technology for analyzing and estimating a user's emotional state from facial expressions, voice, etc.

[1986] "Recommendation" is the act of the system presenting products or information that are appropriate and likely to interest the user.

[1987] A "smartphone" is a multi-functional mobile device with advanced computing capabilities.

[1988] A "front camera" is a camera installed on the screen side of a smartphone or tablet.

[1989] A "microphone" is a device that picks up sound and converts it into an electrical signal.

[1990] "Overlay display" is a method of displaying additional information over an existing image or video.

[1991] An "online shop" is a website or application that sells products or services over the Internet.

[1992] The present invention provides a system that provides an advanced product recommendation and purchasing process that takes into account the user's gaze and emotional state. This system consists of the following main components: a server, a terminal (the user's smartphone), and a user. Next, we will explain the role of each component and its specific implementation.

[1993] server

[1994] The server has the following features:

[1995] 1. Receiving image data: The server receives real-time video data sent from the smartphone terminal.

[1996] 2. Image analysis using generative models: Using generative models, we identify the location, type, brand, and model number of products in video data. Generative models can be created using TensorFlow or Google Cloud Vision API.

[1997] 3. Recommendation generation: Based on the analysis results and emotional data obtained from the emotion engine, the system generates optimal product recommendations for the user and sends them to the smartphone device.

[1998] 4. Completing the purchase process: Based on the user's purchase intent and emotional state, the purchase process is completed using the Stripe API or similar.

[1999] Device (smartphone)

[2000] The smartphone has the following features:

[2001] 1. Capture video data: Capture video data from the smartphone's front camera and send it to the server.

[2002] 2. Displaying recommendation information: The recommendation information sent from the server is overlaid on the online shop screen.

[2003] 3. Eye Tracking and Wink Detection: Use the front camera to track the user's gaze and detect winks towards specific products. This is done using OpenCV.

[2004] 4. Emotion Recognition: Using the front camera and microphone, the system analyzes the user's facial expressions and voice patterns to generate emotion data. The emotion recognition engine is DeepFace.

[2005] 5. Confirmation of purchase intention: Based on the wink and emotion recognition results, a confirmation message for purchasing the product is presented to the user.

[2006] User

[2007] The user performs the following actions:

[2008] 1. Online Shopping Browsing: Browse the online shop page in the usual way.

[2009] 2. Product Selection: When a product of interest appears on the screen, the robot turns its gaze and winks, and also expresses its emotions through facial expressions and voice.

[2010] 3. Confirming purchase intent: In response to a confirmation message displayed based on a wink or emotional state, the user can indicate their intention to purchase by winking again or using a specific finger gesture.

[2011] Specific examples

[2012] For example, consider a user browsing the latest smartphone models in an online shop. The server receives video data sent from the smartphone and uses a generative model to identify product information. The server generates recommendation information and overlays it on the smartphone screen. Meanwhile, the front camera and microphone are used to track the user's gaze and emotional state, and an emotion recognition engine recognizes that the user is pleased. If the user winks at the product, a purchase confirmation message is displayed, and the user confirms their intention to purchase with another wink, and payment is completed via the Stripe API.

[2013] Prompt Sentence Examples

[2014] Capture video from the front camera

[2015] Face detection and eye tracking initialization

[2016] Eye tracking and wink detection

[2017] Emotion Recognition and Recommendation Generation

[2018] Display purchase confirmation message

[2019] Wink again to confirm your intention to purchase

[2020] Completing the purchase process

[2021] This allows users to enjoy an intuitive and emotional purchasing experience.

[2022] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2023] Step 1:

[2024] Video data capture

[2025] The device uses the smartphone's front camera to capture video data of the online shop page the user is viewing. This video data is sent to the server in real time. The video data is the input, and the data sent to the server is the output.

[2026] Step 2:

[2027] Image analysis using generative models

[2028] The server runs a generative model based on the video data received from the device to identify the location, type, brand, and model number of the product in the video data. Generative models such as TensorFlow and Google Cloud Vision API are used. The input is the received video data, and the output is detailed product data such as the product's location and type.

[2029] Step 3:

[2030] Eye tracking and wink detection

[2031] The device uses the front camera to track the user's gaze and OpenCV to determine whether the user is looking at a specific product. It also runs a wink detection algorithm to detect winks toward a specific product. The input is camera image data, and the output is gaze information and wink detection results.

[2032] Step 4:

[2033] emotion recognition

[2034] The device uses the front camera and microphone to capture the user's facial expressions and voice, which are then analyzed by an emotion recognition engine such as DeepFace. This generates user emotion data. The input is facial expression data and voice data, and the output is the emotion recognition result.

[2035] Step 5:

[2036] Generating and displaying recommendation information

[2037] The server combines the gaze tracking results, wink detection results, and emotion data to generate product recommendation information. This information is then sent to the device, which then displays the recommendation information as an overlay on the online shop screen. The inputs are gaze information, wink detection results, and emotion data, and the output is recommendation information.

[2038] Step 6:

[2039] Confirmation of purchase intention

[2040] The device displays a product purchase confirmation message to the user based on the wink detection and emotion recognition results. The user confirms their intention to purchase by winking again or performing a specific finger gesture. The inputs are the wink detection results and emotion recognition results, and the output is the user's intention to purchase.

[2041] Step 7:

[2042] Completing the purchase process

[2043] The server receives confirmation of the user's intent to purchase and completes the purchase procedure using the Stripe API or similar. This confirms the product purchase and sends a purchase completion notification to the user. The input is the user's confirmation of intent to purchase, and the output is purchase confirmation and a purchase completion notification.

[2044] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2045] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2046] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2047] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2048] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2049] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2050] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2051] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2052] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2053] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2054] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2055] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2056] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2057] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2058] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2059] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2060] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2061] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2062] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2063] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2064] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2065] The following is further disclosed regarding the above embodiment.

[2066] (Claim 1)

[2067] A means for identifying products from video data using a generative model;

[2068] A means for tracking the gaze of a user using an eye tracking device and detecting a wink of the user toward a specific product;

[2069] means for presenting a confirmation of the user's intention to purchase a product based on the detection of the wink;

[2070] means for completing the purchase at the server after obtaining said confirmation of intent.

[2071] (Claim 2)

[2072] 2. The system of claim 1, wherein the video data is television video data.

[2073] (Claim 3)

[2074] 2. The system of claim 1, further comprising means for generating product recommendations using the generative model and displaying the recommendation information on a television image.

[2075] "Example 1"

[2076] (Claim 1)

[2077] A means for identifying products from video data using a generative model;

[2078] A means for tracking the gaze of a user using an eye tracking device and detecting a wink of the user toward a specific product;

[2079] means for presenting a confirmation of the user's intention to purchase a product based on the detection of the wink;

[2080] means for completing the sales procedure after obtaining said confirmation of intent;

[2081] A means for generating and displaying product recommendations for a user based on the image analysis results obtained by the generative model;

[2082] a means for displaying a confirmation message for the product selected by wink and allowing the user to indicate their intention to purchase by wink or key operation again;

[2083] The system includes a means for sending a notification to the user after the purchase process is completed.

[2084] (Claim 2)

[2085] 2. The system of claim 1, wherein the video data is television video data.

[2086] (Claim 3)

[2087] 10. The system of claim 1, further comprising means for generating product recommendations using the generative model and displaying the recommendation information on a visual display device.

[2088] "Application Example 1"

[2089] (Claim 1)

[2090] A means for identifying products from video data using a generative model;

[2091] A means for tracking the gaze of a user using an eye tracking device and detecting a wink of the user toward a specific product;

[2092] means for presenting a confirmation of the user's intention to purchase a product based on the detection of the wink;

[2093] means for completing the purchase procedure on the server after obtaining said confirmation of intent;

[2094] A means for tracking the gaze of the in-store video data through smart glasses or a head-mounted display;

[2095] means for displaying a purchase confirmation message within the field of view upon detecting a wink;

[2096] A system that includes a means for reconfirming purchase intent using voice recognition.

[2097] (Claim 2)

[2098] 2. The system according to claim 1, wherein the video data is television video data or in-store video data.

[2099] (Claim 3)

[2100] 10. The system of claim 1, further comprising means for generating product recommendations using the generative model and displaying the recommendations on a television image or visual display.

[2101] "Example 2: Combining Emotion Engines"

[2102] (Claim 1)

[2103] A means for identifying products from video data using a generative model;

[2104] A means for tracking the gaze of a user using an eye tracking device and detecting a wink of the user toward a specific product;

[2105] means for analyzing user emotions using an emotion analysis device;

[2106] means for presenting a confirmation of the user's intention to purchase a product based on the wink and the result of emotion analysis;

[2107] means for completing the purchase at the server after obtaining said confirmation of intent.

[2108] (Claim 2)

[2109] 2. The system of claim 1, wherein the video data is television video data.

[2110] (Claim 3)

[2111] 2. The system of claim 1, further comprising means for generating product recommendations using the generative model and displaying the recommendation information on a television image.

[2112] "Application example 2 when combining emotion engines"

[2113] (Claim 1)

[2114] A means for identifying products from video data using a generative model;

[2115] A means for tracking the gaze of a user using an eye tracking device and detecting a wink of the user toward a specific product;

[2116] means for presenting a confirmation of the user's intention to purchase a product based on the detection of the wink;

[2117] means for completing the purchase procedure on the server after obtaining said confirmation of intent;

[2118] A means for analyzing user emotions using an emotion recognition engine and adjusting and emphasizing product recommendation and intent confirmation messages based on the emotion data;

[2119] A means for performing eye tracking, emotion recognition, and wink detection using the front camera and microphone of a smartphone;

[2120] The system includes a means for overlaying recommendation information on an online shop screen and displaying a purchase confirmation message based on emotion data.

[2121] (Claim 2)

[2122] 2. The system according to claim 1, wherein the video data is web page video data of an online shop.

[2123] (Claim 3)

[2124] 2. The system according to claim 1, further comprising means for generating product recommendations using the generative model and overlaying the recommendation information on the screen of an online shop. [Explanation of symbols]

[2125] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> <...

Claims

1. A means for identifying products from video data using a generative model; A means for tracking the gaze of a user using an eye tracking device and detecting a wink of the user toward a specific product; means for presenting a confirmation of the user's intention to purchase a product based on the detection of the wink; means for completing the purchase at the server after obtaining said confirmation of intent.

2. 2. The system of claim 1, wherein the video data is television video data.

3. The system of claim 1 , further comprising means for generating product recommendations using the generative model and displaying the recommendation information on a television image.

Images