system

Abstract

Provide a system. [Solution] means for analyzing the received media content in real time and identifying objects displayed within the content; means for retrieving information about the identified object from the database; means for displaying the acquired information on a user's display device; A means for carrying out the purchase or reservation process for the product or service selected by the user based on the displayed information; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern media consumption, when viewers become interested in a product or service featured in a television program or other multimedia content, it is often time-consuming and difficult to obtain detailed information about the product or to actually purchase or reserve it. The process of viewers identifying a product or service they are interested in, conducting additional research about it, and then completing the purchase or reservation procedure is cumbersome, and there is room for improvement in terms of usability. For this reason, there is a need to provide a system that allows viewers to easily access products and services featured in content and smoothly make purchases or reservations. [Means for solving the problem]

[0005] The present invention provides a system that analyzes media content received in real time and identifies objects displayed within the content, and also provides a system that includes a means for retrieving information about the identified objects from a database and a means for displaying the retrieved information on a user's display device. Furthermore, the system includes a means for allowing a user to purchase or reserve a product or service selected by the user based on the displayed information, thereby enabling viewers to easily access detailed product or service information and smoothly complete the purchase or reservation process.

[0006] "Real-time" refers to media content being presented to users simultaneously with little delay.

[0007] "Media content" is a general term for content that includes visual or audio information, such as television programs, movies, and internet-distributed videos.

[0008] "Object" refers to a specific object or element such as clothing, a bag, or interior decor that appears within media content.

[0009] "Identification" refers to the process of identifying an object and extracting information related to it.

[0010] "Database" refers to an information management system that stores and makes available information related to identified objects.

[0011] "User" refers to people who use the system.

[0012] "Display device" refers to hardware used to display information on televisions, smart devices, etc.

[0013] "Detailed information" is additional information related to an object, including product name, manufacturer, price, and availability.

[0014] "Purchase" refers to the act of a user acquiring a specific product by paying money.

[0015] "Reservation" refers to the act of a user reserving a service (e.g., accommodation, meals, etc.) in advance.

[0016] "Means" refers to the methods and technologies used by this system to achieve its functions. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve. The system is primarily composed of a server and user terminals (televisions or smart devices).

[0039] Server Roles

[0040] Content Analysis

[0041] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[0042] Database Reference

[0043] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[0044] Information Packet Generation

[0045] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[0046] Information transmission

[0047] The server transmits the generated information packet to the user's terminal.

[0048] Device Role

[0049] Information reception and display

[0050] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[0051] Supporting User Input

[0052] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[0053] User Behavior

[0054] Reviewing and selecting information

[0055] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[0056] Purchase or reservation process

[0057] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[0058] Specific examples

[0059] Buying a dress while watching a drama

[0060] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[0061] Make restaurant reservations while watching a gourmet program

[0062] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[0063] In this way, the system helps users easily obtain detailed information about products and services and make purchases or reservations while watching TV, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] The server receives real-time media content data for viewing by the user, which includes video data such as television programs and streaming videos.

[0067] Step 2:

[0068] The server analyzes the received video data and identifies objects displayed within the content, using video recognition technology and machine learning algorithms.

[0069] Step 3:

[0070] The server extracts metadata about the identified objects, including characteristics such as the object's type, color, shape, brand logo, etc.

[0071] Step 4:

[0072] The server uses the extracted metadata to query a product information database, which contains detailed information about various products and services.

[0073] Step 5:

[0074] The server checks the product or service information retrieved from the database and retrieves the necessary details, such as product name, manufacturer, price, and availability information.

[0075] Step 6:

[0076] The server generates an information packet based on the information obtained, which includes product images, details, prices, and purchase links.

[0077] Step 7:

[0078] The server transmits the generated information packet to the user's terminal.

[0079] Step 8:

[0080] The terminal receives the information packets sent from the server.

[0081] Step 9:

[0082] The device analyzes the contents of the information packet and updates the user interface (UI), displaying detailed product or service information in the corner of the screen.

[0083] Step 10:

[0084] The user reviews the displayed information and selects the product or service they are interested in. For example, they use a remote control to select a particular product and view detailed information.

[0085] Step 11:

[0086] The device displays detailed information about the product or service selected by the user, including a detailed product description, reviews, and availability information.

[0087] Step 12:

[0088] The user confirms the displayed detailed information and performs the purchase or reservation procedure, for example, by entering credit card information and a delivery address.

[0089] Step 13:

[0090] The terminal transmits the purchase or reservation information entered by the user to the server.

[0091] Step 14:

[0092] The server processes the received purchase or reservation information, processes the payment through a payment gateway, and in the case of a reservation, connects to a reservation system to confirm and secure the reservation.

[0093] Step 15:

[0094] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[0095] Step 16:

[0096] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[0097] This series of steps allows users to smoothly obtain detailed information about products and services and make purchases or reservations while watching television.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] In the conventional television viewing experience, users must manually search for information on products and services related to the content they are watching, which disrupts the viewing experience. Furthermore, even if users are interested in a product or service, the process of actually purchasing or reserving it can be cumbersome, potentially discouraging many users from making a purchase. Therefore, there is a need for a system that provides real-time information on related products and services while watching television, enabling seamless purchasing and reserving procedures.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes means for analyzing video information received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a data store, and means for formatting the retrieved information to generate information packets, thereby enabling users to obtain information about products and services related to the content they are viewing in real time and to carry out seamless purchasing and reservation procedures.

[0103] "Real-time" means that data and information are processed immediately and results are provided without delay.

[0104] "Video information" refers to media content that includes video data and related information.

[0105] "Object" refers to any object displayed within the Content, including elements related to specific products or services.

[0106] A "data store" is a database or other persistent storage device for storing data and accessing it as needed.

[0107] An "information packet" is a collection of data that is sent to a user terminal after organizing and processing the acquired information.

[0108] A "user terminal" refers to an information processing device used by a user, including a television or smart device.

[0109] A "display means" is a device or interface for visually displaying information.

[0110] A "remote control device" is a device that allows a user to operate or control a system using a device that includes a remote control or a touch screen.

[0111] "Image recognition technology" is a technology that allows computer systems to identify specific objects or patterns from image data.

[0112] A "machine learning algorithm" is a computational method that allows a computer system to analyze data and learn from experience to improve its performance.

[0113] This invention is a system that analyzes video information received in real time, identifies objects displayed in the content, provides users with information related to those objects, and facilitates purchase and reservation procedures. The system is primarily composed of a server and a user terminal (such as a television or smart device). Specific embodiments are described below.

[0114] 1. Server Role

[0115] The server receives video information from TVs and smart devices in real time and analyzes it. It uses video recognition technology and machine learning algorithms (e.g., OpenCV, TENSORFLOW®) to identify objects displayed within the content. Information about the identified objects is retrieved by submitting SQL queries to a relational database management system (e.g., MySQL®).

[0116] The server generates an information packet based on the information obtained. This information packet contains product images, details, prices, purchase links, etc., formatted in JSON format. The information packet is then sent to the user's device using the HTTP protocol.

[0117] Examples:

[0118] The server analyzes a scene from a drama and identifies the dress worn by the actress.

[0119] The drama footage is analyzed frame by frame to identify the dress.

[0120] The identified dress information is retrieved from the data store and an information packet is generated that includes the brand name, price, and availability information.

[0121] The information packet is sent to the user terminal via an HTTP POST request.

[0122] Example prompt sentence:

[0123] "Please explain the steps to get details about the dress worn by the actress in the drama I'm watching and check out."

[0124] 2. Role of the terminal

[0125] The device receives the information packets sent by the server and displays them in a user interface (UI), for example, product information displayed as a popup in the corner of a TV screen. The device uses the React library to visually represent this information.

[0126] Additionally, the device allows users to interact with the displayed information using a remote control or touchscreen. Users can press buttons on the remote control to select specific products, view more information, or proceed with the purchase or reservation process.

[0127] Examples:

[0128] When a user uses the remote control to view details about a dress

[0129] The information packet is received and details about the dress pop up in the corner of the TV screen.

[0130] The user presses a button on the remote control to go to the details page and proceed with the purchase procedure.

[0131] 3. User Behavior

[0132] The user checks the displayed information and selects the product or service they are interested in. For the selected product or service, the user then proceeds to purchase or reserve it. For example, when purchasing a dress, the user enters shipping information and credit card information to complete the payment. When making a reservation, the user enters the date and number of people to confirm the reservation.

[0133] Examples:

[0134] When a user purchases a dress that appears in a drama

[0135] The user selects detailed information about the dress from the displayed pop-up using the remote control, and is taken to a purchase details page.

[0136] Enter the required shipping and payment information and click the "Purchase" button to confirm your purchase.

[0137] This system allows users to easily obtain information about products and services related to the content they are viewing, and allows them to seamlessly complete purchase and reservation procedures, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1:

[0140] Receiving content in real time

[0141] Input: Video stream from your TV or smart device.

[0142] What it does: The server receives the video stream in real time from the specified channel URL using a high-speed network connection and streaming server software (e.g., Wowza Streaming Engine).

[0143] Data processing: The received video stream is temporarily saved as a local file.

[0144] Output: Locally saved video stream file.

[0145] Step 2:

[0146] Content Analysis and Object Identification

[0147] Input: Locally stored video stream file.

[0148] How it works: The server analyzes the received video frame by frame and uses video recognition software (e.g., OpenCV) and machine learning algorithms (e.g., TensorFlow) to identify objects within the content.

[0149] Data processing: Each frame is analyzed sequentially and identified objects are labeled.

[0150] Output: A list of identified objects and their labels.

[0151] Step 3:

[0152] Retrieving information from a database

[0153] Input: A list of identified objects and their labels.

[0154] Specific operation: The server issues an SQL query to a relational database management system (e.g., MySQL) to retrieve information related to the object.

[0155] Data processing: Send a query to the database using the object label as a key to retrieve the corresponding data.

[0156] Output: Detailed data such as product information, manufacturer information, price information, and stock information retrieved from the database.

[0157] Step 4:

[0158] Generate information packets

[0159] Input: Detailed data retrieved from the database.

[0160] Specific operation: The server formats the acquired information and generates information packets to be provided to the user. This process uses web server software (e.g., Apache) and a scripting language (e.g., Python).

[0161] Data processing: Formatting the data and generating information packets in JSON format.

[0162] Output: The generated information packet in JSON format.

[0163] Step 5:

[0164] Sending information packets

[0165] Input: A packet of information in JSON format.

[0166] Specific operation: The server sends an information packet to the user's terminal using the HTTP protocol.

[0167] Data processing: Create an HTTP POST request and send the information packet to the user terminal.

[0168] Output: Information packets sent to the user terminal.

[0169] Step 6:

[0170] Receiving and displaying information on the user terminal

[0171] Input: A JSON formatted information packet sent by the server.

[0172] Specific operation: The device receives and analyzes the information packet. Then, it displays the analyzed information on the user interface (UI). The React library is used for display.

[0173] Data processing: Analyzes the received JSON data and passes the necessary information to the UI components for display.

[0174] Output: Product information displayed as a pop-up in the corner of the TV screen.

[0175] Step 7:

[0176] User-input operation

[0177] Input: Product information displayed on the user interface, and user input operations.

[0178] Specific behavior: The device accepts user input via a remote control or touchscreen, allowing the user to manipulate the displayed information. For example, the user presses a button on the remote control to select a particular product and view its details.

[0179] Data manipulation: Accepts user input and triggers relevant actions.

[0180] Output: A screen showing detailed product information.

[0181] Step 8:

[0182] Completing a purchase or reservation

[0183] Input: Shipping, payment, and reservation information entered by the user.

[0184] Specific operation: The terminal sends the information entered in the form to the server and calls an API to process the payment or reservation.

[0185] Data processing: The entered information is sent to the server as an HTTP POST request to process payments and reservations.

[0186] Output: A confirmation message that the purchase or reservation process is complete.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] Conventional systems that provide information about products and services while watching television lack the functionality to instantly check, purchase, or reserve products of interest to users. Furthermore, technology to identify specific objects within a video frame and provide related information in real time has not yet been fully developed. Therefore, improving usability and seamlessly integrating the viewing and purchasing experiences have been key challenges.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes means for analyzing the media content received in real time and using a deep learning model to identify a specific object in a video frame, means for retrieving information about the identified object from a database, and means for displaying the retrieved information on a user's display device, thereby providing product information or service information related to the specific object in the media content in real time, allowing the user to make an immediate purchase or reservation.

[0192] In addition, if the display device is a smart device, it further includes means for providing an interface for the user to make selections and inputs by voice or touch operation, thereby enabling the user to easily check product information, purchase, and reserve using voice commands or touch operation, improving convenience.

[0193] "Media content" refers to information in digital form that includes video and audio, and refers to viewable content such as television programs, movies, and videos.

[0194] A "video frame" is one of the still images that make up a digital video, and when displayed in succession, it is perceived as a moving image.

[0195] "Specific objects" refers to specific items or people that appear in the video frame, including clothing, furniture, accessories, etc.

[0196] A "deep learning model" is a type of artificial intelligence technology, a computational model that uses large amounts of data and advanced algorithms to recognize objects and patterns.

[0197] A "database" is a system that systematically organizes and stores specific information in a searchable manner, and stores product information, service information, etc.

[0198] A "display device" is a device for displaying information in a form that can be viewed by a user, and includes smartphones, tablets, televisions, and the like.

[0199] An "interface" is the means by which a user interacts with a system, including remote controls, touchscreens, voice recognition, etc.

[0200] "Usability" refers to the characteristics that make a system or service easy for users to use and understand, and includes efficiency, effectiveness, and satisfaction of use.

[0201] The details of the embodiments of the present invention are described below.

[0202] The server analyzes the received media content in real time and uses a deep learning model to identify specific objects contained within the video frames. For example, the server targets the clothes worn by actors in a movie scene or the furniture used in the scene. Publicly available deep learning models (e.g., YOLO, TensorFlow) can be used for this identification. Information about the identified objects is obtained by accessing a database on the server. This database stores detailed information about products and services (such as names, prices, availability, and image URLs).

[0203] The terminal displays the acquired information on the user's display device. If the display device is a smart device (such as a smartphone, tablet, or television), it provides an interface that allows the user to make selections and inputs using voice or touch operations. This allows the user to check product information related to a specific object and make direct purchases or reservations.

[0204] For example, if a user is interested in a bag that appears in a scene while watching a drama, the server can identify the bag, obtain related details, and send them to the display device. The user can then confirm the bag information by voice or touch operation and proceed with the purchase process.

[0205] The entire system is built using commercially available hardware and software, including high-performance servers, smart devices, deep learning models (YOLO, TensorFlow), and communication interfaces (API, HTTP / HTTPS).

[0206] An example prompt for a generative AI model is:

[0207] "Your goal is to create an AI model that can identify specific objects in a video frame and retrieve related product information. Specifically, it should focus on objects being held or items being worn, and provide that information from a database."

[0208] This allows users to enjoy a seamless viewing and purchasing experience.

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1:

[0211] The server receives and analyzes video frames of media content received in real time. Specifically, the server receives streaming data, extracts video frames from the data, and starts processing. The input data used in this process is the received media content, and the output is a list of analyzed video frames.

[0212] Step 2:

[0213] The server uses a deep learning model (e.g., YOLO, TensorFlow) to identify specific objects contained within video frames. The input is the list of video frames obtained in step 1, and the model detects objects within the frames and identifies their type and location. The output is a list of identified objects and their locations. Specifically, each frame is input into the deep learning model, and the object detection results are obtained.

[0214] Step 3:

[0215] The server retrieves detailed information about the identified objects from the database. The input is a list of object information identified in step 2, and searches and retrieves detailed information about the corresponding objects (e.g., product name, price, stock status, image URL, etc.) from the database. The output is a list of detailed information corresponding to each object. Specifically, the server executes a query to the database using the object name as a key, and compiles the results as a list.

[0216] Step 4:

[0217] The server generates an information packet based on the acquired detailed information and sends it to the terminal. The input is the list of detailed information obtained in step 3, and the output is an information packet. Specifically, the server converts the detailed information of each object into a packet format and generates data to be sent to the terminal.

[0218] Step 5:

[0219] The terminal displays the information packets received from the server on a display device. The input is the information packets sent from the server, and the output is product or service information displayed on the user's display device. Specific operations include analyzing the information packets and displaying product images and detailed information on the screen.

[0220] Step 6:

[0221] The user selects a specific product or service based on the displayed information and completes the purchase or reservation process. The input is the detailed product or service information displayed on the device, and the output is the user's selection and the input information (e.g., delivery address, payment information, reservation date, etc.). Specific operations involve the user selecting information using touch operations or voice commands and entering the necessary information.

[0222] Step 7:

[0223] The terminal sends the user's selection and input information to the server and proceeds with the purchase or reservation procedure. The input is the user's selection and input information, and the output is request data to the server. Specifically, the information entered by the user is packed into packets and sent to the server.

[0224] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0225] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[0226] Server Roles

[0227] Content Analysis

[0228] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[0229] Database Reference

[0230] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[0231] Information Packet Generation

[0232] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[0233] Emotional information reception

[0234] The server receives emotion information sent from the user's device, including the user's facial expression and tone of voice.

[0235] Information transmission

[0236] The server then sends the generated information packet to the user's device. It is also possible to adjust the product recommendations based on the user's emotional information.

[0237] Device Role

[0238] Information reception and display

[0239] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[0240] Emotional information gathering

[0241] The device is equipped with sensors to collect emotional information such as the user's facial expressions and tone of voice, allowing it to recognize the user's emotions such as joy, surprise, and interest in real time.

[0242] Emotional information transmission

[0243] The device transmits the collected emotion information to the server.

[0244] Supporting User Input

[0245] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[0246] User Behavior

[0247] Reviewing and selecting information

[0248] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[0249] Purchase or reservation process

[0250] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[0251] Specific examples

[0252] Buying a dress while watching a drama

[0253] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[0254] Make restaurant reservations while watching a gourmet program

[0255] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[0256] Use of emotional information

[0257] If a user smiles while watching a drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, the server can present accessories and other recommended products related to the dress the user smiled at.

[0258] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[0259] The processing flow will be explained below.

[0260] Step 1:

[0261] The server receives real-time media content data that the user is viewing, including video data to be analyzed.

[0262] Step 2:

[0263] The server analyzes the received video data using machine learning algorithms to identify objects (e.g., clothes, bags, interior decor) displayed within the content.

[0264] Step 3:

[0265] The server extracts metadata about the identified objects (e.g., type, color, shape, brand logo).

[0266] Step 4:

[0267] The server queries the product information database based on the extracted metadata to obtain relevant product information (e.g., product details, manufacturer, price, and stock information).

[0268] Step 5:

[0269] The server checks the product information retrieved from the database and generates an information packet containing the necessary details (e.g., product name, price, image, and purchase link).

[0270] Step 6:

[0271] The server transmits the generated information packet to the user's terminal.

[0272] Step 7:

[0273] The terminal receives the information packets sent from the server.

[0274] Step 8:

[0275] The terminal analyzes the contents of the information packet and updates the user interface, displaying detailed product or service information in the corner of the TV screen.

[0276] Step 9:

[0277] The device uses built-in sensors to collect the user's emotional information (e.g., facial expressions, tone of voice) in real time.

[0278] Step 10:

[0279] The terminal transmits the collected emotion information to the server.

[0280] Step 11:

[0281] The server analyzes the received emotion information and recognizes the user's emotions such as joy, surprise, and interest.

[0282] Step 12:

[0283] The server then tailors the displayed product and service recommendations based on the emotion engine, for example, by presenting options and offers related to the products the user has expressed interest in.

[0284] Step 13:

[0285] The terminal displays the adjusted information again on the user interface.

[0286] Step 14:

[0287] The user uses a remote control or touch screen to review the displayed information and select products or services of interest.

[0288] Step 15:

[0289] The device displays detailed information about the product or service the user selects, including a detailed description, reviews, and availability information.

[0290] Step 16:

[0291] The user decides to purchase or reserve and initiates the process using a remote control or touchscreen.

[0292] Step 17:

[0293] The terminal sends the purchase information (e.g., delivery address, credit card information) or reservation information (e.g., date, number of people) entered by the user to the server.

[0294] Step 18:

[0295] The server processes the payment through the payment gateway based on the received purchase information, and in the case of a reservation, connects to the corresponding reservation system to confirm and secure the reservation.

[0296] Step 19:

[0297] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[0298] Step 20:

[0299] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[0300] This series of steps allows users to easily obtain detailed information about products and services while watching TV, receive optimal suggestions based on their emotions, and smoothly make purchases or reservations.

[0301] Example 2

[0302] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0303] In recent years, the number of users watching content on televisions and smart devices has increased, leading to a growing demand for easy access to information on related products and services while watching, and for making purchases or reservations. However, conventional systems require users to search for information based on their interests, and are unable to provide optimal suggestions based on the user's emotions. This results in a lack of integration between the viewing and purchasing experiences, resulting in a lack of user convenience.

[0304] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for acquiring information about the identified objects from a database, means for displaying the acquired information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for collecting user emotion information, means for transmitting the collected emotion information to the server, and means for adjusting proposal content based on the emotion information. This not only makes it possible to easily acquire information about related products and services while viewing, but also makes it possible to make optimal proposals based on the user's emotions.

[0305] "Real-time" refers to the immediate processing of data and events without delay.

[0306] "Media content" is a general term for digital content that includes information such as audio, video, images, and text.

[0307] "Object" refers to a distinct item or element that appears within media content.

[0308] "Identification" refers to identifying what an object is.

[0309] A "database" refers to a collection of data that stores specific information and allows for rapid search and retrieval.

[0310] "Retrieval" refers to the act of retrieving the necessary information from a database.

[0311] "Display device" refers to a screen or display on which a user views information.

[0312] "Purchase" refers to the act of a user buying a product.

[0313] "Reservation" refers to the act of a user applying for a service in advance.

[0314] "Emotional information" refers to data about emotions obtained from a user's facial expressions, tone of voice, behavior, etc.

[0315] "Collection" refers to the act of obtaining emotional information using devices or sensors.

[0316] "Proposal content" refers to information about products and services offered to users.

[0317] "Adjustment" refers to the act of optimizing the suggestions based on the user's emotional state.

[0318] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[0319] The server analyzes the received media content in real time using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16). Specifically, it identifies objects such as the dress worn by an actress in a drama scene or furniture placed in the scene. This allows the user to obtain real-time information about the content they are watching.

[0320] Next, the server retrieves information about the identified object from a database. This data includes product details, manufacturer, price, and availability information. For example, for an identified dress, the server searches the database for the brand name and price information. At this stage, the server generates an information packet based on the retrieved information. The information packet includes a product image, details, price, and a purchase link.

[0321] The device receives the information packet sent from the server and displays it on a user interface (UI). For example, product information can be displayed as a pop-up in the corner of a TV screen. This allows users to easily view detailed information about products and services related to the content they are currently watching.

[0322] The device also uses emotion recognition software (e.g., FaceAPI, EmotionSDK) that collects emotional information such as the user's facial expressions and tone of voice to recognize the user's emotions, such as joy, surprise, and interest, in real time. This allows the device to transmit the collected emotional information to a server. The server can then adjust the product and service recommendations it provides to the user based on the collected emotional information.

[0323] For example, if a user smiles while watching a TV drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, it can present accessories and other recommended products related to the dress the user smiled at.

[0324] Below are some examples of prompt sentences:

[0325] Example prompt sentence:

[0326] A prompt that displays details about a dress worn by an actress in a scene from a drama, along with a link to purchase it.

[0327] "Please provide the brand name, price, and purchase link of the dress worn by the actress in the scene of the media content you are currently watching."

[0328] A prompt that provides information about a restaurant featured during a foodie show and a reservation link.

[0329] "View detailed information and reservation links for featured restaurants."

[0330] A prompt that recognizes joy in the user's facial expression and suggests specific products based on that emotion.

[0331] "Suggest accessories and other product recommendations related to the specific product that made the user smile."

[0332] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[0333] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0334] Step 1: Content Analysis

[0335] The server analyzes the incoming media content in real time, using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16) to perform frame-by-frame analysis. The input is the video data, and the machine learning model labels the objects in the frames. The output is a list of identified objects.

[0336] Specific behavior:

[0337] The server captures the video data and sends it to the analysis engine.

[0338] A machine learning model labels and identifies objects in each frame.

[0339] Compile information about the identified objects into a list.

[0340] Step 2: Database Reference

[0341] The server uses the list of identified objects to retrieve relevant information from a database. The input is the object's label information, and it generates and executes a database query. The output is product details, manufacturer, price, and availability information.

[0342] Specific behavior:

[0343] The server generates an SQL query based on the identification information.

[0344] A query is sent to the database to retrieve the relevant product information.

[0345] The acquired information is organized and packetized.

[0346] Step 3: Information Packet Generation

[0347] The server generates an information packet to provide to the user based on the retrieved product information. The input is the product information retrieved from the database, and the output is an information packet (JSON format) sent to the user.

[0348] Specific behavior:

[0349] Product information is converted into JSON format and packetized.

[0350] Generate an information packet containing product images, details, prices, and purchase links.

[0351] Step 4: Emotional information gathering

[0352] The device collects emotional information such as the user's facial expressions and tone of voice using emotion recognition software (e.g., FaceAPI, EmotionSDK). The input is sensor data from the camera and microphone, and the output is preprocessed emotional data.

[0353] Specific behavior:

[0354] The device will initialize the camera and microphone and start emotion recognition software.

[0355] Monitors the user's facial expressions and voice in real time.

[0356] The collected data is analyzed to identify emotional states.

[0357] Step 5: Emotional information transmission

[0358] The device sends the collected emotion information to the server. The input is the preprocessed emotion data, and the output is the emotion data sent to the server.

[0359] Specific behavior:

[0360] Convert the emotion data into a suitable format.

[0361] Send emotion data to the server's API endpoint using an HTTP POST request.

[0362] Step 6: Receiving and displaying information

[0363] The terminal receives the information packet sent from the server and displays it on the user interface (UI). The input is the information packet from the server, and the output is the product information displayed on the UI.

[0364] Specific behavior:

[0365] Receives an HTTP response from the server.

[0366] Parse the information packet and update the UI components.

[0367] A pop-up is displayed at an appropriate position on the TV screen to present product information.

[0368] Step 7: Supporting User Input

[0369] The terminal allows the user to operate the displayed information using a remote control or touch screen. The input is an operation signal from the user, and the output is a transition to a purchase screen or detailed information on the selected product.

[0370] Specific behavior:

[0371] It detects input from the remote control and processes the user's selection operations.

[0372] Detailed information about the selected product will be displayed, and you will be taken to the purchase or reservation screen.

[0373] Step 8: Purchase or reservation process

[0374] The user then completes the purchase or reservation procedure for the selected product or service. The input is the user's shipping information and payment information, and the output is a confirmation of the purchase or reservation.

[0375] Specific behavior:

[0376] A user input form is displayed on the device's UI, prompting the user to enter shipping address and payment information.

[0377] The input information is sent to the server and the purchase or reservation procedure is carried out.

[0378] Step 9: Use emotional information

[0379] The server adjusts the product and service recommendations provided to the user based on the emotion information. The input is the collected emotion information, and the output is the adjusted product recommendation information.

[0380] Specific behavior:

[0381] The server analyzes the emotional data and generates a list of recommended products based on pleasure and interest.

[0382] The recommended product list is included in an information packet and sent to the terminal, which then displays it on the UI.

[0383] (Application example 2)

[0384] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] With conventional systems, it was difficult for users to obtain real-time information about products and services displayed while watching television and make purchases or reservations on the spot.In addition, the system did not realize optimal product and service recommendations based on user emotions, making it difficult to improve the user experience.

[0386] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0387] In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a database, means for displaying the retrieved information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for analyzing and retrieving emotional information about the user using an emotion recognition engine, and means for adjusting the content provided to the user based on the retrieved emotional information. This allows the user to obtain information about products and services related to the content they are viewing in real time, making it easy to purchase or reserve, and also allowing them to receive optimal product recommendations based on their emotions.

[0388] "Real-time" refers to information being processed immediately at the moment it is generated.

[0389] "Media content" refers to a collection of visual and audio information such as television programs, videos, and images.

[0390] "Object" refers to a specific item that appears in the content, such as clothing, furniture, or other real-world objects.

[0391] "Database" refers to a centralized information management system for systematically storing and managing detailed information about products and services.

[0392] A "display device" refers to a device that allows a user to visually check information, such as a television or smartphone.

[0393] "Purchase or reservation process" means the series of steps required to purchase a product or service selected by a User or the process for confirming a reservation.

[0394] An "emotion recognition engine" refers to a software system that analyzes and recognizes emotions from a user's facial expressions and voice.

[0395] "Operation device" refers to an interactive device that allows a user to select or input information, such as a remote control or touch screen.

[0396] "Adjusting the content of information provided" refers to changing the content of information presented to the user based on the acquired emotion information.

[0397] This system provides real-time information about products and services related to objects displayed in media content that a user is viewing, and makes optimal suggestions using an emotion recognition engine. This system consists of a server, a user's device (such as a television or smartphone), and an emotion recognition engine.

[0398] Server Roles

[0399] The server analyzes the received media content in real time, identifies objects displayed within the content, retrieves information about the identified objects from a database, generates information packets for providing this information to the user's display device, and receives emotion information transmitted from the user's device and adjusts the content of recommended products and services based on the information.

[0400] Device Role

[0401] The terminal receives the information packets sent from the server and displays them on a user interface. The display device, such as a television or smartphone, presents detailed information about products and services related to the content the user is viewing. The terminal also uses sensors such as a camera and microphone to capture the user's facial expressions and voice and transmits this information to the server as emotional information.

[0402] User Behavior

[0403] Users can check the product or service information displayed on their device and proceed with the purchase or reservation if they are interested. For example, a user who is interested in a dress featured in a drama can view detailed information and proceed with the purchase. In addition, an emotion recognition engine analyzes the user's emotions, and based on the results, the user can receive suggestions for more attractive products and services.

[0404] Hardware and software used

[0405] Hardware: Smartphone camera and microphone

[0406] Software: OpenCV (image processing technology), TensorFlow (machine learning model), EmotionEngine (emotion recognition engine)

[0407] Processing flow

[0408] The server first analyzes the received media content and performs object recognition. Next, it retrieves detailed information about the object from a database, creates a packet containing that information, and sends it to the device. The device receives the packet and displays the information on its screen. At the same time, the device analyzes the user's facial expressions and voice using an emotion recognition engine and sends the emotional information to the server. The server then adjusts product and service recommendations based on the acquired emotional information and provides them to the user in real time.

[0409] Specific examples

[0410] When a user is watching a product introduction video on an online shopping site, their facial expressions are captured with their smartphone camera and analyzed by an emotion recognition engine. Information about the dress recognized in the video is displayed in real time, and if the user shows interest, related accessories and other items are also recommended. The following is an example of a prompt sentence to be input into the generative AI model.

[0411] Prompt Sentence Examples

[0412] Write a Python program that recognizes emotions from facial expressions and voices while a user is watching a video on a shopping site using a smartphone camera, and displays product information (e.g., dresses or furniture) identified in the video in real time. Considering the emotional signals, recommend products that are particularly appealing to the user if they are happy.

[0413] This system simultaneously improves the viewing and purchasing experience, enabling optimal service delivery based on user emotions.

[0414] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0415] Step 1:

[0416] Input: The media content the user is watching

[0417] Specific operation: The server receives media content in real time and performs content analysis.

[0418] Data processing and data calculation: Using object identification algorithms, we identify objects within the content using video recognition technology and machine learning models.

[0419] Output: List of identified objects

[0420] Step 2:

[0421] Input: List of identified objects

[0422] What happens next: The server looks up the database and retrieves detailed information about the identified object.

[0423] Data processing and data manipulation: Using database queries to extract detailed information about an object (e.g., product name, price, availability).

[0424] Output: An information packet containing detailed information

[0425] Step 3:

[0426] Input: Information packet containing detailed information

[0427] Specific operation: The server sends an information packet to the user's terminal.

[0428] Data processing and data calculation: Encoding information packets into the appropriate format and sending them over the network.

[0429] Output: Information packets received at the user's terminal

[0430] Step 4:

[0431] Input: Information packet received at the user's terminal

[0432] Specific operation: The terminal decodes the information packet and displays it on the user interface.

[0433] Data processing and data calculation: Displaying item information (images, text, links, etc.) on the screen to visualize the decoded information.

[0434] Output: Product or service information that users can visually confirm

[0435] Step 5:

[0436] Input: User's face image and voice

[0437] How it works: The device's built-in camera and microphone capture the user's facial expressions and voice in real time.

[0438] Data processing and data calculation: An emotion recognition engine is used to analyze the user's emotional information from the captured facial expressions and voice.

[0439] Output: Emotional information (e.g., joy, surprise, interest)

[0440] Step 6:

[0441] Input: Emotion information

[0442] Specific operation: The device transmits the acquired emotional information to the server.

[0443] Data processing and data calculation: Encode emotional information into an appropriate format and transmit it over the network.

[0444] Output: Emotion information received by the server

[0445] Step 7:

[0446] Input: Emotion information received by the server

[0447] Specific operation: The server adjusts the products and services it recommends to the user based on the emotional information.

[0448] Data processing and calculation: Combining emotional information and object information, we execute algorithms to select optimal products and services.

[0449] Output: Adjusted recommendations for the user

[0450] Step 8:

[0451] Input: Adjusted proposal information

[0452] Specific operation: The server sends the adjusted proposal information back to the device and provides it to the user.

[0453] Data processing and data calculation: The proposed information is encoded as a packet and sent over the network.

[0454] Output: Adjusted proposal information received on the user's device

[0455] Step 9:

[0456] Input: Adjusted proposal information

[0457] Specific operation: The device displays information about suggested products and services tailored to the user interface.

[0458] Data processing and calculation: The proposed information after adjustment is visualized and displayed for the user to check.

[0459] Output: Information about the adjusted products and services that can be visually confirmed by the user

[0460] Step 10:

[0461] Input: Information about the adjusted proposed products and services

[0462] Specific action: The user selects the suggested product or service and completes the purchase or reservation process.

[0463] Data processing and calculation: Enter the necessary information for the purchase or reservation (shipping address and payment information) and finalize the transaction.

[0464] Output: Confirmation of completed purchase or reservation

[0465] By executing the above steps in order, the present invention enables the user to obtain information on products and services related to the content the user is viewing in real time and receive optimal suggestions based on emotions.

[0466] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0467] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0468] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0469] [Second embodiment]

[0470] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0471] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0472] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0473] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0474] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0475] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0476] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0477] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0478] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0479] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0480] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0481] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0482] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve. The system is primarily composed of a server and user terminals (televisions or smart devices).

[0483] Server Roles

[0484] Content Analysis

[0485] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[0486] Database Reference

[0487] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[0488] Information Packet Generation

[0489] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[0490] Information transmission

[0491] The server transmits the generated information packet to the user's terminal.

[0492] Device Role

[0493] Information reception and display

[0494] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[0495] Supporting User Input

[0496] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[0497] User Behavior

[0498] Reviewing and selecting information

[0499] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[0500] Purchase or reservation process

[0501] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[0502] Specific examples

[0503] Buying a dress while watching a drama

[0504] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[0505] Make restaurant reservations while watching a gourmet program

[0506] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[0507] In this way, the system helps users easily obtain detailed information about products and services and make purchases or reservations while watching TV, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[0508] The processing flow will be explained below.

[0509] Step 1:

[0510] The server receives real-time media content data for viewing by the user, which includes video data such as television programs and streaming videos.

[0511] Step 2:

[0512] The server analyzes the received video data and identifies objects displayed within the content, using video recognition technology and machine learning algorithms.

[0513] Step 3:

[0514] The server extracts metadata about the identified objects, including characteristics such as the object's type, color, shape, brand logo, etc.

[0515] Step 4:

[0516] The server uses the extracted metadata to query a product information database, which contains detailed information about various products and services.

[0517] Step 5:

[0518] The server checks the product or service information retrieved from the database and retrieves the necessary details, such as product name, manufacturer, price, and availability information.

[0519] Step 6:

[0520] The server generates an information packet based on the information obtained, which includes product images, details, prices, and purchase links.

[0521] Step 7:

[0522] The server transmits the generated information packet to the user's terminal.

[0523] Step 8:

[0524] The terminal receives the information packets sent from the server.

[0525] Step 9:

[0526] The device analyzes the contents of the information packet and updates the user interface (UI), displaying detailed product or service information in the corner of the screen.

[0527] Step 10:

[0528] The user reviews the displayed information and selects the product or service they are interested in. For example, they use a remote control to select a particular product and view detailed information.

[0529] Step 11:

[0530] The device displays detailed information about the product or service selected by the user, including a detailed product description, reviews, and availability information.

[0531] Step 12:

[0532] The user confirms the displayed detailed information and performs the purchase or reservation procedure, for example, by entering credit card information and a delivery address.

[0533] Step 13:

[0534] The terminal transmits the purchase or reservation information entered by the user to the server.

[0535] Step 14:

[0536] The server processes the received purchase or reservation information, processes the payment through a payment gateway, and in the case of a reservation, connects to a reservation system to confirm and secure the reservation.

[0537] Step 15:

[0538] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[0539] Step 16:

[0540] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[0541] This series of steps allows users to smoothly obtain detailed information about products and services and make purchases or reservations while watching television.

[0542] Example 1

[0543] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0544] In the conventional television viewing experience, users must manually search for information on products and services related to the content they are watching, which disrupts the viewing experience. Furthermore, even if users are interested in a product or service, the process of actually purchasing or reserving it can be cumbersome, potentially discouraging many users from making a purchase. Therefore, there is a need for a system that provides real-time information on related products and services while watching television, enabling seamless purchasing and reserving procedures.

[0545] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0546] In this invention, the server includes means for analyzing video information received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a data store, and means for formatting the retrieved information to generate information packets, thereby enabling users to obtain information about products and services related to the content they are viewing in real time and to carry out seamless purchasing and reservation procedures.

[0547] "Real-time" means that data and information are processed immediately and results are provided without delay.

[0548] "Video information" refers to media content that includes video data and related information.

[0549] "Object" refers to any object displayed within the Content, including elements related to specific products or services.

[0550] A "data store" is a database or other persistent storage device for storing data and accessing it as needed.

[0551] An "information packet" is a collection of data that is sent to a user terminal after organizing and processing the acquired information.

[0552] A "user terminal" refers to an information processing device used by a user, including a television or smart device.

[0553] A "display means" is a device or interface for visually displaying information.

[0554] A "remote control device" is a device that allows a user to operate or control a system using a device that includes a remote control or a touch screen.

[0555] "Image recognition technology" is a technology that allows computer systems to identify specific objects or patterns from image data.

[0556] A "machine learning algorithm" is a computational method that allows a computer system to analyze data and learn from experience to improve its performance.

[0557] This invention is a system that analyzes video information received in real time, identifies objects displayed in the content, provides users with information related to those objects, and facilitates purchase and reservation procedures. The system is primarily composed of a server and a user terminal (such as a television or smart device). Specific embodiments are described below.

[0558] 1. Server Role

[0559] The server receives video information from TVs and smart devices in real time and analyzes it. It uses video recognition technology and machine learning algorithms (e.g., OpenCV and TensorFlow) to identify objects displayed within the content. Information about the identified objects is retrieved by submitting SQL queries to a relational database management system (e.g., MySQL).

[0560] The server generates an information packet based on the information obtained. This information packet contains product images, details, prices, purchase links, etc., formatted in JSON format. The information packet is then sent to the user's device using the HTTP protocol.

[0561] Examples:

[0562] The server analyzes a scene from a drama and identifies the dress worn by the actress.

[0563] The drama footage is analyzed frame by frame to identify the dress.

[0564] The identified dress information is retrieved from the data store and an information packet is generated that includes the brand name, price, and availability information.

[0565] The information packet is sent to the user terminal via an HTTP POST request.

[0566] Example prompt sentence:

[0567] "Please explain the steps to get details about the dress worn by the actress in the drama I'm watching and check out."

[0568] 2. Role of the terminal

[0569] The device receives the information packets sent by the server and displays them in a user interface (UI), for example, product information displayed as a popup in the corner of a TV screen. The device uses the React library to visually represent this information.

[0570] Additionally, the device allows users to interact with the displayed information using a remote control or touchscreen. Users can press buttons on the remote control to select specific products, view more information, or proceed with the purchase or reservation process.

[0571] Examples:

[0572] When a user uses the remote control to view details about a dress

[0573] The information packet is received and details about the dress pop up in the corner of the TV screen.

[0574] The user presses a button on the remote control to go to the details page and proceed with the purchase procedure.

[0575] 3. User Behavior

[0576] The user checks the displayed information and selects the product or service they are interested in. For the selected product or service, the user then proceeds to purchase or reserve it. For example, when purchasing a dress, the user enters shipping information and credit card information to complete the payment. When making a reservation, the user enters the date and number of people to confirm the reservation.

[0577] Examples:

[0578] When a user purchases a dress that appears in a drama

[0579] The user selects detailed information about the dress from the displayed pop-up using the remote control, and is taken to a purchase details page.

[0580] Enter the required shipping and payment information and click the "Purchase" button to confirm your purchase.

[0581] This system allows users to easily obtain information about products and services related to the content they are viewing, and allows them to seamlessly complete purchase and reservation procedures, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[0582] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0583] Step 1:

[0584] Receiving content in real time

[0585] Input: Video stream from your TV or smart device.

[0586] What it does: The server receives the video stream in real time from the specified channel URL using a high-speed network connection and streaming server software (e.g., Wowza Streaming Engine).

[0587] Data processing: The received video stream is temporarily saved as a local file.

[0588] Output: Locally saved video stream file.

[0589] Step 2:

[0590] Content Analysis and Object Identification

[0591] Input: Locally stored video stream file.

[0592] How it works: The server analyzes the received video frame by frame and uses video recognition software (e.g., OpenCV) and machine learning algorithms (e.g., TensorFlow) to identify objects within the content.

[0593] Data processing: Each frame is analyzed sequentially and identified objects are labeled.

[0594] Output: A list of identified objects and their labels.

[0595] Step 3:

[0596] Retrieving information from a database

[0597] Input: A list of identified objects and their labels.

[0598] Specific operation: The server issues an SQL query to a relational database management system (e.g., MySQL) to retrieve information related to the object.

[0599] Data processing: Send a query to the database using the object label as a key to retrieve the corresponding data.

[0600] Output: Detailed data such as product information, manufacturer information, price information, and stock information retrieved from the database.

[0601] Step 4:

[0602] Generate information packets

[0603] Input: Detailed data retrieved from the database.

[0604] What happens: The server formats the information it receives and creates packets of information to serve to the user. This process uses web server software (e.g., Apache) and a scripting language (e.g., Python).

[0605] Data processing: Formatting the data and generating information packets in JSON format.

[0606] Output: The generated information packet in JSON format.

[0607] Step 5:

[0608] Sending information packets

[0609] Input: A packet of information in JSON format.

[0610] Specific operation: The server sends an information packet to the user's terminal using the HTTP protocol.

[0611] Data processing: Create an HTTP POST request and send the information packet to the user terminal.

[0612] Output: Information packets sent to the user terminal.

[0613] Step 6:

[0614] Receiving and displaying information on the user terminal

[0615] Input: A JSON formatted information packet sent by the server.

[0616] Specific operation: The device receives and analyzes the information packet. Then, it displays the analyzed information on the user interface (UI). The React library is used for display.

[0617] Data processing: Analyzes the received JSON data and passes the necessary information to the UI components for display.

[0618] Output: Product information displayed as a pop-up in the corner of the TV screen.

[0619] Step 7:

[0620] User-input operation

[0621] Input: Product information displayed on the user interface, and user input operations.

[0622] Specific behavior: The device accepts user input via a remote control or touchscreen, allowing the user to manipulate the displayed information. For example, the user presses a button on the remote control to select a particular product and view its details.

[0623] Data manipulation: Accepts user input and triggers relevant actions.

[0624] Output: A screen showing detailed product information.

[0625] Step 8:

[0626] Completing a purchase or reservation

[0627] Input: Shipping, payment, and reservation information entered by the user.

[0628] Specific operation: The terminal sends the information entered in the form to the server and calls an API to process the payment or reservation.

[0629] Data processing: The entered information is sent to the server as an HTTP POST request to process payments and reservations.

[0630] Output: A confirmation message that the purchase or reservation process is complete.

[0631] (Application example 1)

[0632] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] Conventional systems that provide information about products and services while watching television lack the functionality to instantly check, purchase, or reserve products of interest to users. Furthermore, technology to identify specific objects within a video frame and provide related information in real time has not yet been fully developed. Therefore, improving usability and seamlessly integrating the viewing and purchasing experiences have been key challenges.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0635] In this invention, the server includes means for analyzing the media content received in real time and using a deep learning model to identify a specific object in a video frame, means for retrieving information about the identified object from a database, and means for displaying the retrieved information on a user's display device, thereby providing product information or service information related to the specific object in the media content in real time, allowing the user to make an immediate purchase or reservation.

[0636] In addition, if the display device is a smart device, it further includes means for providing an interface for the user to make selections and inputs by voice or touch operation, thereby enabling the user to easily check product information, purchase, and reserve using voice commands or touch operation, improving convenience.

[0637] "Media content" refers to information in digital form that includes video and audio, and refers to viewable content such as television programs, movies, and videos.

[0638] A "video frame" is one of the still images that make up a digital video, and when displayed in succession, it is perceived as a moving image.

[0639] "Specific objects" refers to specific items or people that appear in the video frame, including clothing, furniture, accessories, etc.

[0640] A "deep learning model" is a type of artificial intelligence technology, a computational model that uses large amounts of data and advanced algorithms to recognize objects and patterns.

[0641] A "database" is a system that systematically organizes and stores specific information in a searchable manner, and stores product information, service information, etc.

[0642] A "display device" is a device for displaying information in a form that can be viewed by a user, and includes smartphones, tablets, televisions, and the like.

[0643] An "interface" is the means by which a user interacts with a system, including remote controls, touchscreens, voice recognition, etc.

[0644] "Usability" refers to the characteristics that make a system or service easy for users to use and understand, and includes efficiency, effectiveness, and satisfaction of use.

[0645] The details of the embodiments of the present invention are described below.

[0646] The server analyzes the received media content in real time and uses a deep learning model to identify specific objects contained within the video frames. For example, the server targets the clothes worn by actors in a movie scene or the furniture used in the scene. Publicly available deep learning models (e.g., YOLO, TensorFlow) can be used for this identification. Information about the identified objects is obtained by accessing a database on the server. This database stores detailed information about products and services (such as names, prices, availability, and image URLs).

[0647] The terminal displays the acquired information on the user's display device. If the display device is a smart device (such as a smartphone, tablet, or television), it provides an interface that allows the user to make selections and inputs using voice or touch operations. This allows the user to check product information related to a specific object and make direct purchases or reservations.

[0648] For example, if a user is interested in a bag that appears in a scene while watching a drama, the server can identify the bag, obtain related details, and send them to the display device. The user can then confirm the bag information by voice or touch operation and proceed with the purchase process.

[0649] The entire system is built using commercially available hardware and software, including high-performance servers, smart devices, deep learning models (YOLO, TensorFlow), and communication interfaces (API, HTTP / HTTPS).

[0650] An example prompt for a generative AI model is:

[0651] "Your goal is to create an AI model that can identify specific objects in a video frame and retrieve related product information. Specifically, it should focus on objects being held or items being worn, and provide that information from a database."

[0652] This allows users to enjoy a seamless viewing and purchasing experience.

[0653] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0654] Step 1:

[0655] The server receives and analyzes video frames of media content received in real time. Specifically, the server receives streaming data, extracts video frames from the data, and starts processing. The input data used in this process is the received media content, and the output is a list of analyzed video frames.

[0656] Step 2:

[0657] The server uses a deep learning model (e.g., YOLO, TensorFlow) to identify specific objects contained within video frames. The input is the list of video frames obtained in step 1, and the model detects objects within the frames and identifies their type and location. The output is a list of identified objects and their locations. Specifically, each frame is input into the deep learning model, and the object detection results are obtained.

[0658] Step 3:

[0659] The server retrieves detailed information about the identified objects from the database. The input is a list of object information identified in step 2, and searches and retrieves detailed information about the corresponding objects (e.g., product name, price, stock status, image URL, etc.) from the database. The output is a list of detailed information corresponding to each object. Specifically, the server executes a query to the database using the object name as a key, and compiles the results as a list.

[0660] Step 4:

[0661] The server generates an information packet based on the acquired detailed information and sends it to the terminal. The input is the list of detailed information obtained in step 3, and the output is an information packet. Specifically, the server converts the detailed information of each object into a packet format and generates data to be sent to the terminal.

[0662] Step 5:

[0663] The terminal displays the information packets received from the server on a display device. The input is the information packets sent from the server, and the output is product or service information displayed on the user's display device. Specific operations include analyzing the information packets and displaying product images and detailed information on the screen.

[0664] Step 6:

[0665] The user selects a specific product or service based on the displayed information and completes the purchase or reservation process. The input is the detailed product or service information displayed on the device, and the output is the user's selection and the input information (e.g., delivery address, payment information, reservation date, etc.). Specific operations involve the user selecting information using touch operations or voice commands and entering the necessary information.

[0666] Step 7:

[0667] The terminal sends the user's selection and input information to the server and proceeds with the purchase or reservation procedure. The input is the user's selection and input information, and the output is request data to the server. Specifically, the information entered by the user is packed into packets and sent to the server.

[0668] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0669] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[0670] Server Roles

[0671] Content Analysis

[0672] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[0673] Database Reference

[0674] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[0675] Information Packet Generation

[0676] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[0677] Emotional information reception

[0678] The server receives emotion information sent from the user's device, including the user's facial expression and tone of voice.

[0679] Information transmission

[0680] The server then sends the generated information packet to the user's device. It is also possible to adjust the product recommendations based on the user's emotional information.

[0681] Device Role

[0682] Information reception and display

[0683] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[0684] Emotional information gathering

[0685] The device is equipped with sensors to collect emotional information such as the user's facial expressions and tone of voice, allowing it to recognize the user's emotions such as joy, surprise, and interest in real time.

[0686] Emotional information transmission

[0687] The device transmits the collected emotion information to the server.

[0688] Supporting User Input

[0689] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[0690] User Behavior

[0691] Reviewing and selecting information

[0692] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[0693] Purchase or reservation process

[0694] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[0695] Specific examples

[0696] Buying a dress while watching a drama

[0697] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[0698] Make restaurant reservations while watching a gourmet program

[0699] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[0700] Use of emotional information

[0701] If a user smiles while watching a drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, the server can present accessories and other recommended products related to the dress the user smiled at.

[0702] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] The server receives real-time media content data that the user is viewing, including video data to be analyzed.

[0706] Step 2:

[0707] The server analyzes the received video data using machine learning algorithms to identify objects (e.g., clothes, bags, interior decor) displayed within the content.

[0708] Step 3:

[0709] The server extracts metadata about the identified objects (e.g., type, color, shape, brand logo).

[0710] Step 4:

[0711] The server queries the product information database based on the extracted metadata to obtain relevant product information (e.g., product details, manufacturer, price, and stock information).

[0712] Step 5:

[0713] The server checks the product information retrieved from the database and generates an information packet containing the necessary details (e.g., product name, price, image, and purchase link).

[0714] Step 6:

[0715] The server transmits the generated information packet to the user's terminal.

[0716] Step 7:

[0717] The terminal receives the information packets sent from the server.

[0718] Step 8:

[0719] The terminal analyzes the contents of the information packet and updates the user interface, displaying detailed product or service information in the corner of the TV screen.

[0720] Step 9:

[0721] The device uses built-in sensors to collect the user's emotional information (e.g., facial expressions, tone of voice) in real time.

[0722] Step 10:

[0723] The terminal transmits the collected emotion information to the server.

[0724] Step 11:

[0725] The server analyzes the received emotion information and recognizes the user's emotions such as joy, surprise, and interest.

[0726] Step 12:

[0727] The server then tailors the displayed product and service recommendations based on the emotion engine, for example, by presenting options and offers related to the products the user has expressed interest in.

[0728] Step 13:

[0729] The terminal displays the adjusted information again on the user interface.

[0730] Step 14:

[0731] The user uses a remote control or touch screen to review the displayed information and select products or services of interest.

[0732] Step 15:

[0733] The device displays detailed information about the product or service the user selects, including a detailed description, reviews, and availability information.

[0734] Step 16:

[0735] The user decides to purchase or reserve and initiates the process using a remote control or touchscreen.

[0736] Step 17:

[0737] The terminal sends the purchase information (e.g., delivery address, credit card information) or reservation information (e.g., date, number of people) entered by the user to the server.

[0738] Step 18:

[0739] The server processes the payment through the payment gateway based on the received purchase information, and in the case of a reservation, connects to the corresponding reservation system to confirm and secure the reservation.

[0740] Step 19:

[0741] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[0742] Step 20:

[0743] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[0744] This series of steps allows users to easily obtain detailed information about products and services while watching TV, receive optimal suggestions based on their emotions, and smoothly make purchases or reservations.

[0745] Example 2

[0746] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0747] In recent years, the number of users watching content on televisions and smart devices has increased, leading to a growing demand for easy access to information on related products and services while watching, and for making purchases or reservations. However, conventional systems require users to search for information based on their interests, and are unable to provide optimal suggestions based on the user's emotions. This results in a lack of integration between the viewing and purchasing experiences, resulting in a lack of user convenience.

[0748] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for acquiring information about the identified objects from a database, means for displaying the acquired information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for collecting user emotion information, means for transmitting the collected emotion information to the server, and means for adjusting proposal content based on the emotion information. This not only makes it possible to easily acquire information about related products and services while viewing, but also makes it possible to make optimal proposals based on the user's emotions.

[0749] "Real-time" refers to the immediate processing of data and events without delay.

[0750] "Media content" is a general term for digital content that includes information such as audio, video, images, and text.

[0751] "Object" refers to a distinct item or element that appears within media content.

[0752] "Identification" refers to identifying what an object is.

[0753] A "database" refers to a collection of data that stores specific information and allows for rapid search and retrieval.

[0754] "Retrieval" refers to the act of retrieving the necessary information from a database.

[0755] "Display device" refers to a screen or display on which a user views information.

[0756] "Purchase" refers to the act of a user buying a product.

[0757] "Reservation" refers to the act of a user applying for a service in advance.

[0758] "Emotional information" refers to data about emotions obtained from a user's facial expressions, tone of voice, behavior, etc.

[0759] "Collection" refers to the act of obtaining emotional information using devices or sensors.

[0760] "Proposal content" refers to information about products and services offered to users.

[0761] "Adjustment" refers to the act of optimizing the suggestions based on the user's emotional state.

[0762] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[0763] The server analyzes the received media content in real time using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16). Specifically, it identifies objects such as the dress worn by an actress in a drama scene or furniture placed in the scene. This allows the user to obtain real-time information about the content they are watching.

[0764] Next, the server retrieves information about the identified object from a database. This data includes product details, manufacturer, price, and availability information. For example, for an identified dress, the server searches the database for the brand name and price information. At this stage, the server generates an information packet based on the retrieved information. The information packet includes a product image, details, price, and a purchase link.

[0765] The device receives the information packet sent from the server and displays it on a user interface (UI). For example, product information can be displayed as a pop-up in the corner of a TV screen. This allows users to easily view detailed information about products and services related to the content they are currently watching.

[0766] The device also uses emotion recognition software (e.g., FaceAPI, EmotionSDK) that collects emotional information such as the user's facial expressions and tone of voice to recognize the user's emotions, such as joy, surprise, and interest, in real time. This allows the device to transmit the collected emotional information to a server. The server can then adjust the product and service recommendations it provides to the user based on the collected emotional information.

[0767] For example, if a user smiles while watching a TV drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, it can present accessories and other recommended products related to the dress the user smiled at.

[0768] Below are some examples of prompt sentences:

[0769] Example prompt sentence:

[0770] A prompt that displays details about a dress worn by an actress in a scene from a drama, along with a link to purchase it.

[0771] "Please provide the brand name, price, and purchase link of the dress worn by the actress in the scene of the media content you are currently watching."

[0772] A prompt that provides information about a restaurant featured during a foodie show and a reservation link.

[0773] "View detailed information and reservation links for featured restaurants."

[0774] A prompt that recognizes joy in the user's facial expression and suggests specific products based on that emotion.

[0775] "Suggest accessories and other product recommendations related to the specific product that made the user smile."

[0776] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[0777] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0778] Step 1: Content Analysis

[0779] The server analyzes the incoming media content in real time, using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16) to perform frame-by-frame analysis. The input is the video data, and the machine learning model labels the objects in the frames. The output is a list of identified objects.

[0780] Specific behavior:

[0781] The server captures the video data and sends it to the analysis engine.

[0782] A machine learning model labels and identifies objects in each frame.

[0783] Compile information about the identified objects into a list.

[0784] Step 2: Database Reference

[0785] The server uses the list of identified objects to retrieve relevant information from a database. The input is the object's label information, and it generates and executes a database query. The output is product details, manufacturer, price, and availability information.

[0786] Specific behavior:

[0787] The server generates an SQL query based on the identification information.

[0788] A query is sent to the database to retrieve the relevant product information.

[0789] The acquired information is organized and packetized.

[0790] Step 3: Information Packet Generation

[0791] The server generates an information packet to provide to the user based on the retrieved product information. The input is the product information retrieved from the database, and the output is an information packet (JSON format) sent to the user.

[0792] Specific behavior:

[0793] Product information is converted into JSON format and packetized.

[0794] Generate an information packet containing product images, details, prices, and purchase links.

[0795] Step 4: Emotional information gathering

[0796] The device collects emotional information such as the user's facial expressions and tone of voice using emotion recognition software (e.g., FaceAPI, EmotionSDK). The input is sensor data from the camera and microphone, and the output is preprocessed emotional data.

[0797] Specific behavior:

[0798] The device will initialize the camera and microphone and start emotion recognition software.

[0799] Monitors the user's facial expressions and voice in real time.

[0800] The collected data is analyzed to identify emotional states.

[0801] Step 5: Emotional information transmission

[0802] The device sends the collected emotion information to the server. The input is the preprocessed emotion data, and the output is the emotion data sent to the server.

[0803] Specific behavior:

[0804] Convert the emotion data into a suitable format.

[0805] Send emotion data to the server's API endpoint using an HTTP POST request.

[0806] Step 6: Receiving and displaying information

[0807] The terminal receives the information packet sent from the server and displays it on the user interface (UI). The input is the information packet from the server, and the output is the product information displayed on the UI.

[0808] Specific behavior:

[0809] Receives an HTTP response from the server.

[0810] Parse the information packet and update the UI components.

[0811] A pop-up is displayed at an appropriate position on the TV screen to present product information.

[0812] Step 7: Supporting User Input

[0813] The terminal allows the user to operate the displayed information using a remote control or touch screen. The input is an operation signal from the user, and the output is a transition to a purchase screen or detailed information on the selected product.

[0814] Specific behavior:

[0815] It detects input from the remote control and processes the user's selection operations.

[0816] Detailed information about the selected product will be displayed, and you will be taken to the purchase or reservation screen.

[0817] Step 8: Purchase or reservation process

[0818] The user then completes the purchase or reservation procedure for the selected product or service. The input is the user's shipping information and payment information, and the output is a confirmation of the purchase or reservation.

[0819] Specific behavior:

[0820] A user input form is displayed on the device's UI, prompting the user to enter shipping address and payment information.

[0821] The input information is sent to the server and the purchase or reservation procedure is carried out.

[0822] Step 9: Use emotional information

[0823] The server adjusts the product and service recommendations provided to the user based on the emotion information. The input is the collected emotion information, and the output is the adjusted product recommendation information.

[0824] Specific behavior:

[0825] The server analyzes the emotional data and generates a list of recommended products based on pleasure and interest.

[0826] The recommended product list is included in an information packet and sent to the terminal, which then displays it on the UI.

[0827] (Application example 2)

[0828] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0829] With conventional systems, it was difficult for users to obtain real-time information about products and services displayed while watching television and make purchases or reservations on the spot.In addition, the system did not realize optimal product and service recommendations based on user emotions, making it difficult to improve the user experience.

[0830] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0831] In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a database, means for displaying the retrieved information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for analyzing and retrieving emotional information about the user using an emotion recognition engine, and means for adjusting the content provided to the user based on the retrieved emotional information. This allows the user to obtain information about products and services related to the content they are viewing in real time, making it easy to purchase or reserve, and also allowing them to receive optimal product recommendations based on their emotions.

[0832] "Real-time" refers to information being processed immediately at the moment it is generated.

[0833] "Media content" refers to a collection of visual and audio information such as television programs, videos, and images.

[0834] "Object" refers to a specific item that appears in the content, such as clothing, furniture, or other real-world objects.

[0835] "Database" refers to a centralized information management system for systematically storing and managing detailed information about products and services.

[0836] A "display device" refers to a device that allows a user to visually check information, such as a television or smartphone.

[0837] "Purchase or reservation process" means the series of steps required to purchase a product or service selected by a User or the process for confirming a reservation.

[0838] An "emotion recognition engine" refers to a software system that analyzes and recognizes emotions from a user's facial expressions and voice.

[0839] "Operation device" refers to an interactive device that allows a user to select or input information, such as a remote control or touch screen.

[0840] "Adjusting the content of information provided" refers to changing the content of information presented to the user based on the acquired emotion information.

[0841] This system provides real-time information about products and services related to objects displayed in media content that a user is viewing, and makes optimal suggestions using an emotion recognition engine. This system consists of a server, a user's device (such as a television or smartphone), and an emotion recognition engine.

[0842] Server Roles

[0843] The server analyzes the received media content in real time, identifies objects displayed within the content, retrieves information about the identified objects from a database, generates information packets for providing this information to the user's display device, and receives emotion information transmitted from the user's device and adjusts the content of recommended products and services based on the information.

[0844] Device Role

[0845] The terminal receives the information packets sent from the server and displays them on a user interface. The display device, such as a television or smartphone, presents detailed information about products and services related to the content the user is viewing. The terminal also uses sensors such as a camera and microphone to capture the user's facial expressions and voice and transmits this information to the server as emotional information.

[0846] User Behavior

[0847] Users can check the product or service information displayed on their device and proceed with the purchase or reservation if they are interested. For example, a user who is interested in a dress featured in a drama can view detailed information and proceed with the purchase. In addition, an emotion recognition engine analyzes the user's emotions, and based on the results, the user can receive suggestions for more attractive products and services.

[0848] Hardware and software used

[0849] Hardware: Smartphone camera and microphone

[0850] Software: OpenCV (image processing technology), TensorFlow (machine learning model), EmotionEngine (emotion recognition engine)

[0851] Processing flow

[0852] The server first analyzes the received media content and performs object recognition. Next, it retrieves detailed information about the object from a database, creates a packet containing that information, and sends it to the device. The device receives the packet and displays the information on its screen. At the same time, the device analyzes the user's facial expressions and voice using an emotion recognition engine and sends the emotional information to the server. The server then adjusts product and service recommendations based on the acquired emotional information and provides them to the user in real time.

[0853] Specific examples

[0854] When a user is watching a product introduction video on an online shopping site, their facial expressions are captured with their smartphone camera and analyzed by an emotion recognition engine. Information about the dress recognized in the video is displayed in real time, and if the user shows interest, related accessories and other items are also recommended. The following is an example of a prompt sentence to be input into the generative AI model.

[0855] Prompt Sentence Examples

[0856] Write a Python program that recognizes emotions from facial expressions and voices while a user is watching a video on a shopping site using a smartphone camera, and displays product information (e.g., dresses or furniture) identified in the video in real time. Considering the emotional signals, recommend products that are particularly appealing to the user if they are happy.

[0857] This system simultaneously improves the viewing and purchasing experience, enabling optimal service delivery based on user emotions.

[0858] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0859] Step 1:

[0860] Input: The media content the user is watching

[0861] Specific operation: The server receives media content in real time and performs content analysis.

[0862] Data processing and data calculation: Using object identification algorithms, we identify objects within the content using video recognition technology and machine learning models.

[0863] Output: List of identified objects

[0864] Step 2:

[0865] Input: List of identified objects

[0866] What happens next: The server looks up the database and retrieves detailed information about the identified object.

[0867] Data processing and data manipulation: Using database queries to extract detailed information about an object (e.g., product name, price, availability).

[0868] Output: An information packet containing detailed information

[0869] Step 3:

[0870] Input: Information packet containing detailed information

[0871] Specific operation: The server sends an information packet to the user's terminal.

[0872] Data processing and data calculation: Encoding information packets into the appropriate format and sending them over the network.

[0873] Output: Information packets received at the user's terminal

[0874] Step 4:

[0875] Input: Information packet received at the user's terminal

[0876] Specific operation: The terminal decodes the information packet and displays it on the user interface.

[0877] Data processing and data calculation: Displaying item information (images, text, links, etc.) on the screen to visualize the decoded information.

[0878] Output: Product or service information that users can visually confirm

[0879] Step 5:

[0880] Input: User's face image and voice

[0881] How it works: The device's built-in camera and microphone capture the user's facial expressions and voice in real time.

[0882] Data processing and data calculation: An emotion recognition engine is used to analyze the user's emotional information from the captured facial expressions and voice.

[0883] Output: Emotional information (e.g., joy, surprise, interest)

[0884] Step 6:

[0885] Input: Emotion information

[0886] Specific operation: The device transmits the acquired emotional information to the server.

[0887] Data processing and data calculation: Encode emotional information into an appropriate format and transmit it over the network.

[0888] Output: Emotion information received by the server

[0889] Step 7:

[0890] Input: Emotion information received by the server

[0891] Specific operation: The server adjusts the products and services it recommends to the user based on the emotional information.

[0892] Data processing and calculation: Combining emotional information and object information, we execute algorithms to select optimal products and services.

[0893] Output: Adjusted recommendations for the user

[0894] Step 8:

[0895] Input: Adjusted proposal information

[0896] Specific operation: The server sends the adjusted proposal information back to the device and provides it to the user.

[0897] Data processing and data calculation: The proposed information is encoded as a packet and sent over the network.

[0898] Output: Adjusted proposal information received on the user's device

[0899] Step 9:

[0900] Input: Adjusted proposal information

[0901] Specific operation: The device displays information about suggested products and services tailored to the user interface.

[0902] Data processing and calculation: The proposed information after adjustment is visualized and displayed for the user to check.

[0903] Output: Information about the adjusted products and services that can be visually confirmed by the user

[0904] Step 10:

[0905] Input: Information about the adjusted proposed products and services

[0906] Specific action: The user selects the suggested product or service and completes the purchase or reservation process.

[0907] Data processing and calculation: Enter the necessary information for the purchase or reservation (shipping address and payment information) and finalize the transaction.

[0908] Output: Confirmation of completed purchase or reservation

[0909] By executing the above steps in order, the present invention enables the user to obtain information on products and services related to the content the user is viewing in real time and receive optimal suggestions based on emotions.

[0910] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0911] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0912] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0913] [Third embodiment]

[0914] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0915] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0916] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0917] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0918] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0919] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0920] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0921] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0922] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0923] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0924] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0925] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0926] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve. The system is primarily composed of a server and user terminals (televisions or smart devices).

[0927] Server Roles

[0928] Content Analysis

[0929] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[0930] Database Reference

[0931] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[0932] Information Packet Generation

[0933] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[0934] Information transmission

[0935] The server transmits the generated information packet to the user's terminal.

[0936] Device Role

[0937] Information reception and display

[0938] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[0939] Supporting User Input

[0940] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[0941] User Behavior

[0942] Reviewing and selecting information

[0943] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[0944] Purchase or reservation process

[0945] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[0946] Specific examples

[0947] Buying a dress while watching a drama

[0948] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[0949] Make restaurant reservations while watching a gourmet program

[0950] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[0951] In this way, the system helps users easily obtain detailed information about products and services and make purchases or reservations while watching TV, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[0952] The processing flow will be explained below.

[0953] Step 1:

[0954] The server receives real-time media content data for viewing by the user, which includes video data such as television programs and streaming videos.

[0955] Step 2:

[0956] The server analyzes the received video data and identifies objects displayed within the content, using video recognition technology and machine learning algorithms.

[0957] Step 3:

[0958] The server extracts metadata about the identified objects, including characteristics such as the object's type, color, shape, brand logo, etc.

[0959] Step 4:

[0960] The server uses the extracted metadata to query a product information database, which contains detailed information about various products and services.

[0961] Step 5:

[0962] The server checks the product or service information retrieved from the database and retrieves the necessary details, such as product name, manufacturer, price, and availability information.

[0963] Step 6:

[0964] The server generates an information packet based on the information obtained, which includes product images, details, prices, and purchase links.

[0965] Step 7:

[0966] The server transmits the generated information packet to the user's terminal.

[0967] Step 8:

[0968] The terminal receives the information packets sent from the server.

[0969] Step 9:

[0970] The device analyzes the contents of the information packet and updates the user interface (UI), displaying detailed product or service information in the corner of the screen.

[0971] Step 10:

[0972] The user reviews the displayed information and selects the product or service they are interested in. For example, they use a remote control to select a particular product and view detailed information.

[0973] Step 11:

[0974] The device displays detailed information about the product or service selected by the user, including a detailed product description, reviews, and availability information.

[0975] Step 12:

[0976] The user confirms the displayed detailed information and performs the purchase or reservation procedure, for example, by entering credit card information and a delivery address.

[0977] Step 13:

[0978] The terminal transmits the purchase or reservation information entered by the user to the server.

[0979] Step 14:

[0980] The server processes the received purchase or reservation information, processes the payment through a payment gateway, and in the case of a reservation, connects to a reservation system to confirm and secure the reservation.

[0981] Step 15:

[0982] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[0983] Step 16:

[0984] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[0985] This series of steps allows users to smoothly obtain detailed information about products and services and make purchases or reservations while watching television.

[0986] Example 1

[0987] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0988] In the conventional television viewing experience, users must manually search for information on products and services related to the content they are watching, which disrupts the viewing experience. Furthermore, even if users are interested in a product or service, the process of actually purchasing or reserving it can be cumbersome, potentially discouraging many users from making a purchase. Therefore, there is a need for a system that provides real-time information on related products and services while watching television, enabling seamless purchasing and reserving procedures.

[0989] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0990] In this invention, the server includes means for analyzing video information received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a data store, and means for formatting the retrieved information to generate information packets, thereby enabling users to obtain information about products and services related to the content they are viewing in real time and to carry out seamless purchasing and reservation procedures.

[0991] "Real-time" means that data and information are processed immediately and results are provided without delay.

[0992] "Video information" refers to media content that includes video data and related information.

[0993] "Object" refers to any object displayed within the Content, including elements related to specific products or services.

[0994] A "data store" is a database or other persistent storage device for storing data and accessing it as needed.

[0995] An "information packet" is a collection of data that is sent to a user terminal after organizing and processing the acquired information.

[0996] A "user terminal" refers to an information processing device used by a user, including a television or smart device.

[0997] A "display means" is a device or interface for visually displaying information.

[0998] A "remote control device" is a device that allows a user to operate or control a system using a device that includes a remote control or a touch screen.

[0999] "Image recognition technology" is a technology that allows computer systems to identify specific objects or patterns from image data.

[1000] A "machine learning algorithm" is a computational method that allows a computer system to analyze data and learn from experience to improve its performance.

[1001] This invention is a system that analyzes video information received in real time, identifies objects displayed in the content, provides users with information related to those objects, and facilitates purchase and reservation procedures. The system is primarily composed of a server and a user terminal (such as a television or smart device). Specific embodiments are described below.

[1002] 1. Server Role

[1003] The server receives video information from TVs and smart devices in real time and analyzes it. It uses video recognition technology and machine learning algorithms (e.g., OpenCV and TensorFlow) to identify objects displayed within the content. Information about the identified objects is retrieved by submitting SQL queries to a relational database management system (e.g., MySQL).

[1004] The server generates an information packet based on the information obtained. This information packet contains product images, details, prices, purchase links, etc., formatted in JSON format. The information packet is then sent to the user's device using the HTTP protocol.

[1005] Examples:

[1006] The server analyzes a scene from a drama and identifies the dress worn by the actress.

[1007] The drama footage is analyzed frame by frame to identify the dress.

[1008] The identified dress information is retrieved from the data store and an information packet is generated that includes the brand name, price, and availability information.

[1009] The information packet is sent to the user terminal via an HTTP POST request.

[1010] Example prompt sentence:

[1011] "Please explain the steps to get details about the dress worn by the actress in the drama I'm watching and check out."

[1012] 2. Role of the terminal

[1013] The device receives the information packets sent by the server and displays them in a user interface (UI), for example, product information displayed as a popup in the corner of a TV screen. The device uses the React library to visually represent this information.

[1014] Additionally, the device allows users to interact with the displayed information using a remote control or touchscreen. Users can press buttons on the remote control to select specific products, view more information, or proceed with the purchase or reservation process.

[1015] Examples:

[1016] When a user uses the remote control to view details about a dress

[1017] The information packet is received and details about the dress pop up in the corner of the TV screen.

[1018] The user presses a button on the remote control to go to the details page and proceed with the purchase procedure.

[1019] 3. User Behavior

[1020] The user checks the displayed information and selects the product or service they are interested in. For the selected product or service, the user then proceeds to purchase or reserve it. For example, when purchasing a dress, the user enters shipping information and credit card information to complete the payment. When making a reservation, the user enters the date and number of people to confirm the reservation.

[1021] Examples:

[1022] When a user purchases a dress that appears in a drama

[1023] The user selects detailed information about the dress from the displayed pop-up using the remote control, and is taken to a purchase details page.

[1024] Enter the required shipping and payment information and click the "Purchase" button to confirm your purchase.

[1025] This system allows users to easily obtain information about products and services related to the content they are viewing, and allows them to seamlessly complete purchase and reservation procedures, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[1026] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1027] Step 1:

[1028] Receiving content in real time

[1029] Input: Video stream from your TV or smart device.

[1030] What it does: The server receives the video stream in real time from the specified channel URL using a high-speed network connection and streaming server software (e.g., Wowza Streaming Engine).

[1031] Data processing: The received video stream is temporarily saved as a local file.

[1032] Output: Locally saved video stream file.

[1033] Step 2:

[1034] Content Analysis and Object Identification

[1035] Input: Locally stored video stream file.

[1036] How it works: The server analyzes the received video frame by frame and uses video recognition software (e.g., OpenCV) and machine learning algorithms (e.g., TensorFlow) to identify objects within the content.

[1037] Data processing: Each frame is analyzed sequentially and identified objects are labeled.

[1038] Output: A list of identified objects and their labels.

[1039] Step 3:

[1040] Retrieving information from a database

[1041] Input: A list of identified objects and their labels.

[1042] Specific operation: The server issues an SQL query to a relational database management system (e.g., MySQL) to retrieve information related to the object.

[1043] Data processing: Send a query to the database using the object label as a key to retrieve the corresponding data.

[1044] Output: Detailed data such as product information, manufacturer information, price information, and stock information retrieved from the database.

[1045] Step 4:

[1046] Generate information packets

[1047] Input: Detailed data retrieved from the database.

[1048] What happens: The server formats the information it receives and creates packets of information to serve to the user. This process uses web server software (e.g., Apache) and a scripting language (e.g., Python).

[1049] Data processing: Formatting the data and generating information packets in JSON format.

[1050] Output: The generated information packet in JSON format.

[1051] Step 5:

[1052] Sending information packets

[1053] Input: A packet of information in JSON format.

[1054] Specific operation: The server sends an information packet to the user's terminal using the HTTP protocol.

[1055] Data processing: Create an HTTP POST request and send the information packet to the user terminal.

[1056] Output: Information packets sent to the user terminal.

[1057] Step 6:

[1058] Receiving and displaying information on the user terminal

[1059] Input: A JSON formatted information packet sent by the server.

[1060] Specific operation: The device receives and analyzes the information packet. Then, it displays the analyzed information on the user interface (UI). The React library is used for display.

[1061] Data processing: Analyzes the received JSON data and passes the necessary information to the UI components for display.

[1062] Output: Product information displayed as a pop-up in the corner of the TV screen.

[1063] Step 7:

[1064] User-input operation

[1065] Input: Product information displayed on the user interface, and user input operations.

[1066] Specific behavior: The device accepts user input via a remote control or touchscreen, allowing the user to manipulate the displayed information. For example, the user presses a button on the remote control to select a particular product and view its details.

[1067] Data manipulation: Accepts user input and triggers relevant actions.

[1068] Output: A screen showing detailed product information.

[1069] Step 8:

[1070] Completing a purchase or reservation

[1071] Input: Shipping, payment, and reservation information entered by the user.

[1072] Specific operation: The terminal sends the information entered in the form to the server and calls an API to process the payment or reservation.

[1073] Data processing: The entered information is sent to the server as an HTTP POST request to process payments and reservations.

[1074] Output: A confirmation message that the purchase or reservation process is complete.

[1075] (Application example 1)

[1076] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1077] Conventional systems that provide information about products and services while watching television lack the functionality to instantly check, purchase, or reserve products of interest to users. Furthermore, technology to identify specific objects within a video frame and provide related information in real time has not yet been fully developed. Therefore, improving usability and seamlessly integrating the viewing and purchasing experiences have been key challenges.

[1078] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1079] In this invention, the server includes means for analyzing the media content received in real time and using a deep learning model to identify a specific object in a video frame, means for retrieving information about the identified object from a database, and means for displaying the retrieved information on a user's display device, thereby providing product information or service information related to the specific object in the media content in real time, allowing the user to make an immediate purchase or reservation.

[1080] In addition, if the display device is a smart device, it further includes means for providing an interface for the user to make selections and inputs by voice or touch operation, thereby enabling the user to easily check product information, purchase, and reserve using voice commands or touch operation, improving convenience.

[1081] "Media content" refers to information in digital form that includes video and audio, and refers to viewable content such as television programs, movies, and videos.

[1082] A "video frame" is one of the still images that make up a digital video, and when displayed in succession, it is perceived as a moving image.

[1083] "Specific objects" refers to specific items or people that appear in the video frame, including clothing, furniture, accessories, etc.

[1084] A "deep learning model" is a type of artificial intelligence technology, a computational model that uses large amounts of data and advanced algorithms to recognize objects and patterns.

[1085] A "database" is a system that systematically organizes and stores specific information in a searchable manner, and stores product information, service information, etc.

[1086] A "display device" is a device for displaying information in a form that can be viewed by a user, and includes smartphones, tablets, televisions, and the like.

[1087] An "interface" is the means by which a user interacts with a system, including remote controls, touchscreens, voice recognition, etc.

[1088] "Usability" refers to the characteristics that make a system or service easy for users to use and understand, and includes efficiency, effectiveness, and satisfaction of use.

[1089] The details of the embodiments of the present invention are described below.

[1090] The server analyzes the received media content in real time and uses a deep learning model to identify specific objects contained within the video frames. For example, the server targets the clothes worn by actors in a movie scene or the furniture used in the scene. Publicly available deep learning models (e.g., YOLO, TensorFlow) can be used for this identification. Information about the identified objects is obtained by accessing a database on the server. This database stores detailed information about products and services (such as names, prices, availability, and image URLs).

[1091] The terminal displays the acquired information on the user's display device. If the display device is a smart device (such as a smartphone, tablet, or television), it provides an interface that allows the user to make selections and inputs using voice or touch operations. This allows the user to check product information related to a specific object and make direct purchases or reservations.

[1092] For example, if a user is interested in a bag that appears in a scene while watching a drama, the server can identify the bag, obtain related details, and send them to the display device. The user can then confirm the bag information by voice or touch operation and proceed with the purchase process.

[1093] The entire system is built using commercially available hardware and software, including high-performance servers, smart devices, deep learning models (YOLO, TensorFlow), and communication interfaces (API, HTTP / HTTPS).

[1094] An example prompt for a generative AI model is:

[1095] "Your goal is to create an AI model that can identify specific objects in a video frame and retrieve related product information. Specifically, it should focus on objects being held or items being worn, and provide that information from a database."

[1096] This allows users to enjoy a seamless viewing and purchasing experience.

[1097] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1098] Step 1:

[1099] The server receives and analyzes video frames of media content received in real time. Specifically, the server receives streaming data, extracts video frames from the data, and starts processing. The input data used in this process is the received media content, and the output is a list of analyzed video frames.

[1100] Step 2:

[1101] The server uses a deep learning model (e.g., YOLO, TensorFlow) to identify specific objects contained within video frames. The input is the list of video frames obtained in step 1, and the model detects objects within the frames and identifies their type and location. The output is a list of identified objects and their locations. Specifically, each frame is input into the deep learning model, and the object detection results are obtained.

[1102] Step 3:

[1103] The server retrieves detailed information about the identified objects from the database. The input is a list of object information identified in step 2, and searches and retrieves detailed information about the corresponding objects (e.g., product name, price, stock status, image URL, etc.) from the database. The output is a list of detailed information corresponding to each object. Specifically, the server executes a query to the database using the object name as a key, and compiles the results as a list.

[1104] Step 4:

[1105] The server generates an information packet based on the acquired detailed information and sends it to the terminal. The input is the list of detailed information obtained in step 3, and the output is an information packet. Specifically, the server converts the detailed information of each object into a packet format and generates data to be sent to the terminal.

[1106] Step 5:

[1107] The terminal displays the information packets received from the server on a display device. The input is the information packets sent from the server, and the output is product or service information displayed on the user's display device. Specific operations include analyzing the information packets and displaying product images and detailed information on the screen.

[1108] Step 6:

[1109] The user selects a specific product or service based on the displayed information and completes the purchase or reservation process. The input is the detailed product or service information displayed on the device, and the output is the user's selection and the input information (e.g., delivery address, payment information, reservation date, etc.). Specific operations involve the user selecting information using touch operations or voice commands and entering the necessary information.

[1110] Step 7:

[1111] The terminal sends the user's selection and input information to the server and proceeds with the purchase or reservation procedure. The input is the user's selection and input information, and the output is request data to the server. Specifically, the information entered by the user is packed into packets and sent to the server.

[1112] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1113] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[1114] Server Roles

[1115] Content Analysis

[1116] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[1117] Database Reference

[1118] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[1119] Information Packet Generation

[1120] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[1121] Emotional information reception

[1122] The server receives emotion information sent from the user's device, including the user's facial expression and tone of voice.

[1123] Information transmission

[1124] The server then sends the generated information packet to the user's device. It is also possible to adjust the product recommendations based on the user's emotional information.

[1125] Device Role

[1126] Information reception and display

[1127] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[1128] Emotional information gathering

[1129] The device is equipped with sensors to collect emotional information such as the user's facial expressions and tone of voice, allowing it to recognize the user's emotions such as joy, surprise, and interest in real time.

[1130] Emotional information transmission

[1131] The device transmits the collected emotion information to the server.

[1132] Supporting User Input

[1133] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[1134] User Behavior

[1135] Reviewing and selecting information

[1136] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[1137] Purchase or reservation process

[1138] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[1139] Specific examples

[1140] Buying a dress while watching a drama

[1141] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[1142] Make restaurant reservations while watching a gourmet program

[1143] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[1144] Use of emotional information

[1145] If a user smiles while watching a drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, the server can present accessories and other recommended products related to the dress the user smiled at.

[1146] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[1147] The processing flow will be explained below.

[1148] Step 1:

[1149] The server receives real-time media content data that the user is viewing, including video data to be analyzed.

[1150] Step 2:

[1151] The server analyzes the received video data using machine learning algorithms to identify objects (e.g., clothes, bags, interior decor) displayed within the content.

[1152] Step 3:

[1153] The server extracts metadata about the identified objects (e.g., type, color, shape, brand logo).

[1154] Step 4:

[1155] The server queries the product information database based on the extracted metadata to obtain relevant product information (e.g., product details, manufacturer, price, and stock information).

[1156] Step 5:

[1157] The server checks the product information retrieved from the database and generates an information packet containing the necessary details (e.g., product name, price, image, and purchase link).

[1158] Step 6:

[1159] The server transmits the generated information packet to the user's terminal.

[1160] Step 7:

[1161] The terminal receives the information packets sent from the server.

[1162] Step 8:

[1163] The terminal analyzes the contents of the information packet and updates the user interface, displaying detailed product or service information in the corner of the TV screen.

[1164] Step 9:

[1165] The device uses built-in sensors to collect the user's emotional information (e.g., facial expressions, tone of voice) in real time.

[1166] Step 10:

[1167] The terminal transmits the collected emotion information to the server.

[1168] Step 11:

[1169] The server analyzes the received emotion information and recognizes the user's emotions such as joy, surprise, and interest.

[1170] Step 12:

[1171] The server then tailors the displayed product and service recommendations based on the emotion engine, for example, by presenting options and offers related to the products the user has expressed interest in.

[1172] Step 13:

[1173] The terminal displays the adjusted information again on the user interface.

[1174] Step 14:

[1175] The user uses a remote control or touch screen to review the displayed information and select products or services of interest.

[1176] Step 15:

[1177] The device displays detailed information about the product or service the user selects, including a detailed description, reviews, and availability information.

[1178] Step 16:

[1179] The user decides to purchase or reserve and initiates the process using a remote control or touchscreen.

[1180] Step 17:

[1181] The terminal sends the purchase information (e.g., delivery address, credit card information) or reservation information (e.g., date, number of people) entered by the user to the server.

[1182] Step 18:

[1183] The server processes the payment through the payment gateway based on the received purchase information, and in the case of a reservation, connects to the corresponding reservation system to confirm and secure the reservation.

[1184] Step 19:

[1185] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[1186] Step 20:

[1187] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[1188] This series of steps allows users to easily obtain detailed information about products and services while watching TV, receive optimal suggestions based on their emotions, and smoothly make purchases or reservations.

[1189] Example 2

[1190] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1191] In recent years, the number of users watching content on televisions and smart devices has increased, leading to a growing demand for easy access to information on related products and services while watching, and for making purchases or reservations. However, conventional systems require users to search for information based on their interests, and are unable to provide optimal suggestions based on the user's emotions. This results in a lack of integration between the viewing and purchasing experiences, resulting in a lack of user convenience.

[1192] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for acquiring information about the identified objects from a database, means for displaying the acquired information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for collecting user emotion information, means for transmitting the collected emotion information to the server, and means for adjusting proposal content based on the emotion information. This not only makes it possible to easily acquire information about related products and services while viewing, but also makes it possible to make optimal proposals based on the user's emotions.

[1193] "Real-time" refers to the immediate processing of data and events without delay.

[1194] "Media content" is a general term for digital content that includes information such as audio, video, images, and text.

[1195] "Object" refers to a distinct item or element that appears within media content.

[1196] "Identification" refers to identifying what an object is.

[1197] A "database" refers to a collection of data that stores specific information and allows for rapid search and retrieval.

[1198] "Retrieval" refers to the act of retrieving the necessary information from a database.

[1199] "Display device" refers to a screen or display on which a user views information.

[1200] "Purchase" refers to the act of a user buying a product.

[1201] "Reservation" refers to the act of a user applying for a service in advance.

[1202] "Emotional information" refers to data about emotions obtained from a user's facial expressions, tone of voice, behavior, etc.

[1203] "Collection" refers to the act of obtaining emotional information using devices or sensors.

[1204] "Proposal content" refers to information about products and services offered to users.

[1205] "Adjustment" refers to the act of optimizing the suggestions based on the user's emotional state.

[1206] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[1207] The server analyzes the received media content in real time using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16). Specifically, it identifies objects such as the dress worn by an actress in a drama scene or furniture placed in the scene. This allows the user to obtain real-time information about the content they are watching.

[1208] Next, the server retrieves information about the identified object from a database. This data includes product details, manufacturer, price, and availability information. For example, for an identified dress, the server searches the database for the brand name and price information. At this stage, the server generates an information packet based on the retrieved information. The information packet includes a product image, details, price, and a purchase link.

[1209] The device receives the information packet sent from the server and displays it on a user interface (UI). For example, product information can be displayed as a pop-up in the corner of a TV screen. This allows users to easily view detailed information about products and services related to the content they are currently watching.

[1210] The device also uses emotion recognition software (e.g., FaceAPI, EmotionSDK) that collects emotional information such as the user's facial expressions and tone of voice to recognize the user's emotions, such as joy, surprise, and interest, in real time. This allows the device to transmit the collected emotional information to a server. The server can then adjust the product and service recommendations it provides to the user based on the collected emotional information.

[1211] For example, if a user smiles while watching a TV drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, it can present accessories and other recommended products related to the dress the user smiled at.

[1212] Below are some examples of prompt sentences:

[1213] Example prompt sentence:

[1214] A prompt that displays details about a dress worn by an actress in a scene from a drama, along with a link to purchase it.

[1215] "Please provide the brand name, price, and purchase link of the dress worn by the actress in the scene of the media content you are currently watching."

[1216] A prompt that provides information about a restaurant featured during a foodie show and a reservation link.

[1217] "View detailed information and reservation links for featured restaurants."

[1218] A prompt that recognizes joy in the user's facial expression and suggests specific products based on that emotion.

[1219] "Suggest accessories and other product recommendations related to the specific product that made the user smile."

[1220] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[1221] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1222] Step 1: Content Analysis

[1223] The server analyzes the incoming media content in real time, using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16) to perform frame-by-frame analysis. The input is the video data, and the machine learning model labels the objects in the frames. The output is a list of identified objects.

[1224] Specific behavior:

[1225] The server captures the video data and sends it to the analysis engine.

[1226] A machine learning model labels and identifies objects in each frame.

[1227] Compile information about the identified objects into a list.

[1228] Step 2: Database Reference

[1229] The server uses the list of identified objects to retrieve relevant information from a database. The input is the object's label information, and it generates and executes a database query. The output is product details, manufacturer, price, and availability information.

[1230] Specific behavior:

[1231] The server generates an SQL query based on the identification information.

[1232] A query is sent to the database to retrieve the relevant product information.

[1233] The acquired information is organized and packetized.

[1234] Step 3: Information Packet Generation

[1235] The server generates an information packet to provide to the user based on the retrieved product information. The input is the product information retrieved from the database, and the output is an information packet (JSON format) sent to the user.

[1236] Specific behavior:

[1237] Product information is converted into JSON format and packetized.

[1238] Generate an information packet containing product images, details, prices, and purchase links.

[1239] Step 4: Emotional information gathering

[1240] The device collects emotional information such as the user's facial expressions and tone of voice using emotion recognition software (e.g., FaceAPI, EmotionSDK). The input is sensor data from the camera and microphone, and the output is preprocessed emotional data.

[1241] Specific behavior:

[1242] The device will initialize the camera and microphone and start emotion recognition software.

[1243] Monitors the user's facial expressions and voice in real time.

[1244] The collected data is analyzed to identify emotional states.

[1245] Step 5: Emotional information transmission

[1246] The device sends the collected emotion information to the server. The input is the preprocessed emotion data, and the output is the emotion data sent to the server.

[1247] Specific behavior:

[1248] Convert the emotion data into a suitable format.

[1249] Send emotion data to the server's API endpoint using an HTTP POST request.

[1250] Step 6: Receiving and displaying information

[1251] The terminal receives the information packet sent from the server and displays it on the user interface (UI). The input is the information packet from the server, and the output is the product information displayed on the UI.

[1252] Specific behavior:

[1253] Receives an HTTP response from the server.

[1254] Parse the information packet and update the UI components.

[1255] A pop-up is displayed at an appropriate position on the TV screen to present product information.

[1256] Step 7: Supporting User Input

[1257] The terminal allows the user to operate the displayed information using a remote control or touch screen. The input is an operation signal from the user, and the output is a transition to a purchase screen or detailed information on the selected product.

[1258] Specific behavior:

[1259] It detects input from the remote control and processes the user's selection operations.

[1260] Detailed information about the selected product will be displayed, and you will be taken to the purchase or reservation screen.

[1261] Step 8: Purchase or reservation process

[1262] The user then completes the purchase or reservation procedure for the selected product or service. The input is the user's shipping information and payment information, and the output is a confirmation of the purchase or reservation.

[1263] Specific behavior:

[1264] A user input form is displayed on the device's UI, prompting the user to enter shipping address and payment information.

[1265] The input information is sent to the server and the purchase or reservation procedure is carried out.

[1266] Step 9: Use emotional information

[1267] The server adjusts the product and service recommendations provided to the user based on the emotion information. The input is the collected emotion information, and the output is the adjusted product recommendation information.

[1268] Specific behavior:

[1269] The server analyzes the emotional data and generates a list of recommended products based on pleasure and interest.

[1270] The recommended product list is included in an information packet and sent to the terminal, which then displays it on the UI.

[1271] (Application example 2)

[1272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1273] With conventional systems, it was difficult for users to obtain real-time information about products and services displayed while watching television and make purchases or reservations on the spot.In addition, the system did not realize optimal product and service recommendations based on user emotions, making it difficult to improve the user experience.

[1274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1275] In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a database, means for displaying the retrieved information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for analyzing and retrieving emotional information about the user using an emotion recognition engine, and means for adjusting the content provided to the user based on the retrieved emotional information. This allows the user to obtain information about products and services related to the content they are viewing in real time, making it easy to purchase or reserve, and also allowing them to receive optimal product recommendations based on their emotions.

[1276] "Real-time" refers to information being processed immediately at the moment it is generated.

[1277] "Media content" refers to a collection of visual and audio information such as television programs, videos, and images.

[1278] "Object" refers to a specific item that appears in the content, such as clothing, furniture, or other real-world objects.

[1279] "Database" refers to a centralized information management system for systematically storing and managing detailed information about products and services.

[1280] A "display device" refers to a device that allows a user to visually check information, such as a television or smartphone.

[1281] "Purchase or reservation process" means the series of steps required to purchase a product or service selected by a User or the process for confirming a reservation.

[1282] An "emotion recognition engine" refers to a software system that analyzes and recognizes emotions from a user's facial expressions and voice.

[1283] "Operation device" refers to an interactive device that allows a user to select or input information, such as a remote control or touch screen.

[1284] "Adjusting the content of information provided" refers to changing the content of information presented to the user based on the acquired emotion information.

[1285] This system provides real-time information about products and services related to objects displayed in media content that a user is viewing, and makes optimal suggestions using an emotion recognition engine. This system consists of a server, a user's device (such as a television or smartphone), and an emotion recognition engine.

[1286] Server Roles

[1287] The server analyzes the received media content in real time, identifies objects displayed within the content, retrieves information about the identified objects from a database, generates information packets for providing this information to the user's display device, and receives emotion information transmitted from the user's device and adjusts the content of recommended products and services based on the information.

[1288] Device Role

[1289] The terminal receives the information packets sent from the server and displays them on a user interface. The display device, such as a television or smartphone, presents detailed information about products and services related to the content the user is viewing. The terminal also uses sensors such as a camera and microphone to capture the user's facial expressions and voice and transmits this information to the server as emotional information.

[1290] User Behavior

[1291] Users can check the product or service information displayed on their device and proceed with the purchase or reservation if they are interested. For example, a user who is interested in a dress featured in a drama can view detailed information and proceed with the purchase. In addition, an emotion recognition engine analyzes the user's emotions, and based on the results, the user can receive suggestions for more attractive products and services.

[1292] Hardware and software used

[1293] Hardware: Smartphone camera and microphone

[1294] Software: OpenCV (image processing technology), TensorFlow (machine learning model), EmotionEngine (emotion recognition engine)

[1295] Processing flow

[1296] The server first analyzes the received media content and performs object recognition. Next, it retrieves detailed information about the object from a database, creates a packet containing that information, and sends it to the device. The device receives the packet and displays the information on its screen. At the same time, the device analyzes the user's facial expressions and voice using an emotion recognition engine and sends the emotional information to the server. The server then adjusts product and service recommendations based on the acquired emotional information and provides them to the user in real time.

[1297] Specific examples

[1298] When a user is watching a product introduction video on an online shopping site, their facial expressions are captured with their smartphone camera and analyzed by an emotion recognition engine. Information about the dress recognized in the video is displayed in real time, and if the user shows interest, related accessories and other items are also recommended. The following is an example of a prompt sentence to be input into the generative AI model.

[1299] Prompt Sentence Examples

[1300] Write a Python program that recognizes emotions from facial expressions and voices while a user is watching a video on a shopping site using a smartphone camera, and displays product information (e.g., dresses or furniture) identified in the video in real time. Considering the emotional signals, recommend products that are particularly appealing to the user if they are happy.

[1301] This system simultaneously improves the viewing and purchasing experience, enabling optimal service delivery based on user emotions.

[1302] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1303] Step 1:

[1304] Input: The media content the user is watching

[1305] Specific operation: The server receives media content in real time and performs content analysis.

[1306] Data processing and data calculation: Using object identification algorithms, we identify objects within the content using video recognition technology and machine learning models.

[1307] Output: List of identified objects

[1308] Step 2:

[1309] Input: List of identified objects

[1310] What happens next: The server looks up the database and retrieves detailed information about the identified object.

[1311] Data processing and data manipulation: Using database queries to extract detailed information about an object (e.g., product name, price, availability).

[1312] Output: An information packet containing detailed information

[1313] Step 3:

[1314] Input: Information packet containing detailed information

[1315] Specific operation: The server sends an information packet to the user's terminal.

[1316] Data processing and data calculation: Encoding information packets into the appropriate format and sending them over the network.

[1317] Output: Information packets received at the user's terminal

[1318] Step 4:

[1319] Input: Information packet received at the user's terminal

[1320] Specific operation: The terminal decodes the information packet and displays it on the user interface.

[1321] Data processing and data calculation: Displaying item information (images, text, links, etc.) on the screen to visualize the decoded information.

[1322] Output: Product or service information that users can visually confirm

[1323] Step 5:

[1324] Input: User's face image and voice

[1325] How it works: The device's built-in camera and microphone capture the user's facial expressions and voice in real time.

[1326] Data processing and data calculation: An emotion recognition engine is used to analyze the user's emotional information from the captured facial expressions and voice.

[1327] Output: Emotional information (e.g., joy, surprise, interest)

[1328] Step 6:

[1329] Input: Emotion information

[1330] Specific operation: The device transmits the acquired emotional information to the server.

[1331] Data processing and data calculation: Encode emotional information into an appropriate format and transmit it over the network.

[1332] Output: Emotion information received by the server

[1333] Step 7:

[1334] Input: Emotion information received by the server

[1335] Specific operation: The server adjusts the products and services it recommends to the user based on the emotional information.

[1336] Data processing and calculation: Combining emotional information and object information, we execute algorithms to select optimal products and services.

[1337] Output: Adjusted recommendations for the user

[1338] Step 8:

[1339] Input: Adjusted proposal information

[1340] Specific operation: The server sends the adjusted proposal information back to the device and provides it to the user.

[1341] Data processing and data calculation: The proposed information is encoded as a packet and sent over the network.

[1342] Output: Adjusted proposal information received on the user's device

[1343] Step 9:

[1344] Input: Adjusted proposal information

[1345] Specific operation: The device displays information about suggested products and services tailored to the user interface.

[1346] Data processing and calculation: The proposed information after adjustment is visualized and displayed for the user to check.

[1347] Output: Information about the adjusted products and services that can be visually confirmed by the user

[1348] Step 10:

[1349] Input: Information about the adjusted proposed products and services

[1350] Specific action: The user selects the suggested product or service and completes the purchase or reservation process.

[1351] Data processing and calculation: Enter the necessary information for the purchase or reservation (shipping address and payment information) and finalize the transaction.

[1352] Output: Confirmation of completed purchase or reservation

[1353] By executing the above steps in order, the present invention enables the user to obtain information on products and services related to the content the user is viewing in real time and receive optimal suggestions based on emotions.

[1354] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1355] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1356] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1357] [Fourth embodiment]

[1358] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1359] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1360] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1361] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1362] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1363] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1364] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1365] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1366] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1369] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1370] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1371] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve. The system is primarily composed of a server and user terminals (televisions or smart devices).

[1372] Server Roles

[1373] Content Analysis

[1374] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[1375] Database Reference

[1376] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[1377] Information Packet Generation

[1378] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[1379] Information transmission

[1380] The server transmits the generated information packet to the user's terminal.

[1381] Device Role

[1382] Information reception and display

[1383] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[1384] Supporting User Input

[1385] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[1386] User Behavior

[1387] Reviewing and selecting information

[1388] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[1389] Purchase or reservation process

[1390] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[1391] Specific examples

[1392] Buying a dress while watching a drama

[1393] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[1394] Make restaurant reservations while watching a gourmet program

[1395] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[1396] In this way, the system helps users easily obtain detailed information about products and services and make purchases or reservations while watching TV, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[1397] The processing flow will be explained below.

[1398] Step 1:

[1399] The server receives real-time media content data for viewing by the user, which includes video data such as television programs and streaming videos.

[1400] Step 2:

[1401] The server analyzes the received video data and identifies objects displayed within the content, using video recognition technology and machine learning algorithms.

[1402] Step 3:

[1403] The server extracts metadata about the identified objects, including characteristics such as the object's type, color, shape, brand logo, etc.

[1404] Step 4:

[1405] The server uses the extracted metadata to query a product information database, which contains detailed information about various products and services.

[1406] Step 5:

[1407] The server checks the product or service information retrieved from the database and retrieves the necessary details, such as product name, manufacturer, price, and availability information.

[1408] Step 6:

[1409] The server generates an information packet based on the information obtained, which includes product images, details, prices, and purchase links.

[1410] Step 7:

[1411] The server transmits the generated information packet to the user's terminal.

[1412] Step 8:

[1413] The terminal receives the information packets sent from the server.

[1414] Step 9:

[1415] The device analyzes the contents of the information packet and updates the user interface (UI), displaying detailed product or service information in the corner of the screen.

[1416] Step 10:

[1417] The user reviews the displayed information and selects the product or service they are interested in. For example, they use a remote control to select a particular product and view detailed information.

[1418] Step 11:

[1419] The device displays detailed information about the product or service selected by the user, including a detailed product description, reviews, and availability information.

[1420] Step 12:

[1421] The user confirms the displayed detailed information and performs the purchase or reservation procedure, for example, by entering credit card information and a delivery address.

[1422] Step 13:

[1423] The terminal transmits the purchase or reservation information entered by the user to the server.

[1424] Step 14:

[1425] The server processes the received purchase or reservation information, processes the payment through a payment gateway, and in the case of a reservation, connects to a reservation system to confirm and secure the reservation.

[1426] Step 15:

[1427] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[1428] Step 16:

[1429] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[1430] This series of steps allows users to smoothly obtain detailed information about products and services and make purchases or reservations while watching television.

[1431] Example 1

[1432] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1433] In the conventional television viewing experience, users must manually search for information on products and services related to the content they are watching, which disrupts the viewing experience. Furthermore, even if users are interested in a product or service, the process of actually purchasing or reserving it can be cumbersome, potentially discouraging many users from making a purchase. Therefore, there is a need for a system that provides real-time information on related products and services while watching television, enabling seamless purchasing and reserving procedures.

[1434] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1435] In this invention, the server includes means for analyzing video information received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a data store, and means for formatting the retrieved information to generate information packets, thereby enabling users to obtain information about products and services related to the content they are viewing in real time and to carry out seamless purchasing and reservation procedures.

[1436] "Real-time" means that data and information are processed immediately and results are provided without delay.

[1437] "Video information" refers to media content that includes video data and related information.

[1438] "Object" refers to any object displayed within the Content, including elements related to specific products or services.

[1439] A "data store" is a database or other persistent storage device for storing data and accessing it as needed.

[1440] An "information packet" is a collection of data that is sent to a user terminal after organizing and processing the acquired information.

[1441] A "user terminal" refers to an information processing device used by a user, including a television or smart device.

[1442] A "display means" is a device or interface for visually displaying information.

[1443] A "remote control device" is a device that allows a user to operate or control a system using a device that includes a remote control or a touch screen.

[1444] "Image recognition technology" is a technology that allows computer systems to identify specific objects or patterns from image data.

[1445] A "machine learning algorithm" is a computational method that allows a computer system to analyze data and learn from experience to improve its performance.

[1446] This invention is a system that analyzes video information received in real time, identifies objects displayed in the content, provides users with information related to those objects, and facilitates purchase and reservation procedures. The system is primarily composed of a server and a user terminal (such as a television or smart device). Specific embodiments are described below.

[1447] 1. Server Role

[1448] The server receives video information from TVs and smart devices in real time and analyzes it. It uses video recognition technology and machine learning algorithms (e.g., OpenCV and TensorFlow) to identify objects displayed within the content. Information about the identified objects is retrieved by submitting SQL queries to a relational database management system (e.g., MySQL).

[1449] The server generates an information packet based on the information obtained. This information packet contains product images, details, prices, purchase links, etc., formatted in JSON format. The information packet is then sent to the user's device using the HTTP protocol.

[1450] Examples:

[1451] The server analyzes a scene from a drama and identifies the dress worn by the actress.

[1452] The drama footage is analyzed frame by frame to identify the dress.

[1453] The identified dress information is retrieved from the data store and an information packet is generated that includes the brand name, price, and availability information.

[1454] The information packet is sent to the user terminal via an HTTP POST request.

[1455] Example prompt sentence:

[1456] "Please explain the steps to get details about the dress worn by the actress in the drama I'm watching and check out."

[1457] 2. Role of the terminal

[1458] The device receives the information packets sent by the server and displays them in a user interface (UI), for example, product information displayed as a popup in the corner of a TV screen. The device uses the React library to visually represent this information.

[1459] Additionally, the device allows users to interact with the displayed information using a remote control or touchscreen. Users can press buttons on the remote control to select specific products, view more information, or proceed with the purchase or reservation process.

[1460] Examples:

[1461] When a user uses the remote control to view details about a dress

[1462] The information packet is received and details about the dress pop up in the corner of the TV screen.

[1463] The user presses a button on the remote control to go to the details page and proceed with the purchase procedure.

[1464] 3. User Behavior

[1465] The user checks the displayed information and selects the product or service they are interested in. For the selected product or service, the user then proceeds to purchase or reserve it. For example, when purchasing a dress, the user enters shipping information and credit card information to complete the payment. When making a reservation, the user enters the date and number of people to confirm the reservation.

[1466] Examples:

[1467] When a user purchases a dress that appears in a drama

[1468] The user selects detailed information about the dress from the displayed pop-up using the remote control, and is taken to a purchase details page.

[1469] Enter the required shipping and payment information and click the "Purchase" button to confirm your purchase.

[1470] This system allows users to easily obtain information about products and services related to the content they are viewing, and allows them to seamlessly complete purchase and reservation procedures, seamlessly linking the viewing and purchasing experiences and significantly improving usability.

[1471] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1472] Step 1:

[1473] Receiving content in real time

[1474] Input: Video stream from your TV or smart device.

[1475] What it does: The server receives the video stream in real time from the specified channel URL using a high-speed network connection and streaming server software (e.g., Wowza Streaming Engine).

[1476] Data processing: The received video stream is temporarily saved as a local file.

[1477] Output: Locally saved video stream file.

[1478] Step 2:

[1479] Content Analysis and Object Identification

[1480] Input: Locally stored video stream file.

[1481] How it works: The server analyzes the received video frame by frame and uses video recognition software (e.g., OpenCV) and machine learning algorithms (e.g., TensorFlow) to identify objects within the content.

[1482] Data processing: Each frame is analyzed sequentially and identified objects are labeled.

[1483] Output: A list of identified objects and their labels.

[1484] Step 3:

[1485] Retrieving information from a database

[1486] Input: A list of identified objects and their labels.

[1487] Specific operation: The server issues an SQL query to a relational database management system (e.g., MySQL) to retrieve information related to the object.

[1488] Data processing: Send a query to the database using the object label as a key to retrieve the corresponding data.

[1489] Output: Detailed data such as product information, manufacturer information, price information, and stock information retrieved from the database.

[1490] Step 4:

[1491] Generate information packets

[1492] Input: Detailed data retrieved from the database.

[1493] What happens: The server formats the information it receives and creates packets of information to serve to the user. This process uses web server software (e.g., Apache) and a scripting language (e.g., Python).

[1494] Data processing: Formatting the data and generating information packets in JSON format.

[1495] Output: The generated information packet in JSON format.

[1496] Step 5:

[1497] Sending information packets

[1498] Input: A packet of information in JSON format.

[1499] Specific operation: The server sends an information packet to the user's terminal using the HTTP protocol.

[1500] Data processing: Create an HTTP POST request and send the information packet to the user terminal.

[1501] Output: Information packets sent to the user terminal.

[1502] Step 6:

[1503] Receiving and displaying information on the user terminal

[1504] Input: A JSON formatted information packet sent by the server.

[1505] Specific operation: The device receives and analyzes the information packet. Then, it displays the analyzed information on the user interface (UI). The React library is used for display.

[1506] Data processing: Analyzes the received JSON data and passes the necessary information to the UI components for display.

[1507] Output: Product information displayed as a pop-up in the corner of the TV screen.

[1508] Step 7:

[1509] User-input operation

[1510] Input: Product information displayed on the user interface, and user input operations.

[1511] Specific behavior: The device accepts user input via a remote control or touchscreen, allowing the user to manipulate the displayed information. For example, the user presses a button on the remote control to select a particular product and view its details.

[1512] Data manipulation: Accepts user input and triggers relevant actions.

[1513] Output: A screen showing detailed product information.

[1514] Step 8:

[1515] Completing a purchase or reservation

[1516] Input: Shipping, payment, and reservation information entered by the user.

[1517] Specific operation: The terminal sends the information entered in the form to the server and calls an API to process the payment or reservation.

[1518] Data processing: The entered information is sent to the server as an HTTP POST request to process payments and reservations.

[1519] Output: A confirmation message that the purchase or reservation process is complete.

[1520] (Application example 1)

[1521] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1522] Conventional systems that provide information about products and services while watching television lack the functionality to instantly check, purchase, or reserve products of interest to users. Furthermore, technology to identify specific objects within a video frame and provide related information in real time has not yet been fully developed. Therefore, improving usability and seamlessly integrating the viewing and purchasing experiences have been key challenges.

[1523] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1524] In this invention, the server includes means for analyzing the media content received in real time and using a deep learning model to identify a specific object in a video frame, means for retrieving information about the identified object from a database, and means for displaying the retrieved information on a user's display device, thereby providing product information or service information related to the specific object in the media content in real time, allowing the user to make an immediate purchase or reservation.

[1525] In addition, if the display device is a smart device, it further includes means for providing an interface for the user to make selections and inputs by voice or touch operation, thereby enabling the user to easily check product information, purchase, and reserve using voice commands or touch operation, improving convenience.

[1526] "Media content" refers to information in digital form that includes video and audio, and refers to viewable content such as television programs, movies, and videos.

[1527] A "video frame" is one of the still images that make up a digital video, and when displayed in succession, it is perceived as a moving image.

[1528] "Specific objects" refers to specific items or people that appear in the video frame, including clothing, furniture, accessories, etc.

[1529] A "deep learning model" is a type of artificial intelligence technology, a computational model that uses large amounts of data and advanced algorithms to recognize objects and patterns.

[1530] A "database" is a system that systematically organizes and stores specific information in a searchable manner, and stores product information, service information, etc.

[1531] A "display device" is a device for displaying information in a form that can be viewed by a user, and includes smartphones, tablets, televisions, and the like.

[1532] An "interface" is the means by which a user interacts with a system, including remote controls, touchscreens, voice recognition, etc.

[1533] "Usability" refers to the characteristics that make a system or service easy for users to use and understand, and includes efficiency, effectiveness, and satisfaction of use.

[1534] The details of the embodiments of the present invention are described below.

[1535] The server analyzes the received media content in real time and uses a deep learning model to identify specific objects contained within the video frames. For example, the server targets the clothes worn by actors in a movie scene or the furniture used in the scene. Publicly available deep learning models (e.g., YOLO, TensorFlow) can be used for this identification. Information about the identified objects is obtained by accessing a database on the server. This database stores detailed information about products and services (such as names, prices, availability, and image URLs).

[1536] The terminal displays the acquired information on the user's display device. If the display device is a smart device (such as a smartphone, tablet, or television), it provides an interface that allows the user to make selections and inputs using voice or touch operations. This allows the user to check product information related to a specific object and make direct purchases or reservations.

[1537] For example, if a user is interested in a bag that appears in a scene while watching a drama, the server can identify the bag, obtain related details, and send them to the display device. The user can then confirm the bag information by voice or touch operation and proceed with the purchase process.

[1538] The entire system is built using commercially available hardware and software, including high-performance servers, smart devices, deep learning models (YOLO, TensorFlow), and communication interfaces (API, HTTP / HTTPS).

[1539] An example prompt for a generative AI model is:

[1540] "Your goal is to create an AI model that can identify specific objects in a video frame and retrieve related product information. Specifically, it should focus on objects being held or items being worn, and provide that information from a database."

[1541] This allows users to enjoy a seamless viewing and purchasing experience.

[1542] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1543] Step 1:

[1544] The server receives and analyzes video frames of media content received in real time. Specifically, the server receives streaming data, extracts video frames from the data, and starts processing. The input data used in this process is the received media content, and the output is a list of analyzed video frames.

[1545] Step 2:

[1546] The server uses a deep learning model (e.g., YOLO, TensorFlow) to identify specific objects contained within video frames. The input is the list of video frames obtained in step 1, and the model detects objects within the frames and identifies their type and location. The output is a list of identified objects and their locations. Specifically, each frame is input into the deep learning model, and the object detection results are obtained.

[1547] Step 3:

[1548] The server retrieves detailed information about the identified objects from the database. The input is a list of object information identified in step 2, and searches and retrieves detailed information about the corresponding objects (e.g., product name, price, stock status, image URL, etc.) from the database. The output is a list of detailed information corresponding to each object. Specifically, the server executes a query to the database using the object name as a key, and compiles the results as a list.

[1549] Step 4:

[1550] The server generates an information packet based on the acquired detailed information and sends it to the terminal. The input is the list of detailed information obtained in step 3, and the output is an information packet. Specifically, the server converts the detailed information of each object into a packet format and generates data to be sent to the terminal.

[1551] Step 5:

[1552] The terminal displays the information packets received from the server on a display device. The input is the information packets sent from the server, and the output is product or service information displayed on the user's display device. Specific operations include analyzing the information packets and displaying product images and detailed information on the screen.

[1553] Step 6:

[1554] The user selects a specific product or service based on the displayed information and completes the purchase or reservation process. The input is the detailed product or service information displayed on the device, and the output is the user's selection and the input information (e.g., delivery address, payment information, reservation date, etc.). Specific operations involve the user selecting information using touch operations or voice commands and entering the necessary information.

[1555] Step 7:

[1556] The terminal sends the user's selection and input information to the server and proceeds with the purchase or reservation procedure. The input is the user's selection and input information, and the output is request data to the server. Specifically, the information entered by the user is packed into packets and sent to the server.

[1557] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1558] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[1559] Server Roles

[1560] Content Analysis

[1561] The server analyzes the media content received in real time and identifies objects that appear within the content, such as the dress worn by an actress in a TV drama scene or the furniture placed within the scene, using video recognition technology and machine learning algorithms.

[1562] Database Reference

[1563] Based on the identified object, the server retrieves relevant information from the database, including product details, manufacturer, price, availability information, etc. For example, the brand name and price information of the identified dress may be retrieved from the database.

[1564] Information Packet Generation

[1565] Based on the information obtained, the server generates an information packet to provide to the user, which includes product images, details, prices, purchase links, etc.

[1566] Emotional information reception

[1567] The server receives emotion information sent from the user's device, including the user's facial expression and tone of voice.

[1568] Information transmission

[1569] The server then sends the generated information packet to the user's device. It is also possible to adjust the product recommendations based on the user's emotional information.

[1570] Device Role

[1571] Information reception and display

[1572] The device receives the information packet sent from the server and displays it on the user interface (UI). For example, product information may be displayed as a pop-up in the corner of a television screen. This display allows users to easily view detailed information about products and services related to the content they are currently watching.

[1573] Emotional information gathering

[1574] The device is equipped with sensors to collect emotional information such as the user's facial expressions and tone of voice, allowing it to recognize the user's emotions such as joy, surprise, and interest in real time.

[1575] Emotional information transmission

[1576] The device transmits the collected emotion information to the server.

[1577] Supporting User Input

[1578] The terminals are interactive and allow users to manipulate the displayed information using a remote control or touchscreen, for example, by pressing a button on the remote control to select a particular product, view more information, or proceed with a purchase or reservation.

[1579] User Behavior

[1580] Reviewing and selecting information

[1581] The user checks the displayed information and selects a product or service that they are interested in. For example, if a user is interested in a dress that appears in a drama, they press a button on the remote control to view detailed information about it.

[1582] Purchase or reservation process

[1583] The user then completes the purchase or reservation process for the selected product or service. For example, if purchasing a dress, the user enters shipping information and credit card information to complete the payment. If making a reservation, the user enters the date and number of people and confirms the reservation.

[1584] Specific examples

[1585] Buying a dress while watching a drama

[1586] The server analyzes the drama scene and identifies the dress worn by the actress. It retrieves the identified dress information from the database, creates an information packet, and sends it to the terminal. The terminal displays detailed information about the dress in the corner of the TV screen. The user uses the remote control to view the detailed information about the dress and proceeds to purchase it. The user enters shipping and payment information to confirm the purchase.

[1587] Make restaurant reservations while watching a gourmet program

[1588] The server analyzes scenes from a gourmet program and identifies the restaurants featured. It retrieves information about the identified restaurants from a database, creates an information packet, and sends it to the terminal. The terminal displays detailed restaurant information on the TV screen. The user uses the remote control to view the restaurant information and proceed with the reservation. The user enters the reservation date and number of people to confirm the reservation.

[1589] Use of emotional information

[1590] If a user smiles while watching a drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, the server can present accessories and other recommended products related to the dress the user smiled at.

[1591] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[1592] The processing flow will be explained below.

[1593] Step 1:

[1594] The server receives real-time media content data that the user is viewing, including video data to be analyzed.

[1595] Step 2:

[1596] The server analyzes the received video data using machine learning algorithms to identify objects (e.g., clothes, bags, interior decor) displayed within the content.

[1597] Step 3:

[1598] The server extracts metadata about the identified objects (e.g., type, color, shape, brand logo).

[1599] Step 4:

[1600] The server queries the product information database based on the extracted metadata to obtain relevant product information (e.g., product details, manufacturer, price, and stock information).

[1601] Step 5:

[1602] The server checks the product information retrieved from the database and generates an information packet containing the necessary details (e.g., product name, price, image, and purchase link).

[1603] Step 6:

[1604] The server transmits the generated information packet to the user's terminal.

[1605] Step 7:

[1606] The terminal receives the information packets sent from the server.

[1607] Step 8:

[1608] The terminal analyzes the contents of the information packet and updates the user interface, displaying detailed product or service information in the corner of the TV screen.

[1609] Step 9:

[1610] The device uses built-in sensors to collect the user's emotional information (e.g., facial expressions, tone of voice) in real time.

[1611] Step 10:

[1612] The terminal transmits the collected emotion information to the server.

[1613] Step 11:

[1614] The server analyzes the received emotion information and recognizes the user's emotions such as joy, surprise, and interest.

[1615] Step 12:

[1616] The server then tailors the displayed product and service recommendations based on the emotion engine, for example, by presenting options and offers related to the products the user has expressed interest in.

[1617] Step 13:

[1618] The terminal displays the adjusted information again on the user interface.

[1619] Step 14:

[1620] The user uses a remote control or touch screen to review the displayed information and select products or services of interest.

[1621] Step 15:

[1622] The device displays detailed information about the product or service the user selects, including a detailed description, reviews, and availability information.

[1623] Step 16:

[1624] The user decides to purchase or reserve and initiates the process using a remote control or touchscreen.

[1625] Step 17:

[1626] The terminal sends the purchase information (e.g., delivery address, credit card information) or reservation information (e.g., date, number of people) entered by the user to the server.

[1627] Step 18:

[1628] The server processes the payment through the payment gateway based on the received purchase information, and in the case of a reservation, connects to the corresponding reservation system to confirm and secure the reservation.

[1629] Step 19:

[1630] The server sends information that the payment process or reservation confirmation has been completed to the user's terminal.

[1631] Step 20:

[1632] The terminal receives confirmation from the server and displays it on the user interface, allowing the user to confirm that the transaction has been completed.

[1633] This series of steps allows users to easily obtain detailed information about products and services while watching TV, receive optimal suggestions based on their emotions, and smoothly make purchases or reservations.

[1634] Example 2

[1635] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1636] In recent years, the number of users watching content on televisions and smart devices has increased, leading to a growing demand for easy access to information on related products and services while watching, and for making purchases or reservations. However, conventional systems require users to search for information based on their interests, and are unable to provide optimal suggestions based on the user's emotions. This results in a lack of integration between the viewing and purchasing experiences, resulting in a lack of user convenience.

[1637] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for acquiring information about the identified objects from a database, means for displaying the acquired information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for collecting user emotion information, means for transmitting the collected emotion information to the server, and means for adjusting proposal content based on the emotion information. This not only makes it possible to easily acquire information about related products and services while viewing, but also makes it possible to make optimal proposals based on the user's emotions.

[1638] "Real-time" refers to the immediate processing of data and events without delay.

[1639] "Media content" is a general term for digital content that includes information such as audio, video, images, and text.

[1640] "Object" refers to a distinct item or element that appears within media content.

[1641] "Identification" refers to identifying what an object is.

[1642] A "database" refers to a collection of data that stores specific information and allows for rapid search and retrieval.

[1643] "Retrieval" refers to the act of retrieving the necessary information from a database.

[1644] "Display device" refers to a screen or display on which a user views information.

[1645] "Purchase" refers to the act of a user buying a product.

[1646] "Reservation" refers to the act of a user applying for a service in advance.

[1647] "Emotional information" refers to data about emotions obtained from a user's facial expressions, tone of voice, behavior, etc.

[1648] "Collection" refers to the act of obtaining emotional information using devices or sensors.

[1649] "Proposal content" refers to information about products and services offered to users.

[1650] "Adjustment" refers to the act of optimizing the suggestions based on the user's emotional state.

[1651] This invention relates to a system that provides real-time information on products and services related to content displayed while watching television, allowing users to easily purchase or reserve products and services. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest optimal products and services according to the user's emotions. The system is primarily composed of a server, a user's terminal (television or smart device), and the emotion engine.

[1652] The server analyzes the received media content in real time using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16). Specifically, it identifies objects such as the dress worn by an actress in a drama scene or furniture placed in the scene. This allows the user to obtain real-time information about the content they are watching.

[1653] Next, the server retrieves information about the identified object from a database. This data includes product details, manufacturer, price, and availability information. For example, for an identified dress, the server searches the database for the brand name and price information. At this stage, the server generates an information packet based on the retrieved information. The information packet includes a product image, details, price, and a purchase link.

[1654] The device receives the information packet sent from the server and displays it on a user interface (UI). For example, product information can be displayed as a pop-up in the corner of a TV screen. This allows users to easily view detailed information about products and services related to the content they are currently watching.

[1655] The device also uses emotion recognition software (e.g., FaceAPI, EmotionSDK) that collects emotional information such as the user's facial expressions and tone of voice to recognize the user's emotions, such as joy, surprise, and interest, in real time. This allows the device to transmit the collected emotional information to a server. The server can then adjust the product and service recommendations it provides to the user based on the collected emotional information.

[1656] For example, if a user smiles while watching a TV drama, the emotion engine recognizes the user's joy and sends that information to the server. Based on this emotion information, the server can suggest new items or products with special offers to the user. For example, it can present accessories and other recommended products related to the dress the user smiled at.

[1657] Below are some examples of prompt sentences:

[1658] Example prompt sentence:

[1659] A prompt that displays details about a dress worn by an actress in a scene from a drama, along with a link to purchase it.

[1660] "Please provide the brand name, price, and purchase link of the dress worn by the actress in the scene of the media content you are currently watching."

[1661] A prompt that provides information about a restaurant featured during a foodie show and a reservation link.

[1662] "View detailed information and reservation links for featured restaurants."

[1663] A prompt that recognizes joy in the user's facial expression and suggests specific products based on that emotion.

[1664] "Suggest accessories and other product recommendations related to the specific product that made the user smile."

[1665] This system allows users to easily obtain detailed information about products and services and make purchases or reservations while watching TV, as well as receive optimal suggestions based on their emotions, thereby enriching their viewing and purchasing experiences and providing personalized services.

[1666] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1667] Step 1: Content Analysis

[1668] The server analyzes the incoming media content in real time, using video recognition technology and machine learning algorithms (e.g., YoLo, VGG16) to perform frame-by-frame analysis. The input is the video data, and the machine learning model labels the objects in the frames. The output is a list of identified objects.

[1669] Specific behavior:

[1670] The server captures the video data and sends it to the analysis engine.

[1671] A machine learning model labels and identifies objects in each frame.

[1672] Compile information about the identified objects into a list.

[1673] Step 2: Database Reference

[1674] The server uses the list of identified objects to retrieve relevant information from a database. The input is the object's label information, and it generates and executes a database query. The output is product details, manufacturer, price, and availability information.

[1675] Specific behavior:

[1676] The server generates an SQL query based on the identification information.

[1677] A query is sent to the database to retrieve the relevant product information.

[1678] The acquired information is organized and packetized.

[1679] Step 3: Information Packet Generation

[1680] The server generates an information packet to provide to the user based on the retrieved product information. The input is the product information retrieved from the database, and the output is an information packet (JSON format) sent to the user.

[1681] Specific behavior:

[1682] Product information is converted into JSON format and packetized.

[1683] Generate an information packet containing product images, details, prices, and purchase links.

[1684] Step 4: Emotional information gathering

[1685] The device collects emotional information such as the user's facial expressions and tone of voice using emotion recognition software (e.g., FaceAPI, EmotionSDK). The input is sensor data from the camera and microphone, and the output is preprocessed emotional data.

[1686] Specific behavior:

[1687] The device will initialize the camera and microphone and start emotion recognition software.

[1688] Monitors the user's facial expressions and voice in real time.

[1689] The collected data is analyzed to identify emotional states.

[1690] Step 5: Emotional information transmission

[1691] The device sends the collected emotion information to the server. The input is the preprocessed emotion data, and the output is the emotion data sent to the server.

[1692] Specific behavior:

[1693] Convert the emotion data into a suitable format.

[1694] Send emotion data to the server's API endpoint using an HTTP POST request.

[1695] Step 6: Receiving and displaying information

[1696] The terminal receives the information packet sent from the server and displays it on the user interface (UI). The input is the information packet from the server, and the output is the product information displayed on the UI.

[1697] Specific behavior:

[1698] Receives an HTTP response from the server.

[1699] Parse the information packet and update the UI components.

[1700] A pop-up is displayed at an appropriate position on the TV screen to present product information.

[1701] Step 7: Supporting User Input

[1702] The terminal allows the user to operate the displayed information using a remote control or touch screen. The input is an operation signal from the user, and the output is a transition to a purchase screen or detailed information on the selected product.

[1703] Specific behavior:

[1704] It detects input from the remote control and processes the user's selection operations.

[1705] Detailed information about the selected product will be displayed, and you will be taken to the purchase or reservation screen.

[1706] Step 8: Purchase or reservation process

[1707] The user then completes the purchase or reservation procedure for the selected product or service. The input is the user's shipping information and payment information, and the output is a confirmation of the purchase or reservation.

[1708] Specific behavior:

[1709] A user input form is displayed on the device's UI, prompting the user to enter shipping address and payment information.

[1710] The input information is sent to the server and the purchase or reservation procedure is carried out.

[1711] Step 9: Use emotional information

[1712] The server adjusts the product and service recommendations provided to the user based on the emotion information. The input is the collected emotion information, and the output is the adjusted product recommendation information.

[1713] Specific behavior:

[1714] The server analyzes the emotional data and generates a list of recommended products based on pleasure and interest.

[1715] The recommended product list is included in an information packet and sent to the terminal, which then displays it on the UI.

[1716] (Application example 2)

[1717] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1718] With conventional systems, it was difficult for users to obtain real-time information about products and services displayed while watching television and make purchases or reservations on the spot.In addition, the system did not realize optimal product and service recommendations based on user emotions, making it difficult to improve the user experience.

[1719] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1720] In this invention, the server includes means for analyzing media content received in real time and identifying objects displayed in the content, means for retrieving information about the identified objects from a database, means for displaying the retrieved information on the user's display device, means for carrying out procedures for purchasing or reserving products or services selected by the user based on the displayed information, means for analyzing and retrieving emotional information about the user using an emotion recognition engine, and means for adjusting the content provided to the user based on the retrieved emotional information. This allows the user to obtain information about products and services related to the content they are viewing in real time, making it easy to purchase or reserve, and also allowing them to receive optimal product recommendations based on their emotions.

[1721] "Real-time" refers to information being processed immediately at the moment it is generated.

[1722] "Media content" refers to a collection of visual and audio information such as television programs, videos, and images.

[1723] "Object" refers to a specific item that appears in the content, such as clothing, furniture, or other real-world objects.

[1724] "Database" refers to a centralized information management system for systematically storing and managing detailed information about products and services.

[1725] A "display device" refers to a device that allows a user to visually check information, such as a television or smartphone.

[1726] "Purchase or reservation process" means the series of steps required to purchase a product or service selected by a User or the process for confirming a reservation.

[1727] An "emotion recognition engine" refers to a software system that analyzes and recognizes emotions from a user's facial expressions and voice.

[1728] "Operation device" refers to an interactive device that allows a user to select or input information, such as a remote control or touch screen.

[1729] "Adjusting the content of information provided" refers to changing the content of information presented to the user based on the acquired emotion information.

[1730] This system provides real-time information about products and services related to objects displayed in media content that a user is viewing, and makes optimal suggestions using an emotion recognition engine. This system consists of a server, a user's device (such as a television or smartphone), and an emotion recognition engine.

[1731] Server Roles

[1732] The server analyzes the received media content in real time, identifies objects displayed within the content, retrieves information about the identified objects from a database, generates information packets for providing this information to the user's display device, and receives emotion information transmitted from the user's device and adjusts the content of recommended products and services based on the information.

[1733] Device Role

[1734] The terminal receives the information packets sent from the server and displays them on a user interface. The display device, such as a television or smartphone, presents detailed information about products and services related to the content the user is viewing. The terminal also uses sensors such as a camera and microphone to capture the user's facial expressions and voice and transmits this information to the server as emotional information.

[1735] User Behavior

[1736] Users can check the product or service information displayed on their device and proceed with the purchase or reservation if they are interested. For example, a user who is interested in a dress featured in a drama can view detailed information and proceed with the purchase. In addition, an emotion recognition engine analyzes the user's emotions, and based on the results, the user can receive suggestions for more attractive products and services.

[1737] Hardware and software used

[1738] Hardware: Smartphone camera and microphone

[1739] Software: OpenCV (image processing technology), TensorFlow (machine learning model), EmotionEngine (emotion recognition engine)

[1740] Processing flow

[1741] The server first analyzes the received media content and performs object recognition. Next, it retrieves detailed information about the object from a database, creates a packet containing that information, and sends it to the device. The device receives the packet and displays the information on its screen. At the same time, the device analyzes the user's facial expressions and voice using an emotion recognition engine and sends the emotional information to the server. The server then adjusts product and service recommendations based on the acquired emotional information and provides them to the user in real time.

[1742] Specific examples

[1743] When a user is watching a product introduction video on an online shopping site, their facial expressions are captured with their smartphone camera and analyzed by an emotion recognition engine. Information about the dress recognized in the video is displayed in real time, and if the user shows interest, related accessories and other items are also recommended. The following is an example of a prompt sentence to be input into the generative AI model.

[1744] Prompt Sentence Examples

[1745] Write a Python program that recognizes emotions from facial expressions and voices while a user is watching a video on a shopping site using a smartphone camera, and displays product information (e.g., dresses or furniture) identified in the video in real time. Considering the emotional signals, recommend products that are particularly appealing to the user if they are happy.

[1746] This system simultaneously improves the viewing and purchasing experience, enabling optimal service delivery based on user emotions.

[1747] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1748] Step 1:

[1749] Input: The media content the user is watching

[1750] Specific operation: The server receives media content in real time and performs content analysis.

[1751] Data processing and data calculation: Using object identification algorithms, we identify objects within the content using video recognition technology and machine learning models.

[1752] Output: List of identified objects

[1753] Step 2:

[1754] Input: List of identified objects

[1755] What happens next: The server looks up the database and retrieves detailed information about the identified object.

[1756] Data processing and data manipulation: Using database queries to extract detailed information about an object (e.g., product name, price, availability).

[1757] Output: An information packet containing detailed information

[1758] Step 3:

[1759] Input: Information packet containing detailed information

[1760] Specific operation: The server sends an information packet to the user's terminal.

[1761] Data processing and data calculation: Encoding information packets into the appropriate format and sending them over the network.

[1762] Output: Information packets received at the user's terminal

[1763] Step 4:

[1764] Input: Information packet received at the user's terminal

[1765] Specific operation: The terminal decodes the information packet and displays it on the user interface.

[1766] Data processing and data calculation: Displaying item information (images, text, links, etc.) on the screen to visualize the decoded information.

[1767] Output: Product or service information that users can visually confirm

[1768] Step 5:

[1769] Input: User's face image and voice

[1770] How it works: The device's built-in camera and microphone capture the user's facial expressions and voice in real time.

[1771] Data processing and data calculation: An emotion recognition engine is used to analyze the user's emotional information from the captured facial expressions and voice.

[1772] Output: Emotional information (e.g., joy, surprise, interest)

[1773] Step 6:

[1774] Input: Emotion information

[1775] Specific operation: The device transmits the acquired emotional information to the server.

[1776] Data processing and data calculation: Encode emotional information into an appropriate format and transmit it over the network.

[1777] Output: Emotion information received by the server

[1778] Step 7:

[1779] Input: Emotion information received by the server

[1780] Specific operation: The server adjusts the products and services it recommends to the user based on the emotional information.

[1781] Data processing and calculation: Combining emotional information and object information, we execute algorithms to select optimal products and services.

[1782] Output: Adjusted recommendations for the user

[1783] Step 8:

[1784] Input: Adjusted proposal information

[1785] Specific operation: The server sends the adjusted proposal information back to the device and provides it to the user.

[1786] Data processing and data calculation: The proposed information is encoded as a packet and sent over the network.

[1787] Output: Adjusted proposal information received on the user's device

[1788] Step 9:

[1789] Input: Adjusted proposal information

[1790] Specific operation: The device displays information about suggested products and services tailored to the user interface.

[1791] Data processing and calculation: The proposed information after adjustment is visualized and displayed for the user to check.

[1792] Output: Information about the adjusted products and services that can be visually confirmed by the user

[1793] Step 10:

[1794] Input: Information about the adjusted proposed products and services

[1795] Specific action: The user selects the suggested product or service and completes the purchase or reservation process.

[1796] Data processing and calculation: Enter the necessary information for the purchase or reservation (shipping address and payment information) and finalize the transaction.

[1797] Output: Confirmation of completed purchase or reservation

[1798] By executing the above steps in order, the present invention enables the user to obtain information on products and services related to the content the user is viewing in real time and receive optimal suggestions based on emotions.

[1799] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1800] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1801] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1802] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1803] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1804] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1805] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1806] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1807] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1808] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1809] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1810] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1811] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1812] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1813] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1814] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1815] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1816] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1817] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1818] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1819] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1820] The following is further disclosed regarding the above embodiment.

[1821] (Claim 1)

[1822] means for analyzing the received media content in real time and identifying objects displayed within the content;

[1823] means for retrieving information about the identified object from the database;

[1824] means for displaying the acquired information on a user's display device;

[1825] A means for carrying out the purchase or reservation process for the product or service selected by the user based on the displayed information;

[1826] A system including:

[1827] (Claim 2)

[1828] 2. The system according to claim 1, further comprising means for, when an object displayed in the content includes specific identification information, acquiring detailed information from a database based on the identification information.

[1829] (Claim 3)

[1830] 10. The system of claim 1, wherein the display device is a television, and further comprising means for providing an interface for a user to make selections and inputs using a remote control.

[1831] "Example 1"

[1832] (Claim 1)

[1833] means for analyzing video information received in real time and identifying objects displayed within the content;

[1834] means for retrieving information about the identified object from a data store;

[1835] means for displaying the acquired information on a user's display means;

[1836] A means for carrying out the purchase or reservation process for the product or service selected by the user based on the displayed information;

[1837] means for using image recognition techniques and machine learning algorithms for object identification;

[1838] means for formatting the acquired information to generate an information packet;

[1839] means for transmitting information packets to a user's terminal;

[1840] A system including:

[1841] (Claim 2)

[1842] 2. The system of claim 1, further comprising: means for, when an object displayed in the content includes specific identification information, obtaining detailed information from a data store based on the identification information.

[1843] (Claim 3)

[1844] 10. The system of claim 1, wherein the display means is a television, and further comprising means for providing an interface for a user to make selections and inputs using a remote control device.

[1845] "Application Example 1"

[1846] Extracting new parts

[1847] Novel technical features from application examples include:

[1848] 1. It uses deep learning models to detect specific objects within video frames.

[1849] 2. Retrieving product information related to the detected object from a database.

[1850] 3. It has an interface that allows users to manipulate product information (including voice recognition functionality).

[1851] 4. The application is intended to be run on smart devices (smartphones and tablets).

[1852] (Claim 1)

[1853] means for analyzing received media content in real time and using a deep learning model to identify specific objects within video frames;

[1854] means for retrieving information about the identified object from a database;

[1855] means for displaying the acquired information on a user's display device;

[1856] A means for carrying out the purchase or reservation process for the product or service selected by the user based on the displayed information;

[1857] A system including:

[1858] (Claim 2)

[1859] 2. The system according to claim 1, further comprising means for, when an object displayed in the content includes specific identification information, acquiring detailed information from a database based on the identification information.

[1860] (Claim 3)

[1861] 10. The system of claim 1, wherein the display device is a smart device and further comprises means for providing an interface for a user to make selections and inputs using voice or touch operations.

[1862] "Example 2: Combining Emotion Engines"

[1863] (Claim 1)

[1864] means for analyzing the received media content in real time and identifying objects displayed within the content;

[1865] means for retrieving information about the identified object from the database;

[1866] means for displaying the acquired information on a user's display device;

[1867] A means for carrying out the purchase or reservation process for the product or service selected by the user based on the displayed information;

[1868] A means for collecting user emotion information;

[1869] means for transmitting the collected emotion information to a server;

[1870] a means for adjusting the content of the proposal based on the emotional information;

[1871] A system including:

[1872] (Claim 2)

[1873] 2. The system according to claim 1, further comprising means for, when an object displayed in the content includes specific identification information, acquiring detailed information from a database based on the identification information.

[1874] (Claim 3)

[1875] 10. The system of claim 1, wherein the display device is a television, and further comprising means for providing an interface for a user to make selections and inputs using a remote control.

[1876] "Application example 2 when combining emotion engines"

[1877] (Claim 1)

[1878] means for analyzing received media content in real time to identify objects appearing within the content;

[1879] means for retrieving information about the identified object from a database;

[1880] means for displaying the acquired information on a user's display device;

[1881] A means for carrying out the purchase or reservation process for the product or service selected by the user based on the displayed information;

[1882] A means for analyzing and acquiring user emotion information using an emotion recognition engine;

[1883] a means for adjusting the content of information provided to the user based on the acquired emotion information;

[1884] A system including:

[1885] (Claim 2)

[1886] 2. The system according to claim 1, further comprising means for, when an object displayed in the content includes specific identification information, acquiring detailed information from a database based on the identification information.

[1887] (Claim 3)

[1888] 10. The system of claim 1, wherein the display device is a viewing device, and further comprising means for providing an interface for a user to make selections and inputs using a control device. [Explanation of symbols]

[1889] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for analyzing the received media content in real time and identifying objects displayed within the content; means for retrieving information about the identified object from the database; means for displaying the acquired information on a user's display device; A means for carrying out the purchase or reservation process for the product or service selected by the user based on the displayed information; A system including:

2. 2. The system according to claim 1, further comprising means for, when an object displayed in the content includes specific identification information, acquiring detailed information from a database based on the identification information.

3. 10. The system of claim 1, wherein the display device is a television, and further comprising means for providing an interface for a user to make selections and inputs using a remote control.

Images