system

Abstract

We provide the system. [Solution] In order to identify viewers' content preferences, means of collecting viewing history information, To identify the points of viewer attention, a means of analyzing eye-tracking data is used, A means for generating personalized advertisements for viewers based on the aforementioned viewing history information and eye-tracking data, The means for delivering the personalized advertisement to the viewer's device, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional TV advertisement distribution system, uniform advertisements are distributed regardless of the individual interests and concerns of viewers, so there is a problem that the effect of the advertisements is limited and the expectations of advertisers cannot be met. In addition, it is difficult to grasp which advertisement elements the viewers are paying attention to, so it is difficult to maximize the effect of the advertisement creative. Therefore, there is a need to provide a personalized advertisement distribution method that maximizes the effect of advertisements and attracts the interest of viewers.

Means for Solving the Problems

[0005] This invention comprises means for collecting viewer viewing history information and means for analyzing eye-tracking data. This makes it possible to identify viewers' content preferences and points of focus. Furthermore, by providing means for generating and delivering personalized advertisements to viewers based on viewing history information and eye-tracking data, it is possible to deliver advertisements that capture viewers' attention. In addition, by introducing means for reacquiring eye-tracking data to measure viewers' reactions during advertisement delivery, it is possible to maximize the effectiveness of advertisements by optimizing the timing, order, and frequency of advertisements.

[0006] "Viewing history information" refers to data that includes the types of content a viewer has watched in the past, the duration of viewing, and the frequency of viewing.

[0007] "Eye-tracking data" refers to information that indicates which part of the display a viewer is looking at, and is used to identify the points of focus.

[0008] "Personalized advertising" refers to advertising content that is optimized based on the individual interests and preferences of the viewer, and is customized to meet the specific needs and tastes of the viewer.

[0009] "Advertising time slots" refer to the time periods in which specific advertisements are delivered, and are selected to effectively reach the target audience.

[0010] "Advertising order" refers to the sequence in which multiple advertisements are delivered to a viewer, and it is set appropriately to maximize the effectiveness of the advertisements.

[0011] "Ad frequency" refers to how many times a particular ad is delivered to viewers within a certain period, and setting an appropriate frequency optimizes the effectiveness of the ad. [Brief explanation of the drawing]

[0012] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, a labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention relates to a system for delivering advertisements optimized for viewers, and primarily concerns the generation and delivery of personalized advertisements using viewing history information and eye-tracking data.

[0034] Data collection and analysis

[0035] terminal

[0036] The device records information about the program the viewer is watching (program name, start time, end time) in real time. It also acquires viewer eye-tracking data through its built-in camera and sensors, determining which part of the screen the viewer is looking at.

[0037] The device encrypts the collected viewing history data and eye-tracking data and securely transmits it to the server.

[0038] server

[0039] The server stores the received viewing history data and eye-tracking data in a database. Based on this data, it runs a machine learning model to analyze viewers' content preferences and interests.

[0040] The server identifies viewer behavior patterns by comparing them with past similar viewer data. This allows it to predict the types of content that will interest viewers.

[0041] Ad generation and delivery

[0042] server

[0043] The server generates ad creatives based on the viewer's preferences and areas of interest. The generated ads are customized so that the product images and messages are most relevant to the viewer.

[0044] The server calculates the optimal time, order, and frequency for delivering the generated ads and sends the schedule for ad delivery to the devices.

[0045] terminal

[0046] The device delivers personalized advertisements to viewers according to a specified schedule. During this process, eye-tracking data is collected again to record viewers' reactions to the advertisements.

[0047] User

[0048] If a user is interested in an ad they have seen, they can request additional information using a remote control or similar device. This request information is sent to the server as feedback and used to measure the effectiveness of the ad.

[0049] In this way, advertising effectiveness can be maximized by efficiently delivering personalized advertisements to viewers. For example, users who enjoy watching cooking shows can be shown advertisements for the latest cooking appliances at the appropriate time. These advertisements include visual designs and messages that are likely to attract viewers' attention, thereby increasing their interest.

[0050] The following describes the processing flow.

[0051] Step 1:

[0052] The device records information about the program the viewer is watching, including the program name, start time, and end time. Furthermore, the device uses a built-in camera and eye-tracking sensor to acquire data on the viewer's eye movements while they are watching.

[0053] Step 2:

[0054] The device encrypts the collected viewing history data and eye-tracking data and sends it to the server. The transmitted data is used for analysis while protecting the viewer's privacy.

[0055] Step 3:

[0056] The server stores the received viewing data in a database and builds viewer profiles. Using machine learning algorithms, it analyzes viewers' content preferences based on their viewing history and identifies their preferred trends.

[0057] Step 4:

[0058] The server identifies points that viewers are focusing on during an advertisement through the analysis of eye-tracking data. This allows for the evaluation of the effectiveness of advertising elements and the use of this information to improve future advertising.

[0059] Step 5:

[0060] The server generates optimized ads based on analysis of viewer preferences and eye movements. It creates ad creatives that include images and messages of products most relevant to the viewer.

[0061] Step 6:

[0062] The server determines the delivery schedule for the generated ads, deciding when, in what order, and how often the ads will be delivered to viewers. This information is then sent to the terminal.

[0063] Step 7:

[0064] The device delivers optimized advertisements received from the server to viewers according to a specified schedule. Eye-tracking data is also collected during viewing to record viewers' reactions to the advertisements.

[0065] Step 8:

[0066] Users can take interactive actions if they are interested in the advertisement they have seen. For example, they can request additional information using a remote control. This information is then provided to the server as feedback via the device.

[0067] Step 9:

[0068] The server uses the feedback data to evaluate the effectiveness of advertising campaigns and utilizes it to improve future advertising delivery strategies.

[0069] (Example 1)

[0070] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0071] In today's information-saturated society, viewers are often shown many irrelevant advertisements, leading to decreased advertising effectiveness. To address this challenge, it is necessary to make the most of viewers' viewing history and eye-tracking data to provide them with the most relevant content. However, conventional systems are still insufficient in accurately analyzing viewers' specific preferences and points of focus, and in effectively generating and delivering personalized advertisements.

[0072] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0073] In this invention, the server includes means for collecting data on viewing history in order to record viewer information, means for measuring data on eye movements in order to obtain the viewer's gaze, and means for predicting viewer trends using machine learning techniques based on the viewing history. This makes it possible to generate and deliver advertisements optimized for each viewer and maximize advertising effectiveness.

[0074] "Viewing history data" refers to records of content that viewers have watched in the past, including data such as program title, viewing time, and viewing time slot.

[0075] "Eye-gaze data" refers to data that records the movement and points of focus of viewers' eyes, and is used to analyze which parts of the screen their gaze is concentrated on.

[0076] "Machine learning technology" is a technique in which computer systems learn from past data, automatically identify patterns and trends, and make predictions.

[0077] "Predicting viewer trends" is a process that analyzes viewers' responses to preferred content and advertisements based on viewing history data and eye-tracking data, and predicts their future viewing behavior and interests.

[0078] "Personalized advertising" refers to advertisements that are customized based on the individual viewer's preferences and past viewing patterns, and are characterized by content that is more relevant and interesting to the viewer.

[0079] This invention is a system for delivering personalized advertisements according to the viewer's preferences. The system mainly consists of the interaction between terminals, servers, and users.

[0080] The device collects detailed information about the content the viewer is watching in real time. This collection includes viewing history information and gaze data. The device has built-in cameras and sensors to track the viewer's gaze, and through these devices, it obtains information about which parts of the screen the viewer is focusing on. The collected data is encrypted and securely transmitted to the server.

[0081] The server stores received viewing history data and eye-tracking data in a database. Using this stored data, machine learning models are run to analyze viewers' content preferences and behavioral patterns. This predicts the content and relevant advertisements that viewers will like. Generative AI models are used for this analysis, and based on the trends obtained, the content and delivery schedule of advertisements are formulated. Next, ad creatives that match the viewers' preferences are generated and prepared for delivery to devices in the most optimal way.

[0082] If a user is interested in an advertisement displayed on their device, they can request additional information via a remote control or other input device. This feedback information is sent to a server and used for further analysis and measuring the effectiveness of the advertisement.

[0083] As a concrete example, viewers who frequently watch cooking shows will be shown advertisements for new cooking utensils. These advertisements are composed of highly relevant images and messages to capture the viewer's interest. An example of a prompt could be: "Design the most effective advertisement based on the viewer's viewing history and eye-tracking data. What kind of cooking utensil advertisement would be appropriate for viewers who watch a lot of cooking shows?"

[0084] This system enables the efficient delivery of personalized ads tailored to viewers' interests and behaviors, thereby maximizing the effectiveness of advertising.

[0085] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0086] Step 1:

[0087] The device collects information about the content the viewer is watching in real time. Specifically, this includes data such as the time viewing started, the program name, and the time viewing ended. The device takes viewer viewing trends as input and outputs viewing history data based on this. The device also acquires eye-tracking data using its built-in camera and sensors, recording where and to what extent the viewer's gaze is directed. This is the output of eye-tracking data.

[0088] Step 2:

[0089] The device encrypts the collected viewing history data and eye-tracking data. The input for this process is raw data, which is securely processed by an encryption algorithm and output as encrypted data. The encrypted data is then transmitted to the server via the network.

[0090] Step 3:

[0091] The server decrypts the received encrypted data and stores the viewing history data and gaze data in the database. The input for this step includes the encrypted data sent from the terminal, and the output is the decrypted viewing history and gaze data.

[0092] Step 4:

[0093] The server runs a machine learning model using stored viewing history data and eye-tracking data. The input is viewer history and eye-tracking data, which is used to analyze viewer preferences and output data predicting behavioral patterns. The generative AI model uses this analysis to identify patterns and determine what kind of advertisement would be appropriate next.

[0094] Step 5:

[0095] The server generates ad creatives optimized for the viewer based on analysis results from machine learning models. The input includes viewer preference data, which is used to output personalized ads. The ad creatives are customized with images and messages designed to capture the viewer's interest.

[0096] Step 6:

[0097] The server considers the optimal delivery schedule for the generated ads and sends instructions to the terminal. In this step, the server calculates the time of day, order, and frequency of ad delivery, and outputs an optimized delivery schedule based on post-ad generation data and user viewing timing.

[0098] Step 7:

[0099] The device displays advertisements on the viewer's screen at predetermined times according to a schedule received from the server. The input is schedule data from the server, and the output is that the advertisement is displayed to the viewer based on that schedule. Eye-tracking data is collected again during the display to record the viewer's reaction to the advertisement.

[0100] Step 8:

[0101] If a user is interested in a displayed advertisement, they can request additional information via a remote control or similar device. In this step, the user's action is input, and the output is that feedback information is sent to the server. The server uses this feedback to further improve the advertisement.

[0102] (Application Example 1)

[0103] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0104] Traditional advertising delivery systems suffered from problems such as low accuracy in tailoring ads to individual viewers' interests, resulting in limited ad visibility and effectiveness. Furthermore, there was the challenge of rapidly optimizing ads to reflect viewers' real-time visual responses.

[0105] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0106] In this invention, the server includes means for collecting viewing history information to identify the viewer's content preferences, means for analyzing eye-tracking data to identify the viewer's gaze points, means for generating advertisements optimized for the viewer, means for visually highlighting and displaying selected advertisements using the viewer's head-mounted display, and means for measuring visual responses to understand the viewer's interest in the advertisements. This enables highly accurate ad delivery tailored to the viewer's interests and effective real-time ad adjustment.

[0107] "Viewing history information" refers to data that shows information about content and media that viewers have accessed in the past, and is used to analyze viewers' preferences.

[0108] "Eye-tracking data" refers to data that records the movement and focus of a viewer's gaze, and is used to understand which parts of the screen the viewer is paying attention to.

[0109] "Optimized advertising" refers to advertising content that is tailored to the interests and concerns of viewers based on their viewing history and eye-tracking data.

[0110] A "head-mounted display" is a display device worn on the head to provide visual information, and is a device designed to make it easier for users to receive information visually.

[0111] "Visual response" refers to the changes in a viewer's gaze and attention to the content displayed, and is an indicator used to measure the viewer's level of interest and attention.

[0112] This invention provides a system for efficiently delivering personalized advertisements to viewers. The system mainly consists of a server, a terminal which is the viewer's device, and a user.

[0113] The server collects viewers' viewing history information and identifies their content preferences. This includes data related to content and media the viewer has previously accessed. The server also analyzes eye-tracking data transmitted from the device to identify the viewer's gaze points. This eye-tracking data is acquired in real time using eye-tracking technologies such as OpenCV.

[0114] Furthermore, the server generates advertisements optimized for the viewer based on viewing history information and eye-tracking data. This process uses TENSORFLOW® to generate ad content that aligns with the viewer's preferences, creating personalized advertisements. The generated advertisements are delivered to the viewer's head-mounted display, such as smart glasses, and are displayed with visual emphasis.

[0115] The device displays advertisements sent from the server to the viewer at specified times. This allows viewers to see advertisements tailored to their interests in real time. Visual responses during viewing are measured using eye-tracking data and used to evaluate the user's level of interest.

[0116] For example, when a viewer watches a cooking show, if their gaze lingers on a particular recipe for an extended period, detailed advertisements for cooking utensils related to that recipe will be displayed. This ad display is generated based on the user's past viewing history and attention patterns.

[0117] Examples of prompts generated using AI models include the following:

[0118] "User viewing history: Cooking shows; Eye-tracking data: Long duration in the bottom right corner; Ad candidate: Cooking utensil ads. Generate the optimal visual and message."

[0119] This prompt allows the system to improve the user's viewing experience while also increasing the effectiveness of advertisements.

[0120] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0121] Step 1:

[0122] The server receives viewing history information from the terminal. This information includes metadata about the content the viewer has watched and is stored in a database to identify past content preferences. This provides the foundational data for analyzing the viewer's preferences.

[0123] Step 2:

[0124] The device collects eye-tracking data in real time while viewing and sends it to the server. This eye-tracking data is processed using OpenCV to analyze the position of the gaze and identify where the viewer is fixating on the screen. The eye-tracking data also includes time information for each point of fixation, and the eye movement pattern is also stored.

[0125] Step 3:

[0126] The server uses a generative AI model to generate ads optimized for the viewer, based on viewing history information and eye-tracking data. The AI ​​selects content based on specific interests and combines images and textual elements to create an ad with the most relevant message. This process analyzes the viewer's past behavior patterns and customizes the ad's creative elements based on the predicted results.

[0127] Step 4:

[0128] The server delivers the generated advertisements to the devices. During ad delivery, the timing and position of the ads are strategically selected to enhance the viewer's visual experience. The ads are presented in a visually prominent manner, and preparations are made to gauge the viewer's reaction to them.

[0129] Step 5:

[0130] The device displays advertisements to viewers and tracks eye-tracking data to record their visual responses. Viewer responses are quantified by measuring the amount of time they gaze at the ad and how often their eyes linger on that area. This allows for an understanding of how much attention the advertisement received.

[0131] Step 6:

[0132] Users can provide feedback on ads that interest them. For example, they can interact with the ads by pressing buttons to request more information. This user interaction is sent to the server via the device and used for measuring ad effectiveness and further optimizing ads. The server collects the user's interaction history and updates a database to use for future ad generation.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] This invention combines an emotion engine with a system that delivers personalized advertisements to viewers, thereby evaluating the viewer's emotional state in real time and optimizing the advertisements. The emotion engine has the function of analyzing the viewer's facial expression data and identifying their emotional state. This makes it possible to provide content that is tailored to the viewer's psychological state.

[0135] Collection and analysis of emotional data

[0136] terminal

[0137] The device uses its camera to capture the viewer's facial expressions while they are watching an advertisement. This data is analyzed in real time, allowing for an immediate assessment of their emotional state.

[0138] The collected facial expression data is sent to an emotion engine, which analyzes whether the viewer is experiencing joy, surprise, anger, sadness, indifference, or some other emotional state.

[0139] server

[0140] The server receives emotion data sent from the device and optimizes the ad content based on the viewer's emotional state. For example, when a viewer expresses surprise, the ad is adjusted to appeal to that emotion.

[0141] Furthermore, the server combines this data with viewing history and eye-tracking data to achieve more advanced personalization.

[0142] Ad optimization and delivery

[0143] server

[0144] The server adaptively changes the content of personalized ads based on the results of sentiment analysis. For example, if a viewer is feeling happy, it switches to an ad that contains a more positive message.

[0145] Ad delivery schedules are also optimized by taking emotional data into consideration. For example, ads are delivered at times when viewers are in a heightened mood.

[0146] terminal

[0147] The device delivers optimized advertisements received from the server to the viewer in real time. The advertisement display adaptively changes in response to the viewer's reaction.

[0148] User

[0149] Users can express emotional reactions to the ads they receive, and these reactions are collected by the system and used to improve future ads.

[0150] A concrete example of this system would be for the emotion engine to detect the viewer's excitement while they are watching a movie trailer, and then display advertisements for movie-related merchandise as related products. This method can maximize the viewer's interest.

[0151] The following describes the processing flow.

[0152] Step 1:

[0153] The device uses its built-in camera to capture facial expression data from viewers while they are watching advertisements or content. This makes it possible to monitor viewers' emotions in real time.

[0154] Step 2:

[0155] The device sends captured facial data to an emotion engine, which analyzes the viewer's emotional state. The emotion engine detects subtle changes in the viewer's face and classifies them into multiple emotional states, such as joy, surprise, anger, sadness, and indifference.

[0156] Step 3:

[0157] The server receives sentiment analysis results sent from the terminal and optimizes the advertisement according to the viewer's emotional state. For example, if the viewer is feeling surprised, the advertisement will be changed to one with more exciting content.

[0158] Step 4:

[0159] The server combines viewer emotional states, viewing history data, and eye-tracking data to generate personalized advertising content. This process also takes into account the viewer's past behavioral patterns.

[0160] Step 5:

[0161] The server determines the optimal time, order, and frequency for delivering the generated advertising content, and then sends that delivery schedule to the terminal.

[0162] Step 6:

[0163] The device delivers personalized advertisements received from the server to viewers based on a specified schedule. Even while the advertisement is playing, the device continuously collects the viewer's facial expression data and sends changes in emotion to the emotion engine in real time.

[0164] Step 7:

[0165] Users can express various emotional reactions to advertisements, and this feedback is used to improve future ad content and delivery strategies. The server analyzes this feedback to evaluate the effectiveness of the advertising campaign.

[0166] (Example 2)

[0167] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0168] Traditional advertising delivery systems struggled to consider viewers' interests and preferences, making it difficult to deliver the most relevant advertisements. Furthermore, the lack of mechanisms to understand viewers' emotional states in real time and deliver advertisements accordingly made it challenging to maximize the effectiveness of advertising.

[0169] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0170] In this invention, the server includes means for analyzing facial expression data to identify the viewer's emotional state, means for optimizing advertising content based on viewing history information and eye-tracking information, and means for generating personalized advertisements and delivering them to the viewer's device in real time. This enables advertising delivery that takes into account the viewer's emotional response.

[0171] "Viewer emotional state" refers to the emotional reactions, such as joy, surprise, and anger, that viewers exhibit when watching advertisements.

[0172] "Facial expression data" refers to image information obtained from the viewer's face, which is used to analyze their emotional state.

[0173] "Viewing history information" refers to a record of content that a viewer has watched in the past, and is used to identify viewing trends and preferences.

[0174] "Eye-tracking information" refers to data that records the movement of a viewer's gaze when they focus on a specific object, and is used to measure visual interest.

[0175] "Personalized advertising" refers to advertising content that is optimized based on each viewer's emotional state, viewing history, and eye-tracking information.

[0176] "Real-time delivery" refers to the immediate transmission and playback of advertising content based on the viewer's current state.

[0177] The invention will be implemented as an advertising delivery system based on viewer sentiment data. The details are described below.

[0178] 1. Hardware and software configuration

[0179] The device uses its built-in camera to capture the viewer's facial expressions while they are watching an advertisement. This captured data is processed in real time through image analysis software to extract facial feature points, which are then input into an emotion engine.

[0180] The server receives emotional characteristic data sent from the terminal and uses a generative AI model to analyze the viewer's emotional state (joy, surprise, anger, etc.). Using this analysis result along with past viewing history and eye-tracking information, the server provides an algorithm to select the most suitable advertisement.

[0181] Users receive personalized advertisements through their devices. Content messages tailored to the user's interests and emotional state are delivered in real time, and emotional responses to the advertisements are continuously collected by the device.

[0182] 2. Specific Examples

[0183] For example, if a user is watching a movie trailer and shows an excited expression in response to what is displayed, the device sends this data to the server. Based on this information, the server can select advertisements for movie-related posters and promotional items and deliver them to the user through the device. In this way, the advertisements are optimized to attract the user's interest.

[0184] 3. Example of a prompt statement

[0185] An example of a prompt might be, "Analyze what kind of advertising content is most effective based on the viewer's emotions of delight, and suggest the optimal advertisement." Using this prompt, it is expected that the generative AI model will suggest effective advertising content.

[0186] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0187] Step 1:

[0188] The device captures the viewer's facial expressions using its built-in camera while they are watching an advertisement. The input is image data obtained from the camera. Image analysis software is executed to extract feature points of facial expressions from this image data, and the output is generated as facial feature point data. Specifically, the camera is moved at a constant frame rate to continuously acquire images.

[0189] Step 2:

[0190] The device sends extracted facial feature point data to the emotion engine. The input is feature point data, and based on this data, a generative AI model analyzes the viewer's emotional state. The output is data indicating the viewer's emotional state. Specifically, it makes real-time judgments about emotions (e.g., joy, surprise, anger) and updates the data whenever there are changes.

[0191] Step 3:

[0192] The server receives emotional state data from the device and performs analysis to generate personalized advertisements. The input consists of emotional state data and other collected data (viewing history and eye-tracking information). The server processes this data through an AI model to select the most suitable advertisement. The output is a personalized advertising plan. Specifically, it uses emotional data as feedback to sequentially select the most effective advertisement for each emotional state.

[0193] Step 4:

[0194] The server remotely sends ads to devices based on the generated ad plan, and the devices display the ads. The input is optimized ad content, and the output is the ad visuals and audio delivered to the viewer. Specifically, the content is played at the appropriate timing and sequence so that viewers can see the ads in real time.

[0195] Step 5:

[0196] As users view advertisements, the device continuously monitors their reactions and collects data. The input consists of further facial expressions and responses from the user. This new data is then sent back to the emotion engine, which generates output that serves as reference material for optimizing future advertisements. Specific gestures and vocal responses are also incorporated into the data and used to improve the overall advertising effectiveness of the system.

[0197] (Application Example 2)

[0198] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0199] In traditional advertising methods, which do not consider the emotional state of viewers, effective personalization is difficult, and it is difficult to capture viewers' interest. It is necessary to optimize advertisements according to the emotional state of viewers to achieve more effective advertising.

[0200] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0201] In this invention, the server includes means for analyzing the emotional state of the viewer, means for generating personalized advertisements, and means for delivering advertisements to the viewer's information device. This enables optimal advertisement delivery that responds to the viewer's emotions.

[0202] "Viewer" refers to a person who watches an advertisement or content.

[0203] "Content preference" refers to the types and characteristics of content that viewers tend to watch in the past or prefer to watch.

[0204] "Viewing history information" refers to information about content that viewers have watched in the past, and includes data such as the number of views and viewing time.

[0205] "Eye-tracking data" refers to data about viewers' eye movements and points of focus, and is used to measure visual interest.

[0206] "Facial expression data" refers to data based on the facial expressions of viewers and is used for emotion analysis.

[0207] "Emotional state" refers to the psychological or emotional state of the viewer and is categorized into states such as joy, surprise, anger, and sadness.

[0208] "Personalized advertising" refers to advertisements that are customized to the individual preferences and emotional state of the viewer.

[0209] "Information devices" are devices used to display advertisements and content, and include smart glasses and mobile terminals.

[0210] "Optimization" is the process of adjusting conditions and parameters in order to achieve the greatest possible effect on a given goal.

[0211] In this embodiment, a system is constructed that optimizes advertisements based on the emotional state of the viewer and delivers personalized advertisements.

[0212] The server identifies content preferences by collecting viewers' viewing history and eye-tracking data. In addition, a camera on the device captures facial expression data, and an emotion engine analyzes the viewer's emotional state in real time. By using facial recognition APIs and machine learning models (e.g., TensorFlow and OpenCV) for emotion analysis, it is possible to evaluate the viewer's psychological response in detail.

[0213] The server then uses this data to generate advertisements tailored to the viewer's current emotional state. For example, if a viewer expresses surprise, an advertisement containing special offers for related products or services will be generated. This advertisement content is delivered to the viewer's information device in real time, allowing information to be provided at a time that matches the viewer's emotions.

[0214] A concrete example is real-time product promotion. When viewers show excitement while watching a demonstration of a new technology product, an advertisement for a limited-time discount on that product is immediately displayed. This system can capture viewers' interest and directly stimulate their desire to purchase.

[0215] By using generative AI models, the ad optimization process can be made even more sophisticated. An example of a necessary prompt would be, "How can I optimize relevant ads when a user shows a surprised expression?" This prompt allows the generative AI model to suggest the most suitable ad content, providing a more personalized advertising experience.

[0216] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0217] Step 1:

[0218] The device captures the viewer's facial expressions using a camera while they are watching content. The input is camera footage, which is output as facial expression data. This data is sent to a server in real time, ready for emotion analysis.

[0219] Step 2:

[0220] The server analyzes the received facial expression data to identify the viewer's emotional state. The input is facial expression data, and by analyzing this data using an emotion engine, it generates specific emotional states such as joy, surprise, and anger as output. Emotion recognition methods using TensorFlow or OpenCV are employed in this process.

[0221] Step 3:

[0222] The server combines the viewer's viewing history information and eye-tracking data with the emotional state obtained in step 2. The inputs include viewing history information, eye-tracking data, and emotional state, and data processing is performed based on this to determine the conditions for generating personalized ads. The output consists of the metrics and requirements necessary for ad generation.

[0223] Step 4:

[0224] The server generates appropriate ad content based on the viewer's current emotional state and viewing history. A generative AI model is used to design more personalized ads. The input consists of metrics and requirements for ad generation, and the generative AI model outputs specific ad content.

[0225] Step 5:

[0226] The server delivers the advertisement generated in step 4 to the viewer's information device, which is the terminal. The advertisement content is the input, and it is output to the terminal in real time, presenting the advertisement at a time appropriate for the viewer.

[0227] Step 6:

[0228] Users exhibit emotional reactions to the delivered advertisements. The device captures these reactions again and sends them back to the server. The input is facial expression data of the user's reaction to the advertisement, and the output is stored in the system as feedback for optimizing future advertisements.

[0229] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0230] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0231] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0232] [Second Embodiment]

[0233] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0234] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0235] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0236] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0237] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0238] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0239] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0240] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0241] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0242] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0243] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0244] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0245] This invention relates to a system for delivering advertisements optimized for viewers, and primarily concerns the generation and delivery of personalized advertisements using viewing history information and eye-tracking data.

[0246] Data collection and analysis

[0247] terminal

[0248] The device records information about the program the viewer is watching (program name, start time, end time) in real time. It also acquires viewer eye-tracking data through its built-in camera and sensors, determining which part of the screen the viewer is looking at.

[0249] The device encrypts the collected viewing history data and eye-tracking data and securely transmits it to the server.

[0250] server

[0251] The server stores the received viewing history data and eye-tracking data in a database. Based on this data, it runs a machine learning model to analyze viewers' content preferences and interests.

[0252] The server identifies viewer behavior patterns by comparing them with past similar viewer data. This allows it to predict the types of content that will interest viewers.

[0253] Ad generation and delivery

[0254] server

[0255] The server generates ad creatives based on the viewer's preferences and areas of interest. The generated ads are customized so that the product images and messages are most relevant to the viewer.

[0256] The server calculates the optimal time, order, and frequency for delivering the generated ads and sends the schedule for ad delivery to the devices.

[0257] terminal

[0258] The device delivers personalized advertisements to viewers according to a specified schedule. During this process, eye-tracking data is collected again to record viewers' reactions to the advertisements.

[0259] User

[0260] If a user is interested in an ad they have seen, they can request additional information using a remote control or similar device. This request information is sent to the server as feedback and used to measure the effectiveness of the ad.

[0261] In this way, advertising effectiveness can be maximized by efficiently delivering personalized advertisements to viewers. For example, users who enjoy watching cooking shows can be shown advertisements for the latest cooking appliances at the appropriate time. These advertisements include visual designs and messages that are likely to attract viewers' attention, thereby increasing their interest.

[0262] The following describes the processing flow.

[0263] Step 1:

[0264] The device records information about the program the viewer is watching, including the program name, start time, and end time. Furthermore, the device uses a built-in camera and eye-tracking sensor to acquire data on the viewer's eye movements while they are watching.

[0265] Step 2:

[0266] The device encrypts the collected viewing history data and eye-tracking data and sends it to the server. The transmitted data is used for analysis while protecting the viewer's privacy.

[0267] Step 3:

[0268] The server stores the received viewing data in a database and builds viewer profiles. Using machine learning algorithms, it analyzes viewers' content preferences based on their viewing history and identifies their preferred trends.

[0269] Step 4:

[0270] The server identifies points that viewers are focusing on during an advertisement through the analysis of eye-tracking data. This allows for the evaluation of the effectiveness of advertising elements and the use of this information to improve future advertising.

[0271] Step 5:

[0272] The server generates optimized ads based on analysis of viewer preferences and eye movements. It creates ad creatives that include images and messages of products most relevant to the viewer.

[0273] Step 6:

[0274] The server determines the delivery schedule for the generated ads, deciding when, in what order, and how often the ads will be delivered to viewers. This information is then sent to the terminal.

[0275] Step 7:

[0276] The device delivers optimized advertisements received from the server to viewers according to a specified schedule. Eye-tracking data is also collected during viewing to record viewers' reactions to the advertisements.

[0277] Step 8:

[0278] Users can take interactive actions if they are interested in the advertisement they have seen. For example, they can request additional information using a remote control. This information is then provided to the server as feedback via the device.

[0279] Step 9:

[0280] The server uses the feedback data to evaluate the effectiveness of advertising campaigns and utilizes it to improve future advertising delivery strategies.

[0281] (Example 1)

[0282] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0283] In modern society with an overabundance of information, viewers are often shown many advertisements that are not relevant to them, resulting in a decline in advertising effectiveness. To address this issue, it is necessary to make the most of viewers' viewing histories and eye-tracking data to provide the most relevant content to each person. However, conventional systems are not yet sufficient in accurately analyzing viewers' detailed preferences and areas of focus and effectively generating and delivering personalized advertisements.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0285] In this invention, the server includes means for collecting data related to the viewing history to record viewers' information, means for measuring data related to eye movements to obtain viewers' viewpoints, and means for predicting viewers' tendencies using machine learning techniques based on the viewing history. This makes it possible to generate and deliver optimized advertisements for each viewer and maximize the advertising effect.

[0286] "Data related to the viewing history" is a record of the content that a viewer has watched in the past, and includes data such as program names, viewing times, and viewing time zones.

[0287] "Data related to eye movements" is data that records the movements and fixation points of a viewer's eyes, and is information used to analyze which parts the eyes are focused on.

[0288] "Machine learning techniques" are techniques in which a computer system learns from past data, automatically identifies patterns and trends, and makes predictions.

[0289] "Prediction of viewers' tendencies" is a process of analyzing viewers' reactions to content and advertisements they like based on viewing history data and eye-tracking data, and predicting future viewing behavior and interests.

[0290] "Personalized advertising" refers to advertisements that are customized based on the individual viewer's preferences and past viewing patterns, and are characterized by content that is more relevant and interesting to the viewer.

[0291] This invention is a system for delivering personalized advertisements according to the viewer's preferences. The system mainly consists of the interaction between terminals, servers, and users.

[0292] The device collects detailed information about the content the viewer is watching in real time. This collection includes viewing history information and gaze data. The device has built-in cameras and sensors to track the viewer's gaze, and through these devices, it obtains information about which parts of the screen the viewer is focusing on. The collected data is encrypted and securely transmitted to the server.

[0293] The server stores received viewing history data and eye-tracking data in a database. Using this stored data, machine learning models are run to analyze viewers' content preferences and behavioral patterns. This predicts the content and relevant advertisements that viewers will like. Generative AI models are used for this analysis, and based on the trends obtained, the content and delivery schedule of advertisements are formulated. Next, ad creatives that match the viewers' preferences are generated and prepared for delivery to devices in the most optimal way.

[0294] If a user is interested in an advertisement displayed on their device, they can request additional information via a remote control or other input device. This feedback information is sent to a server and used for further analysis and measuring the effectiveness of the advertisement.

[0295] As a concrete example, viewers who frequently watch cooking shows will be shown advertisements for new cooking utensils. These advertisements are composed of highly relevant images and messages to capture the viewer's interest. An example of a prompt could be: "Design the most effective advertisement based on the viewer's viewing history and eye-tracking data. What kind of cooking utensil advertisement would be appropriate for viewers who watch a lot of cooking shows?"

[0296] This system enables the efficient delivery of personalized ads tailored to viewers' interests and behaviors, thereby maximizing the effectiveness of advertising.

[0297] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0298] Step 1:

[0299] The device collects information about the content the viewer is watching in real time. Specifically, this includes data such as the time viewing started, the program name, and the time viewing ended. The device takes viewer viewing trends as input and outputs viewing history data based on this. The device also acquires eye-tracking data using its built-in camera and sensors, recording where and to what extent the viewer's gaze is directed. This is the output of eye-tracking data.

[0300] Step 2:

[0301] The device encrypts the collected viewing history data and eye-tracking data. The input for this process is raw data, which is securely processed by an encryption algorithm and output as encrypted data. The encrypted data is then transmitted to the server via the network.

[0302] Step 3:

[0303] The server decrypts the received encrypted data and stores the viewing history data and gaze data in the database. The input for this step includes the encrypted data transmitted from the terminal, and the decrypted viewing history and gaze data are output.

[0304] Step 4:

[0305] The server runs a machine learning model using the stored viewing history data and gaze data. The inputs are the viewer's history and gaze data, and based on this, data for analyzing the viewer's preferences and predicting behavior patterns is output. The generative AI model uses this analysis result to identify patterns and determine what kind of next advertisement is appropriate.

[0306] Step 5:

[0307] The server generates an optimal ad creative for the viewer based on the analysis result of the machine learning model. The input is the viewer's preference data, and a personalized ad is output based on this. The ad creative is customized with images and messages that attract the viewer's interest.

[0308] Step 6:

[0309] The server considers the optimal delivery schedule of the generated ad and sends an instruction to the terminal. In this step, the delivery time zone, order, and frequency of the ad are calculated, and an optimized delivery schedule is output with reference to the data after ad generation and the user's viewing timing as the input.

[0310] Step 7:

[0311] The terminal displays the ad on the viewer's screen at a predetermined time according to the schedule received from the server. The input is the schedule data from the server, and based on this, the output of displaying the ad to the viewer is obtained. Again, the gaze data during display is collected, and the viewer's reaction to the ad is recorded.

[0312] Step 8:

[0313] If a user is interested in a displayed advertisement, they can request additional information via a remote control or similar device. In this step, the user's action is input, and the output is that feedback information is sent to the server. The server uses this feedback to further improve the advertisement.

[0314] (Application Example 1)

[0315] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0316] Traditional advertising delivery systems suffered from problems such as low accuracy in tailoring ads to individual viewers' interests, resulting in limited ad visibility and effectiveness. Furthermore, there was the challenge of rapidly optimizing ads to reflect viewers' real-time visual responses.

[0317] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0318] In this invention, the server includes means for collecting viewing history information to identify the viewer's content preferences, means for analyzing eye-tracking data to identify the viewer's gaze points, means for generating advertisements optimized for the viewer, means for visually highlighting and displaying selected advertisements using the viewer's head-mounted display, and means for measuring visual responses to understand the viewer's interest in the advertisements. This enables highly accurate ad delivery tailored to the viewer's interests and effective real-time ad adjustment.

[0319] "Viewing history information" refers to data that shows information about content and media that viewers have accessed in the past, and is used to analyze viewers' preferences.

[0320] "Eye-tracking data" refers to data that records the movement and focus of a viewer's gaze, and is used to understand which parts of the screen the viewer is paying attention to.

[0321] "Optimized advertising" refers to advertising content that is tailored to the interests and concerns of viewers based on their viewing history and eye-tracking data.

[0322] A "head-mounted display" is a display device worn on the head to provide visual information, and is a device designed to make it easier for users to receive information visually.

[0323] "Visual response" refers to the changes in a viewer's gaze and attention to the content displayed, and is an indicator used to measure the viewer's level of interest and attention.

[0324] This invention provides a system for efficiently delivering personalized advertisements to viewers. The system mainly consists of a server, a terminal which is the viewer's device, and a user.

[0325] The server collects viewers' viewing history information and identifies their content preferences. This includes data related to content and media the viewer has previously accessed. The server also analyzes eye-tracking data transmitted from the device to identify the viewer's gaze points. This eye-tracking data is acquired in real time using eye-tracking technologies such as OpenCV.

[0326] Furthermore, the server generates advertisements optimized for the viewer based on viewing history information and eye-tracking data. This process uses TensorFlow to generate ad content tailored to the viewer's preferences, creating personalized advertisements. The generated advertisements are delivered to the viewer's head-mounted display, such as smart glasses, and displayed with visual emphasis.

[0327] The device displays advertisements sent from the server to the viewer at specified times. This allows viewers to see advertisements tailored to their interests in real time. Visual responses during viewing are measured using eye-tracking data and used to evaluate the user's level of interest.

[0328] For example, when a viewer watches a cooking show, if their gaze lingers on a particular recipe for an extended period, detailed advertisements for cooking utensils related to that recipe will be displayed. This ad display is generated based on the user's past viewing history and attention patterns.

[0329] Examples of prompts generated using AI models include the following:

[0330] "User viewing history: Cooking shows; Eye-tracking data: Long duration in the bottom right corner; Ad candidate: Cooking utensil ads. Generate the optimal visual and message."

[0331] This prompt allows the system to improve the user's viewing experience while also increasing the effectiveness of advertisements.

[0332] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0333] Step 1:

[0334] The server receives viewing history information from the terminal. This information includes metadata about the content the viewer has watched and is stored in a database to identify past content preferences. This provides the foundational data for analyzing the viewer's preferences.

[0335] Step 2:

[0336] The device collects eye-tracking data in real time while viewing and sends it to the server. This eye-tracking data is processed using OpenCV to analyze the position of the gaze and identify where the viewer is fixating on the screen. The eye-tracking data also includes time information for each point of fixation, and the eye movement pattern is also stored.

[0337] Step 3:

[0338] The server uses a generative AI model to generate ads optimized for the viewer, based on viewing history information and eye-tracking data. The AI ​​selects content based on specific interests and combines images and textual elements to create an ad with the most relevant message. This process analyzes the viewer's past behavior patterns and customizes the ad's creative elements based on the predicted results.

[0339] Step 4:

[0340] The server delivers the generated advertisements to the devices. During ad delivery, the timing and position of the ads are strategically selected to enhance the viewer's visual experience. The ads are presented in a visually prominent manner, and preparations are made to gauge the viewer's reaction to them.

[0341] Step 5:

[0342] The device displays advertisements to viewers and tracks eye-tracking data to record their visual responses. Viewer responses are quantified by measuring the amount of time they gaze at the ad and how often their eyes linger on that area. This allows for an understanding of how much attention the advertisement received.

[0343] Step 6:

[0344] Users can provide feedback on ads that interest them. For example, they can interact with the ads by pressing buttons to request more information. This user interaction is sent to the server via the device and used for measuring ad effectiveness and further optimizing ads. The server collects the user's interaction history and updates a database to use for future ad generation.

[0345] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0346] This invention combines an emotion engine with a system that delivers personalized advertisements to viewers, thereby evaluating the viewer's emotional state in real time and optimizing the advertisements. The emotion engine has the function of analyzing the viewer's facial expression data and identifying their emotional state. This makes it possible to provide content that is tailored to the viewer's psychological state.

[0347] Collection and analysis of emotional data

[0348] terminal

[0349] The device uses its camera to capture the viewer's facial expressions while they are watching an advertisement. This data is analyzed in real time, allowing for an immediate assessment of their emotional state.

[0350] The collected facial expression data is sent to an emotion engine, which analyzes whether the viewer is experiencing joy, surprise, anger, sadness, indifference, or some other emotional state.

[0351] server

[0352] The server receives emotion data sent from the device and optimizes the ad content based on the viewer's emotional state. For example, when a viewer expresses surprise, the ad is adjusted to appeal to that emotion.

[0353] Furthermore, the server combines this data with viewing history and eye-tracking data to achieve more advanced personalization.

[0354] Ad optimization and delivery

[0355] server

[0356] The server adaptively changes the content of personalized ads based on the results of sentiment analysis. For example, if a viewer is feeling happy, it switches to an ad that contains a more positive message.

[0357] Ad delivery schedules are also optimized by taking emotional data into consideration. For example, ads are delivered at times when viewers are in a heightened mood.

[0358] terminal

[0359] The device delivers optimized advertisements received from the server to the viewer in real time. The advertisement display adaptively changes in response to the viewer's reaction.

[0360] User

[0361] Users can express emotional reactions to the ads they receive, and these reactions are collected by the system and used to improve future ads.

[0362] A concrete example of this system would be for the emotion engine to detect the viewer's excitement while they are watching a movie trailer, and then display advertisements for movie-related merchandise as related products. This method can maximize the viewer's interest.

[0363] The following describes the processing flow.

[0364] Step 1:

[0365] The device uses its built-in camera to capture facial expression data from viewers while they are watching advertisements or content. This makes it possible to monitor viewers' emotions in real time.

[0366] Step 2:

[0367] The device sends captured facial data to an emotion engine, which analyzes the viewer's emotional state. The emotion engine detects subtle changes in the viewer's face and classifies them into multiple emotional states, such as joy, surprise, anger, sadness, and indifference.

[0368] Step 3:

[0369] The server receives sentiment analysis results sent from the terminal and optimizes the advertisement according to the viewer's emotional state. For example, if the viewer is feeling surprised, the advertisement will be changed to one with more exciting content.

[0370] Step 4:

[0371] The server combines viewer emotional states, viewing history data, and eye-tracking data to generate personalized advertising content. This process also takes into account the viewer's past behavioral patterns.

[0372] Step 5:

[0373] The server determines the optimal time, order, and frequency for delivering the generated advertising content, and then sends that delivery schedule to the terminal.

[0374] Step 6:

[0375] The device delivers personalized advertisements received from the server to viewers based on a specified schedule. Even while the advertisement is playing, the device continuously collects the viewer's facial expression data and sends changes in emotion to the emotion engine in real time.

[0376] Step 7:

[0377] Users can express various emotional reactions to advertisements, and this feedback is used to improve future ad content and delivery strategies. The server analyzes this feedback to evaluate the effectiveness of the advertising campaign.

[0378] (Example 2)

[0379] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0380] Traditional advertising delivery systems struggled to consider viewers' interests and preferences, making it difficult to deliver the most relevant advertisements. Furthermore, the lack of mechanisms to understand viewers' emotional states in real time and deliver advertisements accordingly made it challenging to maximize the effectiveness of advertising.

[0381] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0382] In this invention, the server includes means for analyzing facial expression data to identify the viewer's emotional state, means for optimizing advertising content based on viewing history information and eye-tracking information, and means for generating personalized advertisements and delivering them to the viewer's device in real time. This enables advertising delivery that takes into account the viewer's emotional response.

[0383] "Viewer emotional state" refers to the emotional reactions, such as joy, surprise, and anger, that viewers exhibit when watching advertisements.

[0384] "Facial expression data" refers to image information obtained from the viewer's face, which is used to analyze their emotional state.

[0385] "Viewing history information" refers to a record of content that a viewer has watched in the past, and is used to identify viewing trends and preferences.

[0386] "Eye-tracking information" refers to data that records the movement of a viewer's gaze when they focus on a specific object, and is used to measure visual interest.

[0387] "Personalized advertising" refers to advertising content that is optimized based on each viewer's emotional state, viewing history, and eye-tracking information.

[0388] "Real-time delivery" refers to the immediate transmission and playback of advertising content based on the viewer's current state.

[0389] The invention will be implemented as an advertising delivery system based on viewer sentiment data. The details are described below.

[0390] 1. Hardware and software configuration

[0391] The device uses its built-in camera to capture the viewer's facial expressions while they are watching an advertisement. This captured data is processed in real time through image analysis software to extract facial feature points, which are then input into an emotion engine.

[0392] The server receives emotional characteristic data sent from the terminal and uses a generative AI model to analyze the viewer's emotional state (joy, surprise, anger, etc.). Using this analysis result along with past viewing history and eye-tracking information, the server provides an algorithm to select the most suitable advertisement.

[0393] Users receive personalized advertisements through their devices. Content messages tailored to the user's interests and emotional state are delivered in real time, and emotional responses to the advertisements are continuously collected by the device.

[0394] 2. Specific Examples

[0395] For example, if a user is watching a movie trailer and shows an excited expression in response to what is displayed, the device sends this data to the server. Based on this information, the server can select advertisements for movie-related posters and promotional items and deliver them to the user through the device. In this way, the advertisements are optimized to attract the user's interest.

[0396] 3. Example of a prompt statement

[0397] An example of a prompt might be, "Analyze what kind of advertising content is most effective based on the viewer's emotions of delight, and suggest the optimal advertisement." Using this prompt, it is expected that the generative AI model will suggest effective advertising content.

[0398] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0399] Step 1:

[0400] The device captures the viewer's facial expressions using its built-in camera while they are watching an advertisement. The input is image data obtained from the camera. Image analysis software is executed to extract feature points of facial expressions from this image data, and the output is generated as facial feature point data. Specifically, the camera is moved at a constant frame rate to continuously acquire images.

[0401] Step 2:

[0402] The device sends extracted facial feature point data to the emotion engine. The input is feature point data, and based on this data, a generative AI model analyzes the viewer's emotional state. The output is data indicating the viewer's emotional state. Specifically, it makes real-time judgments about emotions (e.g., joy, surprise, anger) and updates the data whenever there are changes.

[0403] Step 3:

[0404] The server receives emotional state data from the device and performs analysis to generate personalized advertisements. The input consists of emotional state data and other collected data (viewing history and eye-tracking information). The server processes this data through an AI model to select the most suitable advertisement. The output is a personalized advertising plan. Specifically, it uses emotional data as feedback to sequentially select the most effective advertisement for each emotional state.

[0405] Step 4:

[0406] The server remotely sends ads to devices based on the generated ad plan, and the devices display the ads. The input is optimized ad content, and the output is the ad visuals and audio delivered to the viewer. Specifically, the content is played at the appropriate timing and sequence so that viewers can see the ads in real time.

[0407] Step 5:

[0408] As users view advertisements, the device continuously monitors their reactions and collects data. The input consists of further facial expressions and responses from the user. This new data is then sent back to the emotion engine, which generates output that serves as reference material for optimizing future advertisements. Specific gestures and vocal responses are also incorporated into the data and used to improve the overall advertising effectiveness of the system.

[0409] (Application Example 2)

[0410] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0411] In traditional advertising methods, which do not consider the emotional state of viewers, effective personalization is difficult, and it is difficult to capture viewers' interest. It is necessary to optimize advertisements according to the emotional state of viewers to achieve more effective advertising.

[0412] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0413] In this invention, the server includes means for analyzing the emotional state of the viewer, means for generating personalized advertisements, and means for delivering advertisements to the viewer's information device. This enables optimal advertisement delivery that responds to the viewer's emotions.

[0414] "Viewer" refers to a person who watches an advertisement or content.

[0415] "Content preference" refers to the types and characteristics of content that viewers tend to watch in the past or prefer to watch.

[0416] "Viewing history information" refers to information about content that viewers have watched in the past, and includes data such as the number of views and viewing time.

[0417] "Eye-tracking data" refers to data about viewers' eye movements and points of focus, and is used to measure visual interest.

[0418] "Facial expression data" refers to data based on the facial expressions of viewers and is used for emotion analysis.

[0419] "Emotional state" refers to the psychological or emotional state of the viewer and is categorized into states such as joy, surprise, anger, and sadness.

[0420] "Personalized advertising" refers to advertisements that are customized to the individual preferences and emotional state of the viewer.

[0421] "Information devices" are devices used to display advertisements and content, and include smart glasses and mobile terminals.

[0422] "Optimization" is the process of adjusting conditions and parameters in order to achieve the greatest possible effect on a given goal.

[0423] In this embodiment, a system is constructed that optimizes advertisements based on the emotional state of the viewer and delivers personalized advertisements.

[0424] The server identifies content preferences by collecting viewers' viewing history and eye-tracking data. In addition, a camera on the device captures facial expression data, and an emotion engine analyzes the viewer's emotional state in real time. By using facial recognition APIs and machine learning models (e.g., TensorFlow and OpenCV) for emotion analysis, it is possible to evaluate the viewer's psychological response in detail.

[0425] The server then uses this data to generate advertisements tailored to the viewer's current emotional state. For example, if a viewer expresses surprise, an advertisement containing special offers for related products or services will be generated. This advertisement content is delivered to the viewer's information device in real time, allowing information to be provided at a time that matches the viewer's emotions.

[0426] A concrete example is real-time product promotion. When viewers show excitement while watching a demonstration of a new technology product, an advertisement for a limited-time discount on that product is immediately displayed. This system can capture viewers' interest and directly stimulate their desire to purchase.

[0427] By using generative AI models, the ad optimization process can be made even more sophisticated. An example of a necessary prompt would be, "How can I optimize relevant ads when a user shows a surprised expression?" This prompt allows the generative AI model to suggest the most suitable ad content, providing a more personalized advertising experience.

[0428] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0429] Step 1:

[0430] The device captures the viewer's facial expressions using a camera while they are watching content. The input is camera footage, which is output as facial expression data. This data is sent to a server in real time, ready for emotion analysis.

[0431] Step 2:

[0432] The server analyzes the received facial expression data to identify the viewer's emotional state. The input is facial expression data, and by analyzing this data using an emotion engine, it generates specific emotional states such as joy, surprise, and anger as output. Emotion recognition methods using TensorFlow or OpenCV are employed in this process.

[0433] Step 3:

[0434] The server combines the viewer's viewing history information and eye-tracking data with the emotional state obtained in step 2. The inputs include viewing history information, eye-tracking data, and emotional state, and data processing is performed based on this to determine the conditions for generating personalized ads. The output consists of the metrics and requirements necessary for ad generation.

[0435] Step 4:

[0436] The server generates appropriate ad content based on the viewer's current emotional state and viewing history. A generative AI model is used to design more personalized ads. The input consists of metrics and requirements for ad generation, and the generative AI model outputs specific ad content.

[0437] Step 5:

[0438] The server delivers the advertisement generated in step 4 to the viewer's information device, which is the terminal. The advertisement content is the input, and it is output to the terminal in real time, presenting the advertisement at a time appropriate for the viewer.

[0439] Step 6:

[0440] Users exhibit emotional reactions to the delivered advertisements. The device captures these reactions again and sends them back to the server. The input is facial expression data of the user's reaction to the advertisement, and the output is stored in the system as feedback for optimizing future advertisements.

[0441] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0442] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0443] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0444] [Third Embodiment]

[0445] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0446] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0447] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0448] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0449] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0451] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0452] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0453] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0454] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0455] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0456] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0457] This invention relates to a system for delivering advertisements optimized for viewers, and primarily concerns the generation and delivery of personalized advertisements using viewing history information and eye-tracking data.

[0458] Data collection and analysis

[0459] terminal

[0460] The device records information about the program the viewer is watching (program name, start time, end time) in real time. It also acquires viewer eye-tracking data through its built-in camera and sensors, determining which part of the screen the viewer is looking at.

[0461] The device encrypts the collected viewing history data and eye-tracking data and securely transmits it to the server.

[0462] server

[0463] The server stores the received viewing history data and eye-tracking data in a database. Based on this data, it runs a machine learning model to analyze viewers' content preferences and interests.

[0464] The server identifies viewer behavior patterns by comparing them with past similar viewer data. This allows it to predict the types of content that will interest viewers.

[0465] Ad generation and delivery

[0466] server

[0467] The server generates ad creatives based on the viewer's preferences and areas of interest. The generated ads are customized so that the product images and messages are most relevant to the viewer.

[0468] The server calculates the optimal time, order, and frequency for delivering the generated ads and sends the schedule for ad delivery to the devices.

[0469] terminal

[0470] The device delivers personalized advertisements to viewers according to a specified schedule. During this process, eye-tracking data is collected again to record viewers' reactions to the advertisements.

[0471] User

[0472] If a user is interested in an ad they have seen, they can request additional information using a remote control or similar device. This request information is sent to the server as feedback and used to measure the effectiveness of the ad.

[0473] In this way, advertising effectiveness can be maximized by efficiently delivering personalized advertisements to viewers. For example, users who enjoy watching cooking shows can be shown advertisements for the latest cooking appliances at the appropriate time. These advertisements include visual designs and messages that are likely to attract viewers' attention, thereby increasing their interest.

[0474] The following describes the processing flow.

[0475] Step 1:

[0476] The device records information about the program the viewer is watching, including the program name, start time, and end time. Furthermore, the device uses a built-in camera and eye-tracking sensor to acquire data on the viewer's eye movements while they are watching.

[0477] Step 2:

[0478] The device encrypts the collected viewing history data and eye-tracking data and sends it to the server. The transmitted data is used for analysis while protecting the viewer's privacy.

[0479] Step 3:

[0480] The server stores the received viewing data in a database and builds viewer profiles. Using machine learning algorithms, it analyzes viewers' content preferences based on their viewing history and identifies their preferred trends.

[0481] Step 4:

[0482] The server identifies points that viewers are focusing on during an advertisement through the analysis of eye-tracking data. This allows for the evaluation of the effectiveness of advertising elements and the use of this information to improve future advertising.

[0483] Step 5:

[0484] The server generates optimized ads based on analysis of viewer preferences and eye movements. It creates ad creatives that include images and messages of products most relevant to the viewer.

[0485] Step 6:

[0486] The server determines the delivery schedule for the generated ads, deciding when, in what order, and how often the ads will be delivered to viewers. This information is then sent to the terminal.

[0487] Step 7:

[0488] The device delivers optimized advertisements received from the server to viewers according to a specified schedule. Eye-tracking data is also collected during viewing to record viewers' reactions to the advertisements.

[0489] Step 8:

[0490] Users can take interactive actions if they are interested in the advertisement they have seen. For example, they can request additional information using a remote control. This information is then provided to the server as feedback via the device.

[0491] Step 9:

[0492] The server uses the feedback data to evaluate the effectiveness of advertising campaigns and utilizes it to improve future advertising delivery strategies.

[0493] (Example 1)

[0494] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0495] In today's information-saturated society, viewers are often shown many irrelevant advertisements, leading to decreased advertising effectiveness. To address this challenge, it is necessary to make the most of viewers' viewing history and eye-tracking data to provide them with the most relevant content. However, conventional systems are still insufficient in accurately analyzing viewers' specific preferences and points of focus, and in effectively generating and delivering personalized advertisements.

[0496] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0497] In this invention, the server includes means for collecting data on viewing history in order to record viewer information, means for measuring data on eye movements in order to obtain the viewer's gaze, and means for predicting viewer trends using machine learning techniques based on the viewing history. This makes it possible to generate and deliver advertisements optimized for each viewer and maximize advertising effectiveness.

[0498] "Viewing history data" refers to records of content that viewers have watched in the past, including data such as program title, viewing time, and viewing time slot.

[0499] "Eye-gaze data" refers to data that records the movement and points of focus of viewers' eyes, and is used to analyze which parts of the screen their gaze is concentrated on.

[0500] "Machine learning technology" is a technique in which computer systems learn from past data, automatically identify patterns and trends, and make predictions.

[0501] "Predicting viewer trends" is a process that analyzes viewers' responses to preferred content and advertisements based on viewing history data and eye-tracking data, and predicts their future viewing behavior and interests.

[0502] "Personalized advertising" refers to advertisements that are customized based on the individual viewer's preferences and past viewing patterns, and are characterized by content that is more relevant and interesting to the viewer.

[0503] This invention is a system for delivering personalized advertisements according to the viewer's preferences. The system mainly consists of the interaction between terminals, servers, and users.

[0504] The device collects detailed information about the content the viewer is watching in real time. This collection includes viewing history information and gaze data. The device has built-in cameras and sensors to track the viewer's gaze, and through these devices, it obtains information about which parts of the screen the viewer is focusing on. The collected data is encrypted and securely transmitted to the server.

[0505] The server stores received viewing history data and eye-tracking data in a database. Using this stored data, machine learning models are run to analyze viewers' content preferences and behavioral patterns. This predicts the content and relevant advertisements that viewers will like. Generative AI models are used for this analysis, and based on the trends obtained, the content and delivery schedule of advertisements are formulated. Next, ad creatives that match the viewers' preferences are generated and prepared for delivery to devices in the most optimal way.

[0506] If a user is interested in an advertisement displayed on their device, they can request additional information via a remote control or other input device. This feedback information is sent to a server and used for further analysis and measuring the effectiveness of the advertisement.

[0507] As a concrete example, viewers who frequently watch cooking shows will be shown advertisements for new cooking utensils. These advertisements are composed of highly relevant images and messages to capture the viewer's interest. An example of a prompt could be: "Design the most effective advertisement based on the viewer's viewing history and eye-tracking data. What kind of cooking utensil advertisement would be appropriate for viewers who watch a lot of cooking shows?"

[0508] This system enables the efficient delivery of personalized ads tailored to viewers' interests and behaviors, thereby maximizing the effectiveness of advertising.

[0509] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0510] Step 1:

[0511] The device collects information about the content the viewer is watching in real time. Specifically, this includes data such as the time viewing started, the program name, and the time viewing ended. The device takes viewer viewing trends as input and outputs viewing history data based on this. The device also acquires eye-tracking data using its built-in camera and sensors, recording where and to what extent the viewer's gaze is directed. This is the output of eye-tracking data.

[0512] Step 2:

[0513] The device encrypts the collected viewing history data and eye-tracking data. The input for this process is raw data, which is securely processed by an encryption algorithm and output as encrypted data. The encrypted data is then transmitted to the server via the network.

[0514] Step 3:

[0515] The server decrypts the received encrypted data and stores the viewing history data and gaze data in the database. The input for this step includes the encrypted data sent from the terminal, and the output is the decrypted viewing history and gaze data.

[0516] Step 4:

[0517] The server runs a machine learning model using stored viewing history data and eye-tracking data. The input is viewer history and eye-tracking data, which is used to analyze viewer preferences and output data predicting behavioral patterns. The generative AI model uses this analysis to identify patterns and determine what kind of advertisement would be appropriate next.

[0518] Step 5:

[0519] The server generates ad creatives optimized for the viewer based on analysis results from machine learning models. The input includes viewer preference data, which is used to output personalized ads. The ad creatives are customized with images and messages designed to capture the viewer's interest.

[0520] Step 6:

[0521] The server considers the optimal delivery schedule for the generated ads and sends instructions to the terminal. In this step, the server calculates the time of day, order, and frequency of ad delivery, and outputs an optimized delivery schedule based on post-ad generation data and user viewing timing.

[0522] Step 7:

[0523] The device displays advertisements on the viewer's screen at predetermined times according to a schedule received from the server. The input is schedule data from the server, and the output is that the advertisement is displayed to the viewer based on that schedule. Eye-tracking data is collected again during the display to record the viewer's reaction to the advertisement.

[0524] Step 8:

[0525] If a user is interested in a displayed advertisement, they can request additional information via a remote control or similar device. In this step, the user's action is input, and the output is that feedback information is sent to the server. The server uses this feedback to further improve the advertisement.

[0526] (Application Example 1)

[0527] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0528] Traditional advertising delivery systems suffered from problems such as low accuracy in tailoring ads to individual viewers' interests, resulting in limited ad visibility and effectiveness. Furthermore, there was the challenge of rapidly optimizing ads to reflect viewers' real-time visual responses.

[0529] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0530] In this invention, the server includes means for collecting viewing history information to identify the viewer's content preferences, means for analyzing eye-tracking data to identify the viewer's gaze points, means for generating advertisements optimized for the viewer, means for visually highlighting and displaying selected advertisements using the viewer's head-mounted display, and means for measuring visual responses to understand the viewer's interest in the advertisements. This enables highly accurate ad delivery tailored to the viewer's interests and effective real-time ad adjustment.

[0531] "Viewing history information" refers to data that shows information about content and media that viewers have accessed in the past, and is used to analyze viewers' preferences.

[0532] "Eye-tracking data" refers to data that records the movement and focus of a viewer's gaze, and is used to understand which parts of the screen the viewer is paying attention to.

[0533] "Optimized advertising" refers to advertising content that is tailored to the interests and concerns of viewers based on their viewing history and eye-tracking data.

[0534] A "head-mounted display" is a display device worn on the head to provide visual information, and is a device designed to make it easier for users to receive information visually.

[0535] "Visual response" refers to the changes in a viewer's gaze and attention to the content displayed, and is an indicator used to measure the viewer's level of interest and attention.

[0536] This invention provides a system for efficiently delivering personalized advertisements to viewers. The system mainly consists of a server, a terminal which is the viewer's device, and a user.

[0537] The server collects viewers' viewing history information and identifies their content preferences. This includes data related to content and media the viewer has previously accessed. The server also analyzes eye-tracking data transmitted from the device to identify the viewer's gaze points. This eye-tracking data is acquired in real time using eye-tracking technologies such as OpenCV.

[0538] Furthermore, the server generates advertisements optimized for the viewer based on viewing history information and eye-tracking data. This process uses TensorFlow to generate ad content tailored to the viewer's preferences, creating personalized advertisements. The generated advertisements are delivered to the viewer's head-mounted display, such as smart glasses, and displayed with visual emphasis.

[0539] The device displays advertisements sent from the server to the viewer at specified times. This allows viewers to see advertisements tailored to their interests in real time. Visual responses during viewing are measured using eye-tracking data and used to evaluate the user's level of interest.

[0540] For example, when a viewer watches a cooking show, if their gaze lingers on a particular recipe for an extended period, detailed advertisements for cooking utensils related to that recipe will be displayed. This ad display is generated based on the user's past viewing history and attention patterns.

[0541] Examples of prompts generated using AI models include the following:

[0542] "User viewing history: Cooking shows; Eye-tracking data: Long duration in the bottom right corner; Ad candidate: Cooking utensil ads. Generate the optimal visual and message."

[0543] This prompt allows the system to improve the user's viewing experience while also increasing the effectiveness of advertisements.

[0544] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0545] Step 1:

[0546] The server receives viewing history information from the terminal. This information includes metadata about the content the viewer has watched and is stored in a database to identify past content preferences. This provides the foundational data for analyzing the viewer's preferences.

[0547] Step 2:

[0548] The device collects eye-tracking data in real time while viewing and sends it to the server. This eye-tracking data is processed using OpenCV to analyze the position of the gaze and identify where the viewer is fixating on the screen. The eye-tracking data also includes time information for each point of fixation, and the eye movement pattern is also stored.

[0549] Step 3:

[0550] The server uses a generative AI model to generate ads optimized for the viewer, based on viewing history information and eye-tracking data. The AI ​​selects content based on specific interests and combines images and textual elements to create an ad with the most relevant message. This process analyzes the viewer's past behavior patterns and customizes the ad's creative elements based on the predicted results.

[0551] Step 4:

[0552] The server delivers the generated advertisements to the devices. During ad delivery, the timing and position of the ads are strategically selected to enhance the viewer's visual experience. The ads are presented in a visually prominent manner, and preparations are made to gauge the viewer's reaction to them.

[0553] Step 5:

[0554] The device displays advertisements to viewers and tracks eye-tracking data to record their visual responses. Viewer responses are quantified by measuring the amount of time they gaze at the ad and how often their eyes linger on that area. This allows for an understanding of how much attention the advertisement received.

[0555] Step 6:

[0556] Users can provide feedback on ads that interest them. For example, they can interact with the ads by pressing buttons to request more information. This user interaction is sent to the server via the device and used for measuring ad effectiveness and further optimizing ads. The server collects the user's interaction history and updates a database to use for future ad generation.

[0557] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0558] This invention combines an emotion engine with a system that delivers personalized advertisements to viewers, thereby evaluating the viewer's emotional state in real time and optimizing the advertisements. The emotion engine has the function of analyzing the viewer's facial expression data and identifying their emotional state. This makes it possible to provide content that is tailored to the viewer's psychological state.

[0559] Collection and analysis of emotional data

[0560] terminal

[0561] The device uses its camera to capture the viewer's facial expressions while they are watching an advertisement. This data is analyzed in real time, allowing for an immediate assessment of their emotional state.

[0562] The collected facial expression data is sent to an emotion engine, which analyzes whether the viewer is experiencing joy, surprise, anger, sadness, indifference, or some other emotional state.

[0563] server

[0564] The server receives emotion data sent from the device and optimizes the ad content based on the viewer's emotional state. For example, when a viewer expresses surprise, the ad is adjusted to appeal to that emotion.

[0565] Furthermore, the server combines this data with viewing history and eye-tracking data to achieve more advanced personalization.

[0566] Ad optimization and delivery

[0567] server

[0568] The server adaptively changes the content of personalized ads based on the results of sentiment analysis. For example, if a viewer is feeling happy, it switches to an ad that contains a more positive message.

[0569] Ad delivery schedules are also optimized by taking emotional data into consideration. For example, ads are delivered at times when viewers are in a heightened mood.

[0570] terminal

[0571] The device delivers optimized advertisements received from the server to the viewer in real time. The advertisement display adaptively changes in response to the viewer's reaction.

[0572] User

[0573] Users can express emotional reactions to the ads they receive, and these reactions are collected by the system and used to improve future ads.

[0574] A concrete example of this system would be for the emotion engine to detect the viewer's excitement while they are watching a movie trailer, and then display advertisements for movie-related merchandise as related products. This method can maximize the viewer's interest.

[0575] The following describes the processing flow.

[0576] Step 1:

[0577] The device uses its built-in camera to capture facial expression data from viewers while they are watching advertisements or content. This makes it possible to monitor viewers' emotions in real time.

[0578] Step 2:

[0579] The device sends captured facial data to an emotion engine, which analyzes the viewer's emotional state. The emotion engine detects subtle changes in the viewer's face and classifies them into multiple emotional states, such as joy, surprise, anger, sadness, and indifference.

[0580] Step 3:

[0581] The server receives sentiment analysis results sent from the terminal and optimizes the advertisement according to the viewer's emotional state. For example, if the viewer is feeling surprised, the advertisement will be changed to one with more exciting content.

[0582] Step 4:

[0583] The server combines viewer emotional states, viewing history data, and eye-tracking data to generate personalized advertising content. This process also takes into account the viewer's past behavioral patterns.

[0584] Step 5:

[0585] The server determines the optimal time, order, and frequency for delivering the generated advertising content, and then sends that delivery schedule to the terminal.

[0586] Step 6:

[0587] The device delivers personalized advertisements received from the server to viewers based on a specified schedule. Even while the advertisement is playing, the device continuously collects the viewer's facial expression data and sends changes in emotion to the emotion engine in real time.

[0588] Step 7:

[0589] Users can express various emotional reactions to advertisements, and this feedback is used to improve future ad content and delivery strategies. The server analyzes this feedback to evaluate the effectiveness of the advertising campaign.

[0590] (Example 2)

[0591] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0592] Traditional advertising delivery systems struggled to consider viewers' interests and preferences, making it difficult to deliver the most relevant advertisements. Furthermore, the lack of mechanisms to understand viewers' emotional states in real time and deliver advertisements accordingly made it challenging to maximize the effectiveness of advertising.

[0593] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0594] In this invention, the server includes means for analyzing facial expression data to identify the viewer's emotional state, means for optimizing advertising content based on viewing history information and eye-tracking information, and means for generating personalized advertisements and delivering them to the viewer's device in real time. This enables advertising delivery that takes into account the viewer's emotional response.

[0595] "Viewer emotional state" refers to the emotional reactions, such as joy, surprise, and anger, that viewers exhibit when watching advertisements.

[0596] "Facial expression data" refers to image information obtained from the viewer's face, which is used to analyze their emotional state.

[0597] "Viewing history information" refers to a record of content that a viewer has watched in the past, and is used to identify viewing trends and preferences.

[0598] "Eye-tracking information" refers to data that records the movement of a viewer's gaze when they focus on a specific object, and is used to measure visual interest.

[0599] "Personalized advertising" refers to advertising content that is optimized based on each viewer's emotional state, viewing history, and eye-tracking information.

[0600] "Real-time delivery" refers to the immediate transmission and playback of advertising content based on the viewer's current state.

[0601] The invention will be implemented as an advertising delivery system based on viewer sentiment data. The details are described below.

[0602] 1. Hardware and software configuration

[0603] The device uses its built-in camera to capture the viewer's facial expressions while they are watching an advertisement. This captured data is processed in real time through image analysis software to extract facial feature points, which are then input into an emotion engine.

[0604] The server receives emotional characteristic data sent from the terminal and uses a generative AI model to analyze the viewer's emotional state (joy, surprise, anger, etc.). Using this analysis result along with past viewing history and eye-tracking information, the server provides an algorithm to select the most suitable advertisement.

[0605] Users receive personalized advertisements through their devices. Content messages tailored to the user's interests and emotional state are delivered in real time, and emotional responses to the advertisements are continuously collected by the device.

[0606] 2. Specific Examples

[0607] For example, if a user is watching a movie trailer and shows an excited expression in response to what is displayed, the device sends this data to the server. Based on this information, the server can select advertisements for movie-related posters and promotional items and deliver them to the user through the device. In this way, the advertisements are optimized to attract the user's interest.

[0608] 3. Example of a prompt statement

[0609] An example of a prompt might be, "Analyze what kind of advertising content is most effective based on the viewer's emotions of delight, and suggest the optimal advertisement." Using this prompt, it is expected that the generative AI model will suggest effective advertising content.

[0610] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0611] Step 1:

[0612] The device captures the viewer's facial expressions using its built-in camera while they are watching an advertisement. The input is image data obtained from the camera. Image analysis software is executed to extract feature points of facial expressions from this image data, and the output is generated as facial feature point data. Specifically, the camera is moved at a constant frame rate to continuously acquire images.

[0613] Step 2:

[0614] The device sends extracted facial feature point data to the emotion engine. The input is feature point data, and based on this data, a generative AI model analyzes the viewer's emotional state. The output is data indicating the viewer's emotional state. Specifically, it makes real-time judgments about emotions (e.g., joy, surprise, anger) and updates the data whenever there are changes.

[0615] Step 3:

[0616] The server receives emotional state data from the device and performs analysis to generate personalized advertisements. The input consists of emotional state data and other collected data (viewing history and eye-tracking information). The server processes this data through an AI model to select the most suitable advertisement. The output is a personalized advertising plan. Specifically, it uses emotional data as feedback to sequentially select the most effective advertisement for each emotional state.

[0617] Step 4:

[0618] The server remotely sends ads to devices based on the generated ad plan, and the devices display the ads. The input is optimized ad content, and the output is the ad visuals and audio delivered to the viewer. Specifically, the content is played at the appropriate timing and sequence so that viewers can see the ads in real time.

[0619] Step 5:

[0620] As users view advertisements, the device continuously monitors their reactions and collects data. The input consists of further facial expressions and responses from the user. This new data is then sent back to the emotion engine, which generates output that serves as reference material for optimizing future advertisements. Specific gestures and vocal responses are also incorporated into the data and used to improve the overall advertising effectiveness of the system.

[0621] (Application Example 2)

[0622] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0623] In traditional advertising methods, which do not consider the emotional state of viewers, effective personalization is difficult, and it is difficult to capture viewers' interest. It is necessary to optimize advertisements according to the emotional state of viewers to achieve more effective advertising.

[0624] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0625] In this invention, the server includes means for analyzing the emotional state of the viewer, means for generating personalized advertisements, and means for delivering advertisements to the viewer's information device. This enables optimal advertisement delivery that responds to the viewer's emotions.

[0626] "Viewer" refers to a person who watches an advertisement or content.

[0627] "Content preference" refers to the types and characteristics of content that viewers tend to watch in the past or prefer to watch.

[0628] "Viewing history information" refers to information about content that viewers have watched in the past, and includes data such as the number of views and viewing time.

[0629] "Eye-tracking data" refers to data about viewers' eye movements and points of focus, and is used to measure visual interest.

[0630] "Facial expression data" refers to data based on the facial expressions of viewers and is used for emotion analysis.

[0631] "Emotional state" refers to the psychological or emotional state of the viewer and is categorized into states such as joy, surprise, anger, and sadness.

[0632] "Personalized advertising" refers to advertisements that are customized to the individual preferences and emotional state of the viewer.

[0633] "Information devices" are devices used to display advertisements and content, and include smart glasses and mobile terminals.

[0634] "Optimization" is the process of adjusting conditions and parameters in order to achieve the greatest possible effect on a given goal.

[0635] In this embodiment, a system is constructed that optimizes advertisements based on the emotional state of the viewer and delivers personalized advertisements.

[0636] The server identifies content preferences by collecting viewers' viewing history and eye-tracking data. In addition, a camera on the device captures facial expression data, and an emotion engine analyzes the viewer's emotional state in real time. By using facial recognition APIs and machine learning models (e.g., TensorFlow and OpenCV) for emotion analysis, it is possible to evaluate the viewer's psychological response in detail.

[0637] The server then uses this data to generate advertisements tailored to the viewer's current emotional state. For example, if a viewer expresses surprise, an advertisement containing special offers for related products or services will be generated. This advertisement content is delivered to the viewer's information device in real time, allowing information to be provided at a time that matches the viewer's emotions.

[0638] A concrete example is real-time product promotion. When viewers show excitement while watching a demonstration of a new technology product, an advertisement for a limited-time discount on that product is immediately displayed. This system can capture viewers' interest and directly stimulate their desire to purchase.

[0639] By using generative AI models, the ad optimization process can be made even more sophisticated. An example of a necessary prompt would be, "How can I optimize relevant ads when a user shows a surprised expression?" This prompt allows the generative AI model to suggest the most suitable ad content, providing a more personalized advertising experience.

[0640] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0641] Step 1:

[0642] The device captures the viewer's facial expressions using a camera while they are watching content. The input is camera footage, which is output as facial expression data. This data is sent to a server in real time, ready for emotion analysis.

[0643] Step 2:

[0644] The server analyzes the received facial expression data to identify the viewer's emotional state. The input is facial expression data, and by analyzing this data using an emotion engine, it generates specific emotional states such as joy, surprise, and anger as output. Emotion recognition methods using TensorFlow or OpenCV are employed in this process.

[0645] Step 3:

[0646] The server combines the viewer's viewing history information and eye-tracking data with the emotional state obtained in step 2. The inputs include viewing history information, eye-tracking data, and emotional state, and data processing is performed based on this to determine the conditions for generating personalized ads. The output consists of the metrics and requirements necessary for ad generation.

[0647] Step 4:

[0648] The server generates appropriate ad content based on the viewer's current emotional state and viewing history. A generative AI model is used to design more personalized ads. The input consists of metrics and requirements for ad generation, and the generative AI model outputs specific ad content.

[0649] Step 5:

[0650] The server delivers the advertisement generated in step 4 to the viewer's information device, which is the terminal. The advertisement content is the input, and it is output to the terminal in real time, presenting the advertisement at a time appropriate for the viewer.

[0651] Step 6:

[0652] Users exhibit emotional reactions to the delivered advertisements. The device captures these reactions again and sends them back to the server. The input is facial expression data of the user's reaction to the advertisement, and the output is stored in the system as feedback for optimizing future advertisements.

[0653] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0654] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0655] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0656] [Fourth Embodiment]

[0657] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0658] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0659] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0660] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0661] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0662] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0663] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0664] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0665] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0666] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0667] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0668] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0669] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0670] This invention relates to a system for delivering advertisements optimized for viewers, and primarily concerns the generation and delivery of personalized advertisements using viewing history information and eye-tracking data.

[0671] Data collection and analysis

[0672] terminal

[0673] The device records information about the program the viewer is watching (program name, start time, end time) in real time. It also acquires viewer eye-tracking data through its built-in camera and sensors, determining which part of the screen the viewer is looking at.

[0674] The device encrypts the collected viewing history data and eye-tracking data and securely transmits it to the server.

[0675] server

[0676] The server stores the received viewing history data and eye-tracking data in a database. Based on this data, it runs a machine learning model to analyze viewers' content preferences and interests.

[0677] The server identifies viewer behavior patterns by comparing them with past similar viewer data. This allows it to predict the types of content that will interest viewers.

[0678] Ad generation and delivery

[0679] server

[0680] The server generates ad creatives based on the viewer's preferences and areas of interest. The generated ads are customized so that the product images and messages are most relevant to the viewer.

[0681] The server calculates the optimal time, order, and frequency for delivering the generated ads and sends the schedule for ad delivery to the devices.

[0682] terminal

[0683] The device delivers personalized advertisements to viewers according to a specified schedule. During this process, eye-tracking data is collected again to record viewers' reactions to the advertisements.

[0684] User

[0685] If a user is interested in an ad they have seen, they can request additional information using a remote control or similar device. This request information is sent to the server as feedback and used to measure the effectiveness of the ad.

[0686] In this way, advertising effectiveness can be maximized by efficiently delivering personalized advertisements to viewers. For example, users who enjoy watching cooking shows can be shown advertisements for the latest cooking appliances at the appropriate time. These advertisements include visual designs and messages that are likely to attract viewers' attention, thereby increasing their interest.

[0687] The following describes the processing flow.

[0688] Step 1:

[0689] The device records information about the program the viewer is watching, including the program name, start time, and end time. Furthermore, the device uses a built-in camera and eye-tracking sensor to acquire data on the viewer's eye movements while they are watching.

[0690] Step 2:

[0691] The device encrypts the collected viewing history data and eye-tracking data and sends it to the server. The transmitted data is used for analysis while protecting the viewer's privacy.

[0692] Step 3:

[0693] The server stores the received viewing data in a database and builds viewer profiles. Using machine learning algorithms, it analyzes viewers' content preferences based on their viewing history and identifies their preferred trends.

[0694] Step 4:

[0695] The server identifies points that viewers are focusing on during an advertisement through the analysis of eye-tracking data. This allows for the evaluation of the effectiveness of advertising elements and the use of this information to improve future advertising.

[0696] Step 5:

[0697] The server generates optimized ads based on analysis of viewer preferences and eye movements. It creates ad creatives that include images and messages of products most relevant to the viewer.

[0698] Step 6:

[0699] The server determines the delivery schedule for the generated ads, deciding when, in what order, and how often the ads will be delivered to viewers. This information is then sent to the terminal.

[0700] Step 7:

[0701] The device delivers optimized advertisements received from the server to viewers according to a specified schedule. Eye-tracking data is also collected during viewing to record viewers' reactions to the advertisements.

[0702] Step 8:

[0703] Users can take interactive actions if they are interested in the advertisement they have seen. For example, they can request additional information using a remote control. This information is then provided to the server as feedback via the device.

[0704] Step 9:

[0705] The server uses the feedback data to evaluate the effectiveness of advertising campaigns and utilizes it to improve future advertising delivery strategies.

[0706] (Example 1)

[0707] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0708] In today's information-saturated society, viewers are often shown many irrelevant advertisements, leading to decreased advertising effectiveness. To address this challenge, it is necessary to make the most of viewers' viewing history and eye-tracking data to provide them with the most relevant content. However, conventional systems are still insufficient in accurately analyzing viewers' specific preferences and points of focus, and in effectively generating and delivering personalized advertisements.

[0709] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0710] In this invention, the server includes means for collecting data on viewing history in order to record viewer information, means for measuring data on eye movements in order to obtain the viewer's gaze, and means for predicting viewer trends using machine learning techniques based on the viewing history. This makes it possible to generate and deliver advertisements optimized for each viewer and maximize advertising effectiveness.

[0711] "Viewing history data" refers to records of content that viewers have watched in the past, including data such as program title, viewing time, and viewing time slot.

[0712] "Eye-gaze data" refers to data that records the movement and points of focus of viewers' eyes, and is used to analyze which parts of the screen their gaze is concentrated on.

[0713] "Machine learning technology" is a technique in which computer systems learn from past data, automatically identify patterns and trends, and make predictions.

[0714] "Predicting viewer trends" is a process that analyzes viewers' responses to preferred content and advertisements based on viewing history data and eye-tracking data, and predicts their future viewing behavior and interests.

[0715] "Personalized advertising" refers to advertisements that are customized based on the individual viewer's preferences and past viewing patterns, and are characterized by content that is more relevant and interesting to the viewer.

[0716] This invention is a system for delivering personalized advertisements according to the viewer's preferences. The system mainly consists of the interaction between terminals, servers, and users.

[0717] The device collects detailed information about the content the viewer is watching in real time. This collection includes viewing history information and gaze data. The device has built-in cameras and sensors to track the viewer's gaze, and through these devices, it obtains information about which parts of the screen the viewer is focusing on. The collected data is encrypted and securely transmitted to the server.

[0718] The server stores received viewing history data and eye-tracking data in a database. Using this stored data, machine learning models are run to analyze viewers' content preferences and behavioral patterns. This predicts the content and relevant advertisements that viewers will like. Generative AI models are used for this analysis, and based on the trends obtained, the content and delivery schedule of advertisements are formulated. Next, ad creatives that match the viewers' preferences are generated and prepared for delivery to devices in the most optimal way.

[0719] If a user is interested in an advertisement displayed on their device, they can request additional information via a remote control or other input device. This feedback information is sent to a server and used for further analysis and measuring the effectiveness of the advertisement.

[0720] As a concrete example, viewers who frequently watch cooking shows will be shown advertisements for new cooking utensils. These advertisements are composed of highly relevant images and messages to capture the viewer's interest. An example of a prompt could be: "Design the most effective advertisement based on the viewer's viewing history and eye-tracking data. What kind of cooking utensil advertisement would be appropriate for viewers who watch a lot of cooking shows?"

[0721] This system enables the efficient delivery of personalized ads tailored to viewers' interests and behaviors, thereby maximizing the effectiveness of advertising.

[0722] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0723] Step 1:

[0724] The device collects information about the content the viewer is watching in real time. Specifically, this includes data such as the time viewing started, the program name, and the time viewing ended. The device takes viewer viewing trends as input and outputs viewing history data based on this. The device also acquires eye-tracking data using its built-in camera and sensors, recording where and to what extent the viewer's gaze is directed. This is the output of eye-tracking data.

[0725] Step 2:

[0726] The device encrypts the collected viewing history data and eye-tracking data. The input for this process is raw data, which is securely processed by an encryption algorithm and output as encrypted data. The encrypted data is then transmitted to the server via the network.

[0727] Step 3:

[0728] The server decrypts the received encrypted data and stores the viewing history data and gaze data in the database. The input for this step includes the encrypted data sent from the terminal, and the output is the decrypted viewing history and gaze data.

[0729] Step 4:

[0730] The server runs a machine learning model using stored viewing history data and eye-tracking data. The input is viewer history and eye-tracking data, which is used to analyze viewer preferences and output data predicting behavioral patterns. The generative AI model uses this analysis to identify patterns and determine what kind of advertisement would be appropriate next.

[0731] Step 5:

[0732] The server generates ad creatives optimized for the viewer based on analysis results from machine learning models. The input includes viewer preference data, which is used to output personalized ads. The ad creatives are customized with images and messages designed to capture the viewer's interest.

[0733] Step 6:

[0734] The server considers the optimal delivery schedule for the generated ads and sends instructions to the terminal. In this step, the server calculates the time of day, order, and frequency of ad delivery, and outputs an optimized delivery schedule based on post-ad generation data and user viewing timing.

[0735] Step 7:

[0736] The device displays advertisements on the viewer's screen at predetermined times according to a schedule received from the server. The input is schedule data from the server, and the output is that the advertisement is displayed to the viewer based on that schedule. Eye-tracking data is collected again during the display to record the viewer's reaction to the advertisement.

[0737] Step 8:

[0738] If a user is interested in a displayed advertisement, they can request additional information via a remote control or similar device. In this step, the user's action is input, and the output is that feedback information is sent to the server. The server uses this feedback to further improve the advertisement.

[0739] (Application Example 1)

[0740] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0741] Traditional advertising delivery systems suffered from problems such as low accuracy in tailoring ads to individual viewers' interests, resulting in limited ad visibility and effectiveness. Furthermore, there was the challenge of rapidly optimizing ads to reflect viewers' real-time visual responses.

[0742] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0743] In this invention, the server includes means for collecting viewing history information to identify the viewer's content preferences, means for analyzing eye-tracking data to identify the viewer's gaze points, means for generating advertisements optimized for the viewer, means for visually highlighting and displaying selected advertisements using the viewer's head-mounted display, and means for measuring visual responses to understand the viewer's interest in the advertisements. This enables highly accurate ad delivery tailored to the viewer's interests and effective real-time ad adjustment.

[0744] "Viewing history information" refers to data that shows information about content and media that viewers have accessed in the past, and is used to analyze viewers' preferences.

[0745] "Eye-tracking data" refers to data that records the movement and focus of a viewer's gaze, and is used to understand which parts of the screen the viewer is paying attention to.

[0746] "Optimized advertising" refers to advertising content that is tailored to the interests and concerns of viewers based on their viewing history and eye-tracking data.

[0747] A "head-mounted display" is a display device worn on the head to provide visual information, and is a device designed to make it easier for users to receive information visually.

[0748] "Visual response" refers to the changes in a viewer's gaze and attention to the content displayed, and is an indicator used to measure the viewer's level of interest and attention.

[0749] This invention provides a system for efficiently delivering personalized advertisements to viewers. The system mainly consists of a server, a terminal which is the viewer's device, and a user.

[0750] The server collects viewers' viewing history information and identifies their content preferences. This includes data related to content and media the viewer has previously accessed. The server also analyzes eye-tracking data transmitted from the device to identify the viewer's gaze points. This eye-tracking data is acquired in real time using eye-tracking technologies such as OpenCV.

[0751] Furthermore, the server generates advertisements optimized for the viewer based on viewing history information and eye-tracking data. This process uses TensorFlow to generate ad content tailored to the viewer's preferences, creating personalized advertisements. The generated advertisements are delivered to the viewer's head-mounted display, such as smart glasses, and displayed with visual emphasis.

[0752] The device displays advertisements sent from the server to the viewer at specified times. This allows viewers to see advertisements tailored to their interests in real time. Visual responses during viewing are measured using eye-tracking data and used to evaluate the user's level of interest.

[0753] For example, when a viewer watches a cooking show, if their gaze lingers on a particular recipe for an extended period, detailed advertisements for cooking utensils related to that recipe will be displayed. This ad display is generated based on the user's past viewing history and attention patterns.

[0754] Examples of prompts generated using AI models include the following:

[0755] "User viewing history: Cooking shows; Eye-tracking data: Long duration in the bottom right corner; Ad candidate: Cooking utensil ads. Generate the optimal visual and message."

[0756] This prompt allows the system to improve the user's viewing experience while also increasing the effectiveness of advertisements.

[0757] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0758] Step 1:

[0759] The server receives viewing history information from the terminal. This information includes metadata about the content the viewer has watched and is stored in a database to identify past content preferences. This provides the foundational data for analyzing the viewer's preferences.

[0760] Step 2:

[0761] The device collects eye-tracking data in real time while viewing and sends it to the server. This eye-tracking data is processed using OpenCV to analyze the position of the gaze and identify where the viewer is fixating on the screen. The eye-tracking data also includes time information for each point of fixation, and the eye movement pattern is also stored.

[0762] Step 3:

[0763] The server uses a generative AI model to generate ads optimized for the viewer, based on viewing history information and eye-tracking data. The AI ​​selects content based on specific interests and combines images and textual elements to create an ad with the most relevant message. This process analyzes the viewer's past behavior patterns and customizes the ad's creative elements based on the predicted results.

[0764] Step 4:

[0765] The server delivers the generated advertisements to the devices. During ad delivery, the timing and position of the ads are strategically selected to enhance the viewer's visual experience. The ads are presented in a visually prominent manner, and preparations are made to gauge the viewer's reaction to them.

[0766] Step 5:

[0767] The device displays advertisements to viewers and tracks eye-tracking data to record their visual responses. Viewer responses are quantified by measuring the amount of time they gaze at the ad and how often their eyes linger on that area. This allows for an understanding of how much attention the advertisement received.

[0768] Step 6:

[0769] Users can provide feedback on ads that interest them. For example, they can interact with the ads by pressing buttons to request more information. This user interaction is sent to the server via the device and used for measuring ad effectiveness and further optimizing ads. The server collects the user's interaction history and updates a database to use for future ad generation.

[0770] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0771] This invention combines an emotion engine with a system that delivers personalized advertisements to viewers, thereby evaluating the viewer's emotional state in real time and optimizing the advertisements. The emotion engine has the function of analyzing the viewer's facial expression data and identifying their emotional state. This makes it possible to provide content that is tailored to the viewer's psychological state.

[0772] Collection and analysis of emotional data

[0773] terminal

[0774] The device uses its camera to capture the viewer's facial expressions while they are watching an advertisement. This data is analyzed in real time, allowing for an immediate assessment of their emotional state.

[0775] The collected facial expression data is sent to an emotion engine, which analyzes whether the viewer is experiencing joy, surprise, anger, sadness, indifference, or some other emotional state.

[0776] server

[0777] The server receives emotion data sent from the device and optimizes the ad content based on the viewer's emotional state. For example, when a viewer expresses surprise, the ad is adjusted to appeal to that emotion.

[0778] Furthermore, the server combines this data with viewing history and eye-tracking data to achieve more advanced personalization.

[0779] Ad optimization and delivery

[0780] server

[0781] The server adaptively changes the content of personalized ads based on the results of sentiment analysis. For example, if a viewer is feeling happy, it switches to an ad that contains a more positive message.

[0782] Ad delivery schedules are also optimized by taking emotional data into consideration. For example, ads are delivered at times when viewers are in a heightened mood.

[0783] terminal

[0784] The device delivers optimized advertisements received from the server to the viewer in real time. The advertisement display adaptively changes in response to the viewer's reaction.

[0785] User

[0786] Users can express emotional reactions to the ads they receive, and these reactions are collected by the system and used to improve future ads.

[0787] A concrete example of this system would be for the emotion engine to detect the viewer's excitement while they are watching a movie trailer, and then display advertisements for movie-related merchandise as related products. This method can maximize the viewer's interest.

[0788] The following describes the processing flow.

[0789] Step 1:

[0790] The device uses its built-in camera to capture facial expression data from viewers while they are watching advertisements or content. This makes it possible to monitor viewers' emotions in real time.

[0791] Step 2:

[0792] The device sends captured facial data to an emotion engine, which analyzes the viewer's emotional state. The emotion engine detects subtle changes in the viewer's face and classifies them into multiple emotional states, such as joy, surprise, anger, sadness, and indifference.

[0793] Step 3:

[0794] The server receives sentiment analysis results sent from the terminal and optimizes the advertisement according to the viewer's emotional state. For example, if the viewer is feeling surprised, the advertisement will be changed to one with more exciting content.

[0795] Step 4:

[0796] The server combines viewer emotional states, viewing history data, and eye-tracking data to generate personalized advertising content. This process also takes into account the viewer's past behavioral patterns.

[0797] Step 5:

[0798] The server determines the optimal time, order, and frequency for delivering the generated advertising content, and then sends that delivery schedule to the terminal.

[0799] Step 6:

[0800] The device delivers personalized advertisements received from the server to viewers based on a specified schedule. Even while the advertisement is playing, the device continuously collects the viewer's facial expression data and sends changes in emotion to the emotion engine in real time.

[0801] Step 7:

[0802] Users can express various emotional reactions to advertisements, and this feedback is used to improve future ad content and delivery strategies. The server analyzes this feedback to evaluate the effectiveness of the advertising campaign.

[0803] (Example 2)

[0804] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0805] Traditional advertising delivery systems struggled to consider viewers' interests and preferences, making it difficult to deliver the most relevant advertisements. Furthermore, the lack of mechanisms to understand viewers' emotional states in real time and deliver advertisements accordingly made it challenging to maximize the effectiveness of advertising.

[0806] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0807] In this invention, the server includes means for analyzing facial expression data to identify the viewer's emotional state, means for optimizing advertising content based on viewing history information and eye-tracking information, and means for generating personalized advertisements and delivering them to the viewer's device in real time. This enables advertising delivery that takes into account the viewer's emotional response.

[0808] "Viewer emotional state" refers to the emotional reactions, such as joy, surprise, and anger, that viewers exhibit when watching advertisements.

[0809] "Facial expression data" refers to image information obtained from the viewer's face, which is used to analyze their emotional state.

[0810] "Viewing history information" refers to a record of content that a viewer has watched in the past, and is used to identify viewing trends and preferences.

[0811] "Eye-tracking information" refers to data that records the movement of a viewer's gaze when they focus on a specific object, and is used to measure visual interest.

[0812] "Personalized advertising" refers to advertising content that is optimized based on each viewer's emotional state, viewing history, and eye-tracking information.

[0813] "Real-time delivery" refers to the immediate transmission and playback of advertising content based on the viewer's current state.

[0814] The invention will be implemented as an advertising delivery system based on viewer sentiment data. The details are described below.

[0815] 1. Hardware and software configuration

[0816] The device uses its built-in camera to capture the viewer's facial expressions while they are watching an advertisement. This captured data is processed in real time through image analysis software to extract facial feature points, which are then input into an emotion engine.

[0817] The server receives emotional characteristic data sent from the terminal and uses a generative AI model to analyze the viewer's emotional state (joy, surprise, anger, etc.). Using this analysis result along with past viewing history and eye-tracking information, the server provides an algorithm to select the most suitable advertisement.

[0818] Users receive personalized advertisements through their devices. Content messages tailored to the user's interests and emotional state are delivered in real time, and emotional responses to the advertisements are continuously collected by the device.

[0819] 2. Specific Examples

[0820] For example, if a user is watching a movie trailer and shows an excited expression in response to what is displayed, the device sends this data to the server. Based on this information, the server can select advertisements for movie-related posters and promotional items and deliver them to the user through the device. In this way, the advertisements are optimized to attract the user's interest.

[0821] 3. Example of a prompt statement

[0822] An example of a prompt might be, "Analyze what kind of advertising content is most effective based on the viewer's emotions of delight, and suggest the optimal advertisement." Using this prompt, it is expected that the generative AI model will suggest effective advertising content.

[0823] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0824] Step 1:

[0825] The device captures the viewer's facial expressions using its built-in camera while they are watching an advertisement. The input is image data obtained from the camera. Image analysis software is executed to extract feature points of facial expressions from this image data, and the output is generated as facial feature point data. Specifically, the camera is moved at a constant frame rate to continuously acquire images.

[0826] Step 2:

[0827] The device sends extracted facial feature point data to the emotion engine. The input is feature point data, and based on this data, a generative AI model analyzes the viewer's emotional state. The output is data indicating the viewer's emotional state. Specifically, it makes real-time judgments about emotions (e.g., joy, surprise, anger) and updates the data whenever there are changes.

[0828] Step 3:

[0829] The server receives emotional state data from the device and performs analysis to generate personalized advertisements. The input consists of emotional state data and other collected data (viewing history and eye-tracking information). The server processes this data through an AI model to select the most suitable advertisement. The output is a personalized advertising plan. Specifically, it uses emotional data as feedback to sequentially select the most effective advertisement for each emotional state.

[0830] Step 4:

[0831] The server remotely sends ads to devices based on the generated ad plan, and the devices display the ads. The input is optimized ad content, and the output is the ad visuals and audio delivered to the viewer. Specifically, the content is played at the appropriate timing and sequence so that viewers can see the ads in real time.

[0832] Step 5:

[0833] As users view advertisements, the device continuously monitors their reactions and collects data. The input consists of further facial expressions and responses from the user. This new data is then sent back to the emotion engine, which generates output that serves as reference material for optimizing future advertisements. Specific gestures and vocal responses are also incorporated into the data and used to improve the overall advertising effectiveness of the system.

[0834] (Application Example 2)

[0835] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0836] In traditional advertising methods, which do not consider the emotional state of viewers, effective personalization is difficult, and it is difficult to capture viewers' interest. It is necessary to optimize advertisements according to the emotional state of viewers to achieve more effective advertising.

[0837] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0838] In this invention, the server includes means for analyzing the emotional state of the viewer, means for generating personalized advertisements, and means for delivering advertisements to the viewer's information device. This enables optimal advertisement delivery that responds to the viewer's emotions.

[0839] "Viewer" refers to a person who watches an advertisement or content.

[0840] "Content preference" refers to the types and characteristics of content that viewers tend to watch in the past or prefer to watch.

[0841] "Viewing history information" refers to information about content that viewers have watched in the past, and includes data such as the number of views and viewing time.

[0842] "Eye-tracking data" refers to data about viewers' eye movements and points of focus, and is used to measure visual interest.

[0843] "Facial expression data" refers to data based on the facial expressions of viewers and is used for emotion analysis.

[0844] "Emotional state" refers to the psychological or emotional state of the viewer and is categorized into states such as joy, surprise, anger, and sadness.

[0845] "Personalized advertising" refers to advertisements that are customized to the individual preferences and emotional state of the viewer.

[0846] "Information devices" are devices used to display advertisements and content, and include smart glasses and mobile terminals.

[0847] "Optimization" is the process of adjusting conditions and parameters in order to achieve the greatest possible effect on a given goal.

[0848] In this embodiment, a system is constructed that optimizes advertisements based on the emotional state of the viewer and delivers personalized advertisements.

[0849] The server identifies content preferences by collecting viewers' viewing history and eye-tracking data. In addition, a camera on the device captures facial expression data, and an emotion engine analyzes the viewer's emotional state in real time. By using facial recognition APIs and machine learning models (e.g., TensorFlow and OpenCV) for emotion analysis, it is possible to evaluate the viewer's psychological response in detail.

[0850] The server then uses this data to generate advertisements tailored to the viewer's current emotional state. For example, if a viewer expresses surprise, an advertisement containing special offers for related products or services will be generated. This advertisement content is delivered to the viewer's information device in real time, allowing information to be provided at a time that matches the viewer's emotions.

[0851] A concrete example is real-time product promotion. When viewers show excitement while watching a demonstration of a new technology product, an advertisement for a limited-time discount on that product is immediately displayed. This system can capture viewers' interest and directly stimulate their desire to purchase.

[0852] By using generative AI models, the ad optimization process can be made even more sophisticated. An example of a necessary prompt would be, "How can I optimize relevant ads when a user shows a surprised expression?" This prompt allows the generative AI model to suggest the most suitable ad content, providing a more personalized advertising experience.

[0853] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0854] Step 1:

[0855] The device captures the viewer's facial expressions using a camera while they are watching content. The input is camera footage, which is output as facial expression data. This data is sent to a server in real time, ready for emotion analysis.

[0856] Step 2:

[0857] The server analyzes the received facial expression data to identify the viewer's emotional state. The input is facial expression data, and by analyzing this data using an emotion engine, it generates specific emotional states such as joy, surprise, and anger as output. Emotion recognition methods using TensorFlow or OpenCV are employed in this process.

[0858] Step 3:

[0859] The server combines the viewer's viewing history information and eye-tracking data with the emotional state obtained in step 2. The inputs include viewing history information, eye-tracking data, and emotional state, and data processing is performed based on this to determine the conditions for generating personalized ads. The output consists of the metrics and requirements necessary for ad generation.

[0860] Step 4:

[0861] The server generates appropriate ad content based on the viewer's current emotional state and viewing history. A generative AI model is used to design more personalized ads. The input consists of metrics and requirements for ad generation, and the generative AI model outputs specific ad content.

[0862] Step 5:

[0863] The server delivers the advertisement generated in step 4 to the viewer's information device, which is the terminal. The advertisement content is the input, and it is output to the terminal in real time, presenting the advertisement at a time appropriate for the viewer.

[0864] Step 6:

[0865] Users exhibit emotional reactions to the delivered advertisements. The device captures these reactions again and sends them back to the server. The input is facial expression data of the user's reaction to the advertisement, and the output is stored in the system as feedback for optimizing future advertisements.

[0866] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0867] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0868] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0869] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0870] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0871] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0872] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0873] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0874] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0875] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0876] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0877] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0878] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0879] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0880] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0881] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0882] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0883] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0884] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0885] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0886] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0887] The following is further disclosed regarding the embodiments described above.

[0888] (Claim 1)

[0889] In order to identify viewers' content preferences, means of collecting viewing history information,

[0890] To identify the points of viewer attention, a means of analyzing eye-tracking data is used,

[0891] A means for generating personalized advertisements for viewers based on the aforementioned viewing history information and eye-tracking data,

[0892] The means for delivering the personalized advertisement to the viewer's device,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, further comprising means for adjusting the timing, order, and frequency of advertisements in order to optimize advertisement delivery based on the aforementioned viewing history information and eye-tracking data.

[0896] (Claim 3)

[0897] The system according to claim 1, further comprising means for reacquiring eye-tracking data during ad delivery in order to measure viewer reactions.

[0898] "Example 1"

[0899] (Claim 1)

[0900] In order to record viewer information, means of collecting data related to viewing history,

[0901] In order to obtain the viewer's perspective, a means of measuring data related to eye movements,

[0902] A means for predicting viewer trends using machine learning technology based on the aforementioned viewing history,

[0903] A means of creating personalized advertising materials based on viewer preferences and perspectives,

[0904] Means for transmitting the personalized advertisement to the viewer's device at an appropriate time,

[0905] In order to evaluate the effectiveness of advertising, means of obtaining viewer feedback,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, which uses the aforementioned viewing history data and gaze data to adjust the ad delivery schedule and optimize delivery performance.

[0909] (Claim 3)

[0910] The system according to claim 1, which acquires eye-tracking data again during ad display in order to analyze in detail the viewer's response to the ad.

[0911] "Application Example 1"

[0912] (Claim 1)

[0913] In order to identify viewers' content preferences, means of collecting viewing history information,

[0914] To identify the points of viewer attention, a means of analyzing eye-tracking data is used,

[0915] A means for generating advertisements optimized for viewers based on the aforementioned viewing history information and eye-tracking data,

[0916] A means of visually highlighting selected advertisements using the viewer's head-mounted display,

[0917] In order to understand viewers' interest in advertising, a means of measuring visual responses,

[0918] A system that includes this.

[0919] (Claim 2)

[0920] The system according to claim 1, further comprising adjustments using a generative AI model for optimizing advertising content in real time based on the aforementioned visual response.

[0921] (Claim 3)

[0922] The system according to claim 1, further comprising means for dynamically changing the placement and display time of advertisements on a viewer's display based on the eye-tracking data and visual responses.

[0923] "Example 2 of combining an emotion engine"

[0924] (Claim 1)

[0925] To identify the emotional state of viewers, a means of analyzing facial expression data,

[0926] A means of optimizing ad content based on viewing history information and eye-tracking information,

[0927] A means of generating personalized advertisements and delivering them to viewers' devices in real time,

[0928] A means of monitoring viewers' emotional reactions and collecting data to improve the content of future advertisements,

[0929] A system that includes this.

[0930] (Claim 2)

[0931] The system according to claim 1, further comprising means for adaptively changing the advertising message in response to an emotional state.

[0932] (Claim 3)

[0933] The system according to claim 1, further comprising means for optimizing the ad delivery schedule based on the emotional state of the audience.

[0934] "Application example 2 when combining with an emotional engine"

[0935] (Claim 1)

[0936] In order to identify viewers' content preferences, means of collecting viewing history information,

[0937] To identify the points of viewer attention, a means of analyzing eye-tracking data is used,

[0938] A means for analyzing the emotional state of the viewer based on the aforementioned viewing history information, eye-tracking data, and facial expression data,

[0939] A means of generating personalized advertisements in response to the viewer's emotions,

[0940] The means for delivering the personalized advertisement to the viewer's information device,

[0941] A system that includes this.

[0942] (Claim 2)

[0943] The system according to claim 1, further comprising means for adjusting the timing, order, and frequency of advertisements in order to optimize advertisement delivery based on the aforementioned viewing history information, eye-tracking data, and emotional state.

[0944] (Claim 3)

[0945] The system according to claim 1, further comprising means for reacquiring eye-tracking data and facial expression data during ad delivery in order to accurately measure viewer reactions. [Explanation of symbols]

[0946] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. In order to identify viewers' content preferences, means of collecting viewing history information, To identify the points of viewer attention, a means of analyzing eye-tracking data is used, A means for generating personalized advertisements for viewers based on the aforementioned viewing history information and eye-tracking data, The means for delivering the personalized advertisement to the viewer's device, A system that includes this.

2. The system according to claim 1, further comprising means for adjusting the timing, order, and frequency of advertisements in order to optimize advertisement delivery based on the aforementioned viewing history information and eye-tracking data.

3. The system according to claim 1, further comprising means for reacquiring eye-tracking data during ad delivery in order to measure viewer reactions.

Images