System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting viewer data from a terminal; means for analyzing the viewer data and estimating age, sex, and emotion; means for selecting a suitable advertisement based on the analyzed viewer data; means for automatically generating an advertisement based on the selected advertisement material; means for transmitting the automatically generated advertisement to the terminal; means for displaying the advertisement on the terminal; and means for analyzing advertising effectiveness based on the collected reaction data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional advertising systems are limited to providing one-way information, making it difficult to tailor advertisements to reflect the viewer's interests. This results in insufficient advertising effectiveness and a decline in ROI (return on investment). Furthermore, there is a lack of methods for delivering personalized advertisements that can respond to the diverse interests of viewers. The purpose of this invention is to solve these problems and maximize advertising effectiveness by providing optimal advertisements tailored to the viewer in real time. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes the following means: means for collecting viewer data from a terminal; means for analyzing the viewer data and estimating age, gender, and emotion; means for selecting appropriate advertisements based on the analyzed viewer data; means for automatically generating advertisements based on the selected advertising materials; means for transmitting the automatically generated advertisements to the terminals; means for displaying the advertisements on the terminals; means for collecting viewer responses while the advertisements are displayed; and means for analyzing advertising effectiveness based on the collected response data. The system also includes means for receiving the viewer data transmitted from the terminals and identifying repeat customers based on the data, and means for updating an advertising model based on the analysis of advertising effectiveness and reflecting the results in the next advertisement delivery. In this way, advertisements optimized for the viewer can be provided in real time, maximizing advertising effectiveness.

[0006] A "terminal" is a device for collecting viewer data and displaying advertising content.

[0007] "Viewer data" refers to data used to analyze advertising effectiveness, including the viewer's age, gender, emotions, gaze, etc.

[0008] "Analysis" is the act of inferring relevant attributes based on collected viewer data.

[0009] "Age" is data indicating the viewer's date of birth or estimated age group.

[0010] "Gender" is data indicating the gender of the viewer.

[0011] "Emotion" is data that indicates the emotional state of the viewer as estimated from their facial expressions and other physiological responses.

[0012] "Advertising materials" are content elements such as images, text, and videos for advertising.

[0013] "Automatic generation" is the process of using AI and algorithms to create advertising content based on collected and analyzed data.

[0014] "Advertising content" refers to the specific advertising content displayed to the viewer.

[0015] "Response data" refers to data such as the viewer's behavior and facial expressions while the advertisement is being displayed.

[0016] "Advertising effectiveness" is a measure of the success and influence of an advertisement, evaluated based on viewer response.

[0017] An "advertising model" is an optimization algorithm for advertising display that is generated based on viewer data and response data.

[0018] "Repeat viewers" is a term used to describe viewers who have previously viewed the same ad. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention relates to a system for displaying advertisements optimized for viewers in real time. The program processing of the entire system will be explained below in natural language.

[0041] Collection and analysis of user data

[0042] Subject: Device

[0043] The device uses a camera to collect video data of the viewer, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[0044] Subject: Server

[0045] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses a facial recognition model to analyze the image data and identify attributes such as "female in her 20s" or "smiling." The server also references data collected in the past to determine whether the viewer is the same person. This also makes it possible to identify repeat viewers.

[0046] Ad selection and automatic generation

[0047] Subject: Server

[0048] The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces." It then searches a database of advertising materials provided by the advertiser to create a list of the most suitable candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[0049] Ad display and effectiveness measurement

[0050] Subject: Device

[0051] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[0052] Subject: Server

[0053] While the advertisement is being displayed, the device collects viewer response data, which is then sent to a server to measure the effectiveness of the advertisement. The server then uses this data to analyze the success and impact of the advertisement and update the advertising model for further optimization.

[0054] Specific examples

[0055] Signage in a shopping mall

[0056] Subject: Server

[0057] The server collects and analyzes viewer data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[0058] Digital signage in public transport

[0059] Subject: Server

[0060] The server collects and analyzes viewer data from terminals installed on public transport station platforms. For example, if the analysis results indicate "men in their 30s," it selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response data is collected and the effectiveness of the advertisements is measured.

[0061] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] Subject: Device

[0065] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[0066] Step 2:

[0067] Subject: Device

[0068] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[0069] Step 3:

[0070] Subject: Device

[0071] The terminal transmits the detected and tracked face data to the server.

[0072] Step 4:

[0073] Subject: Server

[0074] The server receives the face data sent from the terminal.

[0075] Step 5:

[0076] Subject: Server

[0077] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[0078] Step 6:

[0079] Subject: Server

[0080] The server references previously collected data and checks it against a database to identify whether the viewer is a repeat visitor.

[0081] Step 7:

[0082] Subject: Server

[0083] The server selects appropriate advertisements based on the analyzed viewer data.

[0084] Step 8:

[0085] Subject: Server

[0086] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[0087] Step 9:

[0088] Subject: Server

[0089] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[0090] Step 10:

[0091] Subject: Server

[0092] The server transmits the automatically generated advertising content to the terminal.

[0093] Step 11:

[0094] Subject: Device

[0095] The terminal receives the advertisement content transmitted from the server.

[0096] Step 12:

[0097] Subject: Device

[0098] The terminal displays the received advertising content on the display in real time, for example, an advertisement for a new fashion item.

[0099] Step 13:

[0100] Subject: Device

[0101] While the ad is being displayed, the device uses its camera to collect viewer response data, including eye tracking and facial expression analysis.

[0102] Step 14:

[0103] Subject: Device

[0104] The terminal transmits the collected viewer reaction data to the server.

[0105] Step 15:

[0106] Subject: Server

[0107] The server receives the response data sent from the terminal.

[0108] Step 16:

[0109] Subject: Server

[0110] The server analyzes the effectiveness of the advertisement based on the reaction data, specifically evaluating the length of time the viewer's gaze remains on the screen and changes in facial expression, and quantifies the effectiveness of the advertisement.

[0111] Step 17:

[0112] Subject: Server

[0113] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[0114] This allows viewers to receive ads that are optimized for them in real time, maximizing advertising effectiveness.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] Conventional advertising display systems have had difficulty effectively delivering advertisements to viewers. In particular, they have had issues with being unable to select the most appropriate advertisement based on the viewer's attributes and display it in real time. Furthermore, there are limitations to accurately measuring advertising effectiveness and reflecting this in future ad distribution. Furthermore, there are insufficient methods for collecting data to analyze how ads affect viewers.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes means for preprocessing viewer data, detecting and tracking face data, means for estimating the viewer's age, gender, and emotion based on the face data, and means for selecting optimal advertisements based on the estimated viewer attributes. This makes it possible to display advertisements optimized for the viewer in real time, accurately measure the effectiveness of the advertisements, and reflect this in the next distribution.

[0120] A "terminal" is a device installed to collect data from viewers and transmit the collected data to a server.

[0121] "Viewer data" refers to viewer video data and other related data collected by a terminal via a camera or the like.

[0122] "Preprocessing" refers to processing such as image resizing and noise removal that is performed on collected video data.

[0123] "Face data" refers to data of the viewer's face regions detected and tracked from the pre-processed video data.

[0124] "Means for estimating age, gender, and emotion of a viewer based on facial data" refers to the processes and models within the server that analyze received facial data and estimate the age, gender, and emotional attributes of the target viewer.

[0125] "Viewer demographics" are data that refer to specific characteristics of viewers, such as their age, gender, and emotions.

[0126] The "means for selecting an advertisement" refers to the mechanisms and algorithms within the server for selecting the most suitable advertisement based on the estimated viewer attributes.

[0127] "Advertising materials" are material data such as images, text, and videos used to generate advertisements.

[0128] "Advertising content" refers to advertising content that is automatically generated based on selected advertising materials.

[0129] "Response data" refers to data such as the viewer's gaze direction and changes in facial expression that are collected while the advertisement is being displayed.

[0130] "Means for analyzing advertising effectiveness" refers to the processes and algorithms within the server that analyze the effectiveness and influence of displayed advertisements based on collected response data.

[0131] The present invention relates to a system for displaying advertisements optimized for viewers in real time, which operates through multiple steps involving terminals, servers, and users.

[0132] 1. Collection of User Data

[0133] Subject: Device

[0134] The device collects viewer video data using a camera. The camera is a commonly used hardware device, such as a USB or built-in camera. The video data is pre-processed using an image processing library such as OpenCV, which results in the detection and tracking of viewer facial data. This facial data is then sent to the server in real time.

[0135] 2. Data Preprocessing and Analysis

[0136] Subject: Device

[0137] The device performs preprocessing on the collected video data. This preprocessing includes image resizing and noise reduction. It uses libraries such as OpenCV to adjust the image quality of the video data and detects the viewer's face using a face detection model (e.g., Haar Cascades or MTCNN). The device then extracts the detected face area and sends the data to the server.

[0138] 3. Estimation of viewer attributes

[0139] Subject: Server

[0140] The server receives the facial data sent from the device. Based on the received data, it uses a facial recognition model (e.g., Dlib or FaceNet) to estimate the viewer's age, gender, and emotion. For example, it identifies attributes such as "female in her 20s" or "smiling." In this process, it extracts features from the facial data and uses a pre-trained model to make an estimation.

[0141] 4. Ad selection and automatic generation

[0142] Subject: Server

[0143] The server selects the most suitable advertisement based on the analysis results. For example, it selects advertisements for fashion brands or cosmetics based on the attributes "women in their 20s" and "smiling face." The server searches an advertising material database (e.g., cloud storage service) and automatically generates advertising content using deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The advertising content is generated in formats such as HTML5 or MP4.

[0144] 5. Submission of advertising content

[0145] Subject: Server

[0146] The server sends the generated advertising content to the device using real-time communication technology such as WebSocket. The server identifies the device's IP address and sends the advertising data to the endpoint.

[0147] 6. Display of advertisements

[0148] Subject: Device

[0149] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[0150] 7. Collecting viewer response data

[0151] Subject: Device

[0152] While the ad is being displayed, the device uses a camera to collect viewer response data, capturing gaze direction and facial expression changes in real time for visual analysis, which is then sent back to the server.

[0153] 8. Analysis of advertising effectiveness and feedback

[0154] Subject: Server

[0155] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. For example, if the viewer's gaze is frequently directed at the advertisement while viewing it, the advertisement is deemed to be highly successful. Based on this data, the server evaluates the effectiveness and influence of the advertisement and updates the model to help with future advertisement selection.

[0156] Specific examples

[0157] Signage in a shopping mall

[0158] Terminals installed at the entrance to the shopping mall collect viewer data. If the analysis results indicate a "woman in her early twenties," the server selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. The generated advertisements are sent to the terminals and displayed on displays within the mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[0159] Digital signage in public transport

[0160] Terminals installed on public transport station platforms collect viewer data. For example, if the analysis results show "men in their 30s," the server selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. The generated advertisements are sent to the terminals and displayed on digital signage on the station platforms. Viewer response data is collected during display and sent to the server, where the advertising effectiveness is measured.

[0161] Prompt Sentence Examples

[0162] "Select an ad for a tech gadget for a man in his 30s and generate and display an ad for the latest smartphone."

[0163] keyword

[0164] Generative AI model, prompt sentence

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1:

[0167] Subject: Device

[0168] The device uses a camera to collect video data of the viewer. This video data is captured from the camera in real time. Specifically, it stores frame data continuously acquired from the camera in memory and performs image preprocessing using the OpenCV library. Preprocessing includes image resizing and noise removal. The input is raw video data, and the output is preprocessed video data.

[0169] Step 2:

[0170] Subject: Device

[0171] The device detects and tracks the viewer's facial data from the preprocessed video data. Specifically, it identifies the facial region using a face detection model such as OpenCV, Haar Cascades, or MTCNN. The input is the preprocessed video data, and the output is the viewer's facial region data. This data is sent to the server for further processing.

[0172] Step 3:

[0173] Subject: Server

[0174] The server receives the face region data sent from the device. Based on the received data, it uses a face recognition model such as Dlib or FaceNet to estimate the viewer's age, gender, and emotion. The input is the face region data, and the output is the estimated viewer attributes (e.g., "female in her 20s" and "smiling"). Specifically, it extracts features and analyzes them using a pre-trained model.

[0175] Step 4:

[0176] Subject: Server

[0177] The server selects the most suitable advertisement based on the viewer attributes obtained as a result of the analysis. As a specific example, it searches an advertising material database to select advertisements for fashion brands and cosmetics for the attributes "women in their 20s" and "smiling faces." The input is the estimated viewer attributes, and the output is the most suitable advertising material.

[0178] Step 5:

[0179] Subject: Server

[0180] The server automatically generates advertising content based on the selected advertising materials. This process uses deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The input is the selected advertising materials, and the output is the generated advertising content. Specifically, the server creates advertisements in HTML5 or MP4 format based on the material data.

[0181] Step 6:

[0182] Subject: Server

[0183] The server transmits the generated advertising content to the device using real-time communication technology such as WebSocket. The input is the generated advertising content, and the output is data transmission to the device.

[0184] Step 7:

[0185] Subject: Device

[0186] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, an advertisement for a new fashion item is displayed on a display device connected to the terminal. The input is the received advertising content, and the output is the display on which the advertisement is displayed.

[0187] Step 8:

[0188] Subject: Device

[0189] While the ad is being displayed, the device uses a camera to collect viewer response data. Specifically, it captures changes in gaze direction and facial expressions. The input is the displayed ad content, and the output is viewer response data. This data is then sent back to the server.

[0190] Step 9:

[0191] Subject: Server

[0192] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. Specifically, if the viewer's gaze is frequently directed at the advertisement, the success of the advertisement is evaluated highly. The input is the viewer response data, and the output is the analysis result of the advertisement's effectiveness. The server updates the advertising model based on this analysis result and reflects it in subsequent advertisement selections.

[0193] (Application example 1)

[0194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0195] Conventional advertising display systems did not fully utilize viewer information, making it difficult to select the optimal advertisement to maximize advertising effectiveness. Furthermore, they lacked the technology to collect viewer response information in real time and immediately reflect it in advertising strategies. As a result, advertisements that were not appealing to viewers were displayed, reducing advertising effectiveness.

[0196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0197] In this invention, the server includes means for collecting viewer data from the terminal, means for analyzing the viewer data and estimating age, gender, and emotion, means for selecting an appropriate advertisement based on the analyzed viewer data, means for automatically generating an advertisement based on the selected advertising material, means for transmitting the automatically generated advertisement to the terminal, means for displaying the advertisement on the terminal, means for collecting viewer responses while the advertisement is being displayed, means for analyzing the effectiveness of the advertisement based on the collected response data, means for collecting and analyzing video of the viewer using a camera mounted on the smart glasses, and means for displaying the advertisement on the display of the smart glasses. This enables real-time display of personalized advertisements tailored to the characteristics of the viewer and rapid feedback of the effectiveness of the advertisement.

[0198] "Terminal" refers to a device used to collect viewer data and display advertisements, including smartphones, tablets, smart glasses, etc.

[0199] "Viewer data" refers to data that includes information such as the viewer's age, gender, and emotions, and is collected through cameras and sensors.

[0200] "Analysis" is the process of estimating attributes such as age, gender, and emotions based on collected viewer data.

[0201] "Ad selection" is the process of selecting the most suitable advertisement based on analyzed viewer data.

[0202] "Advertising materials" are materials such as databases, images, videos, and text used to generate advertisements.

[0203] "Automatic generation" is the process of automatically creating advertising content based on advertising materials after an advertisement has been selected.

[0204] "Advertising effectiveness" is an indicator that evaluates the impact that an advertisement has on viewers, and is analyzed based on viewer response data.

[0205] "Smart glasses" are wearable devices equipped with a camera and a display that can collect video data from viewers and display advertisements.

[0206] A "camera" is a device that collects video data from viewers and is installed in smart glasses or terminals.

[0207] A "display" is a display device that displays the generated advertisement to the viewer, and is installed in a terminal or smart glasses.

[0208] "Response data" refers to data such as the viewer's gaze, facial expressions, and movements while the advertisement is being displayed.

[0209] This invention relates to a system for displaying advertisements optimized for viewers in real time and measuring their effectiveness. This system is composed of the following components:

[0210] System Configuration

[0211] 1. Hardware

[0212] Devices: Includes smartphones, tablets, smart glasses, etc.

[0213] Camera: A device installed in smart glasses or a device that collects video data of the viewer.

[0214] Display: A device mounted on the smart glasses or device that displays the generated advertisements.

[0215] 2. Software

[0216] Facial recognition technology: For example, OpenCV is used.

[0217] Data analysis platform: For example, AWS Lambda is used.

[0218] Ad selection algorithm: For example, TensorFlow is used.

[0219] System processing procedure

[0220] Collection and analysis of user data

[0221] The device collects the viewer's video data, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[0222] Ad selection and automatic generation

[0223] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotions. To do this, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling." The server then references previously collected data to determine whether the viewer is the same person. The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on the attributes "female in her 20s" and "smiling." It then searches a database of advertising materials provided by the advertiser and lists the most suitable advertising candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[0224] Ad display and effectiveness measurement

[0225] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be displayed on the display. While the advertisement is being displayed, the device collects viewer response data. The collected response data is sent to the server, where the effectiveness of the advertisement is measured. The server uses this data to analyze the success and influence of the advertisement and update the advertising model for further optimization.

[0226] Specific examples

[0227] Signage in a shopping mall

[0228] When a user is wearing smart glasses while walking through a shopping mall, the glasses collect the user's facial data in real time and send it to a server. Based on this data, the server determines that the user is a woman in her early twenties and selects advertisements for fashion brands and accessories. Advertisements for new collections are automatically generated and instantly displayed on the smart glasses' display. The user's reactions are also collected again, and the effectiveness of the advertisements is measured.

[0229] Digital signage in public transport

[0230] Terminals installed on public transport station platforms collect the viewer's facial data and send it to a server. If the user is identified as a "male in his 30s," ads for technology gadgets and sports equipment are selected, and the latest smartphone ads are automatically generated. These ads are then displayed on digital signage on the station platforms. Viewer response data is collected in the same way, and the effectiveness of the ads is measured.

[0231] Prompt Sentence Examples

[0232] "Woman in her 20s, smiling. Generate an ad for new fashion items. Also consider the option to show an ad for your summer accessories collection instead."

[0233] "Based on facial recognition data, we've identified her as a woman in her 20s. Then generate ads for a fashion brand that would be suitable for her. Also, create content that reflects positive emotions, taking into account smiling faces."

[0234] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[0235] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0236] Step 1:

[0237] User Data Collection

[0238] Input: Video data from a camera mounted on smart glasses

[0239] Processing: The device uses the camera to collect the user's video data. The video data is captured in real time and pre-processed (resolution adjustment, noise reduction).

[0240] Output: Pre-processed video data

[0241] Step 2:

[0242] Face Detection and Tracking

[0243] Input: Preprocessed video data

[0244] Processing: The device uses OpenCV to detect faces in the video data and perform face tracking. The face detection algorithm identifies the position of the face within the video frame.

[0245] Output: Face data (face position, facial feature points, etc.)

[0246] Step 3:

[0247] Data transmission

[0248] Input: Face data

[0249] Processing: The device sends the detected face data to the server in real time using the secure HTTP(S) protocol.

[0250] Output: Face data sent to the server

[0251] Step 4:

[0252] Face Recognition and Attribute Estimation

[0253] Input: Face data sent to the server

[0254] Processing: The server analyzes the facial data using facial recognition technology (e.g., a facial recognition model) to estimate age, gender, and emotion. Analysis includes extracting facial features and estimating attributes using a classifier.

[0255] Output: Viewer attribute data (e.g., "Female in her 20s," "Smiling," etc.)

[0256] Step 5:

[0257] Repeater Identification

[0258] Input: Viewer demographic data

[0259] Processing: The server refers to a database of previously collected data and compares it with the viewer's facial features to identify whether they are repeat visitors. For this purpose, it uses an approximate nearest neighbor search algorithm.

[0260] Output: Repeater identification result

[0261] Step 6:

[0262] Ad selection

[0263] Input: Viewer attribute data, repeater identification results

[0264] Processing: The server selects the optimal advertisement using an advertisement selection algorithm (for example, an algorithm using TensorFlow) based on the viewer attribute data and the results of repeat customer identification.

[0265] Output: Ad candidate list

[0266] Step 7:

[0267] Auto-generated ads

[0268] Input: Ad candidate list

[0269] Processing: The server automatically generates optimal advertising content from a selected list of ad candidates based on the advertising material database provided by the advertiser. For generation, a generative AI model is used.

[0270] Output: Generated ad content

[0271] Step 8:

[0272] Sending Ads

[0273] Input: Generated ad content

[0274] Processing: The server sends the generated advertising content to the device using secure HTTP(S) as the communication protocol.

[0275] Output: Ad content sent to the device

[0276] Step 9:

[0277] Displaying ads

[0278] Input: Ad content sent to the device

[0279] Processing: The device displays the received advertising content on the display in real time. The advertisement is displayed to the user using the display of the smart glasses.

[0280] Output: The ad shown to the user

[0281] Step 10:

[0282] Collecting viewer responses

[0283] Input: The ad shown to the user

[0284] Processing: The device collects user response data (eye gaze, facial expressions, movements, etc.) while the ad is being displayed. The device uses cameras and sensors to capture response data in real time.

[0285] Output: Collected reaction data

[0286] Step 11:

[0287] Sending reaction data

[0288] Input: Collected reaction data

[0289] Processing: The device sends the collected reaction data to the server using secure HTTP(S).

[0290] Output: Response data sent to the server

[0291] Step 12:

[0292] Measuring advertising effectiveness

[0293] Input: Reaction data sent to the server

[0294] Processing: The server analyzes the effectiveness of the ads based on the collected response data, using statistical analysis and machine learning models.

[0295] Output: Advertising effectiveness data

[0296] Step 13:

[0297] Updated advertising model

[0298] Input: Advertising effectiveness data

[0299] Processing: The server updates the advertising model based on the advertising effectiveness data. The updated model is reflected in the next ad delivery.

[0300] Output: Updated ad model

[0301] The above are the specific processing steps for carrying out the present invention.

[0302] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0303] The present invention combines a system that displays advertisements optimized for each viewer in real time with an emotion engine that recognizes the user's emotions. Below, the program processing of this entire system is explained in natural language.

[0304] Collection and analysis of user data

[0305] Subject: Device

[0306] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[0307] Subject: Device

[0308] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[0309] Subject: Device

[0310] The terminal transmits the detected and tracked face data to the server.

[0311] Subject: Server

[0312] The server receives the face data sent from the terminal.

[0313] Subject: Server

[0314] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[0315] Subject: Server

[0316] The server uses an emotion engine to analyze the user's facial expressions, voice, posture, and other characteristics to estimate their emotions. For example, it can recognize emotions such as smile, surprise, and interest.

[0317] Ad selection and automatic generation

[0318] Subject: Server

[0319] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces."

[0320] Subject: Server

[0321] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[0322] Subject: Server

[0323] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[0324] Subject: Server

[0325] The automatically generated advertising content is transmitted to the terminal.

[0326] Ad display and effectiveness measurement

[0327] Subject: Device

[0328] The device receives the advertising content sent from the server and displays it on the display in real time, for example, advertising new fashion items.

[0329] Subject: Device

[0330] While the ad is being displayed, the device uses its camera and emotion engine to collect viewer response data, including eye tracking and facial expression analysis, to measure whether the viewer is paying attention to the ad.

[0331] Subject: Device

[0332] The terminal transmits the collected viewer reaction data and emotion data to the server.

[0333] Subject: Server

[0334] The server receives the reaction data and emotion data transmitted from the terminal.

[0335] Subject: Server

[0336] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. Specifically, it evaluates the length of time the viewer's gaze remains on the screen and changes in emotions, and quantifies the effectiveness of the advertisement.

[0337] Subject: Server

[0338] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[0339] Specific examples

[0340] Signage in a shopping mall

[0341] Subject: Server

[0342] The server collects and analyzes viewer and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate a "woman in her early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. Meanwhile, viewer response and emotional data are collected again, and the effectiveness of the advertisements is measured.

[0343] Digital signage in public transport

[0344] Subject: Server

[0345] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a "male in his 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response and emotional data are collected, and the effectiveness of the advertisements is measured.

[0346] The above is a specific embodiment for implementing the invention of a system incorporating an emotion engine, which allows advertisements optimized for viewers to be provided in real time, maximizing the effectiveness of the advertisements.

[0347] The processing flow will be explained below.

[0348] Step 1:

[0349] Subject: Device

[0350] The device uses a camera to collect real-time video data of viewers, including facial images, eye movements, and facial expressions.

[0351] Step 2:

[0352] Subject: Device

[0353] The device pre-processes the collected video data and performs face detection and tracking for facial recognition, specifically by using a specific algorithm to detect the position of the face and track its movement.

[0354] Step 3:

[0355] Subject: Device

[0356] The device sends face data to the server, including face position information and tracking data.

[0357] Step 4:

[0358] Subject: Server

[0359] The server receives the face data sent from the terminal.

[0360] Step 5:

[0361] Subject: Server

[0362] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Specifically, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling."

[0363] Step 6:

[0364] Subject: Server

[0365] The server uses an emotion engine to analyze the viewer's facial expressions, voice, posture, and other characteristics to estimate their emotions in real time, for example, determining whether they are smiling, surprised, or interested.

[0366] Step 7:

[0367] Subject: Server

[0368] The server references previously collected viewer data and checks it against a database to identify repeat viewers.

[0369] Step 8:

[0370] Subject: Server

[0371] The server then selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands or cosmetics that match the attributes of "women in their 20s" and "smiling faces."

[0372] Step 9:

[0373] Subject: Server

[0374] The server searches an advertising material database provided by the advertiser and lists suitable advertisements as candidates.

[0375] Step 10:

[0376] Subject: Server

[0377] The server then uses AI and algorithms to automatically generate personalized advertising content based on the listed advertising materials, a process that involves combining images, text, and videos.

[0378] Step 11:

[0379] Subject: Server

[0380] The automatically generated advertising content is transmitted to the terminal.

[0381] Step 12:

[0382] Subject: Device

[0383] The terminal receives the advertisement content transmitted from the server.

[0384] Step 13:

[0385] Subject: Device

[0386] The terminal displays the received advertising content on the display in real time. For example, an advertisement for "new fashion items" is displayed on the display.

[0387] Step 14:

[0388] Subject: Device

[0389] While the ad is being displayed, the device uses a camera and emotion engine to collect viewer response data, specifically tracking eye movements and analyzing facial expressions to record changes in viewer attention and emotions.

[0390] Step 15:

[0391] Subject: Device

[0392] The terminal transmits the collected viewer reaction data and emotion data to the server.

[0393] Step 16:

[0394] Subject: Server

[0395] The server receives the reaction data and emotion data transmitted from the terminal.

[0396] Step 17:

[0397] Subject: Server

[0398] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. For example, it quantifies the time the viewer pays attention to the advertisement and changes in their emotions to evaluate the success and impact of the advertisement.

[0399] Step 18:

[0400] Subject: Server

[0401] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[0402] That's all. This system, combined with an emotion engine, makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[0403] Example 2

[0404] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] Conventional advertising display systems have difficulty delivering personalized ads to viewers in real time, making it difficult to maximize advertising effectiveness. They also lack the means to accurately analyze viewers' emotions and reactions and dynamically optimize ads based on that analysis. This has prevented advertisers from delivering optimal ads to viewers, resulting in reduced advertising effectiveness.

[0406] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving viewer data transmitted from a terminal, means for estimating the viewer's age, gender, and emotion using face recognition technology, means for analyzing the user's facial expression, voice, and posture using an emotion engine to estimate emotion, means for selecting an optimal advertisement based on the analyzed viewer data and emotion data, means for referencing an advertising material database and listing appropriate advertisement candidates, means for automatically generating personalized advertising content using a generative AI model, and means for receiving response data and emotion data transmitted from the terminal and analyzing advertising effectiveness. This makes it possible to provide viewers with optimized advertisements in real time, maximizing advertising effectiveness.

[0407] A "terminal" is a hardware device for collecting and processing viewer video data.

[0408] A "server" is a computer system that receives data sent from a terminal, analyzes it, and generates advertisements.

[0409] "Viewer data" refers to data that includes information about the viewer, such as an image of the viewer's face and eye movements.

[0410] "Facial recognition technology" is a technology for detecting human faces from video data and estimating their attributes.

[0411] The "emotion engine" is software that analyzes and estimates emotions from a user's facial expressions, voice, and posture.

[0412] The "advertising material database" is a database for managing various advertising materials provided by advertisers.

[0413] A "generative AI model" is an artificial intelligence model that automatically generates advertising content based on input data.

[0414] "Response data" refers to data related to viewers' reactions to advertisements, such as eye tracking and facial expression analysis.

[0415] "Advertising effectiveness" refers to the effectiveness of an advertisement as evaluated based on viewer reactions and emotional data.

[0416] An "advertising model" is a set of algorithms or rules used to maximize the effectiveness of advertising.

[0417] This invention is a system that provides viewers with advertisements optimized for them in real time, and combines it with an emotion engine that recognizes the user's emotions. This system is composed of multiple elements, including a terminal, a server, and an emotion engine, and these elements work together to maximize the effectiveness of advertisements.

[0418] Collection and analysis of user data

[0419] Subject: Device

[0420] The device is equipped with a camera to collect video data of viewers. The video data collected using the camera includes images of the viewer's face and eye movements. The device uses libraries such as OpenCV to detect and track faces from the collected video data. The detected and tracked face data is sent to the server in JSON format or protocol buffers.

[0421] Subject: Server

[0422] The server receives the facial data sent from the device. Based on the received facial data, it uses facial recognition technology such as dlib to estimate the viewer's age, gender, and emotion. The server then uses an emotion engine (for example, a model trained with TensorFlow or PyTorch) to analyze the user's facial expression, voice, and posture to estimate their emotion. Specifically, it recognizes emotions such as smile, surprise, and interest.

[0423] Ad selection and automatic generation

[0424] Subject: Server

[0425] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetic products based on attributes such as "women in their 20s" and "smiling faces." It references a database of advertising materials provided by the advertiser to create a list of suitable advertisement candidates, and then automatically generates personalized advertising content using a generative AI model (such as GPT-3). The generated advertising content is then sent to the device.

[0426] Ad display and effectiveness measurement

[0427] Subject: Device

[0428] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be shown on the display. While the advertisement is being displayed, the device uses a camera and an emotion engine to collect viewer reaction data. Specifically, it uses eye-tracking technology to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions.

[0429] Subject: Server

[0430] The server receives the reaction and emotion data sent from the device and performs detailed analysis. It evaluates factors such as gaze duration and changes in facial expression to quantify advertising effectiveness. The advertising model is updated based on the analysis results and reflected in the next ad delivery. This maximizes advertising effectiveness and enables more effective ad delivery.

[0431] Specific examples

[0432] Signage in a shopping mall

[0433] Subject: Server

[0434] The server collects and analyzes viewer data and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays within the shopping mall. While the advertisements are displayed, the terminals collect viewer response data and emotional data, which are then sent back to the server to measure the effectiveness of the advertisements.

[0435] Example prompt sentence:

[0436] A video shows a woman in her early twenties at the entrance of a shopping mall. She is smiling and seems interested in the new fashion items. Generate the best ad for her.

[0437] Digital signage in public transport

[0438] Subject: Server

[0439] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a target audience of "men in their 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are then sent to the terminals and displayed on digital signage on the station platforms. At the same time, the terminals collect viewer response and emotional data, which are then sent to the server to measure the effectiveness of the advertisements.

[0440] Example prompt sentence:

[0441] A video shows a man in his 30s on a train platform. He seems interested in technology gadgets. Generate an ad for the latest smartphone that is perfect for him.

[0442] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[0443] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0444] Step 1:

[0445] Video data collection

[0446] Subject: Device

[0447] The device uses a camera to collect real-time video data of the viewer, including facial images and eye movements. Specifically, the device captures the camera's video stream and captures multiple frames per second. The captured frames are stored in a buffer and sent to the next processing step.

[0448] Step 2:

[0449] Face Detection and Tracking

[0450] Subject: Device

[0451] The device detects and tracks faces from the collected video data. It uses OpenCV's face recognition algorithm to detect faces. Specifically, it detects the position of a face in each frame and tracks the same face in the next frame. It uses the acquired video frame as input and generates coordinate data of the detected face as output.

[0452] Step 3:

[0453] Sending data

[0454] Subject: Device

[0455] The device sends the detected and tracked face data to the server. The data is sent in JSON format and is encoded before being sent to the server. It uses the detected face coordinate data as input and generates encoded JSON data as output, which is sent to the server.

[0456] Step 4:

[0457] Receiving data

[0458] Subject: Server

[0459] The server receives the face data sent from the device. Specifically, the server receives an HTTP request and parses the JSON data. It uses the received JSON data as input and obtains parsed face coordinate data as output.

[0460] Step 5:

[0461] Face Recognition and Attribute Estimation

[0462] Subject: Server

[0463] The server uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses dlib's facial recognition model to analyze the viewer's facial features. It uses the received facial coordinate data as input and generates attribute data such as age, gender, and emotion as output.

[0464] Step 6:

[0465] Emotion Analysis

[0466] Subject: Server

[0467] The server uses an emotion engine to analyze the user's facial expressions, voice, and posture to estimate their emotions. Emotion analysis is performed using a model using TensorFlow and PyTorch. Specifically, it analyzes changes in facial expressions, tone of voice, and posture to estimate emotions such as "smile" or "surprise." It uses facial feature data as input and generates emotion data as output.

[0468] Step 7:

[0469] Ad selection

[0470] Subject: Server

[0471] The server selects the optimal advertisement based on the analyzed viewer data and emotional data. It searches the database for an appropriate advertisement according to the viewer's attributes. It uses attribute data such as age, gender, and emotion as input and generates the selected advertisement data as output.

[0472] Step 8:

[0473] Listing advertising materials

[0474] Subject: Server

[0475] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates. Using the selected advertisement data as input, the server generates the listed advertisement material data as output.

[0476] Step 9:

[0477] Auto-generated personalized ads

[0478] Subject: Server

[0479] The server uses a generative AI model to automatically generate personalized advertising content based on the listed advertising materials. It generates advertising text and images using a generative AI model (such as GPT-3). It uses the listed advertising material data as input and obtains generated advertising content as output.

[0480] Step 10:

[0481] Sending advertising content

[0482] Subject: Server

[0483] The server sends the automatically generated advertising content to the terminal. Specifically, the server sends the generated advertising content to the terminal as an HTTP response. The server uses the generated advertising content as input and sends encoded data to the terminal as output.

[0484] Step 11:

[0485] Displaying ads

[0486] Subject: Device

[0487] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, the terminal displays advertising text and images on the display. The terminal uses the received advertising content as input and obtains the displayed advertisement as output.

[0488] Step 12:

[0489] Collecting viewer response data

[0490] Subject: Device

[0491] While the advertisement is being displayed, the device uses a camera and emotion engine to collect viewer response data. Eye-tracking technology is used to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions. The input is the video data during the advertisement display, and the output is the collected viewer response data.

[0492] Step 13:

[0493] Sending reaction data

[0494] Subject: Device

[0495] The device sends the collected viewer reaction data and emotion data to the server. Specifically, it encodes the collected data in JSON format and sends it as an HTTP request. It uses the collected reaction data as input and sends the encoded data to the server as output.

[0496] Step 14:

[0497] Receiving and analyzing data

[0498] Subject: Server

[0499] The server receives the reaction data and emotion data sent from the device and performs detailed analysis. It evaluates gaze duration and changes in facial expressions to quantify the effectiveness of the advertisement. It uses the received reaction data as input and obtains the quantified advertising effectiveness as output.

[0500] Step 15:

[0501] Updated advertising model

[0502] Subject: Server

[0503] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery. Specifically, the server reflects the newly obtained advertising effectiveness data as learning data in the model. The server uses the quantified advertising effectiveness data as input and obtains an updated advertising model as output.

[0504] This concludes the detailed explanation of the processing flow of the program for this system. These processing steps enable personalized advertisements to be provided to viewers in real time, maximizing their effectiveness.

[0505] (Application example 2)

[0506] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0507] Conventional advertising systems have difficulty delivering personalized ads that reflect the emotions and interests of individual users, making it difficult to maximize the effectiveness of advertising. Furthermore, while there is a need to analyze user behavior and emotional data in real time and apply the results immediately, there has been a lack of technology to solve this problem.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0509] In this invention, the server includes means for collecting viewer data from terminals, means for analyzing the viewer data to estimate age, gender, and emotion, means for selecting appropriate advertisements based on the analyzed viewer data, means for automatically generating advertisements based on the selected advertising materials, means for transmitting the automatically generated advertisements to the terminals, means for displaying advertisements on the terminals, means for collecting viewer responses while the advertisements are being displayed, means for analyzing the effectiveness of the advertisements based on the collected response data, and means for analyzing user behavior, gaze data, and emotions and providing information about in-store products and discount coupons in real time. This allows advertisements optimized for individual users to be provided in real time, maximizing the effectiveness of the advertisements.

[0510] A "terminal" is a device that processes information and has functions such as collecting viewer data and displaying advertisements.

[0511] "Viewer data" refers to data that includes information such as the viewer's facial image, eye movements, and facial expressions, and is used to analyze emotions and attributes.

[0512] "Emotion" refers to the psychological state of the viewer that can be inferred from their facial expressions and gestures, including, for example, smile, surprise, interest, etc.

[0513] "Advertising materials" refers to content such as images, text, and videos used in advertising distribution.

[0514] "Auto-generating ads" refers to the process of automatically creating appropriate advertising content based on analyzed viewer data.

[0515] "Real-time" means that data collection, analysis, ad generation and delivery occur almost simultaneously.

[0516] "Behavioral data" refers to data including users' movements within a store, their line of sight, and the length of time they spend there.

[0517] A "discount coupon" is an electronic or paper voucher offering a special discount on the purchase of a product.

[0518] A "server" is a central computer system for receiving, analyzing, and processing data sent from terminals.

[0519] "Advertising effectiveness" refers to the results of evaluating viewers' reactions and purchasing intentions after viewing an advertisement.

[0520] To implement this invention, the following system configuration and processes are required: The system functions through cooperation between the terminal, the server, and the user.

[0521] System configuration

[0522] 1. Terminal

[0523] Hardware: Smart glasses (including camera, display, and communication module)

[0524] Software: OpenCV (for face recognition), emotion_engine (for emotion analysis)

[0525] 2. Server

[0526] Software: Data receiving module, data analysis module (viewer data analysis, emotion estimation), ad selection module, ad generation module, ad transmission module

[0527] 3. Users

[0528] An individual wearing smart glasses and moving around in a physical store

[0529] Program processing

[0530] 1. Collecting viewer data via devices

[0531] The device uses the camera installed in the smart glasses to collect real-time video data of the user, including facial images and eye movements (eye tracking).

[0532] 2. Preprocessing and analysis of viewer data

[0533] The device preprocesses the collected video data using OpenCV, detects and tracks faces, and transmits the detected and tracked face data to the server.

[0534] 3. Data analysis and ad selection by the server

[0535] The server receives the facial data sent from the device and uses the emotion_engine to estimate the viewer's attributes, such as age, gender, and emotions. Based on the analysis results, the server selects the most suitable advertisement. Specifically, it selects products and discount coupons that are likely to interest the viewer.

[0536] 4. Auto-generated ads

[0537] The server references a database of advertising materials provided by advertisers and uses AI algorithms to automatically generate personalized advertising content, which is then sent to the device.

[0538] 5. Display of advertisements by device

[0539] The terminal receives the advertising content sent from the server and displays it on the display of the smart glasses in real time.

[0540] 6. Measuring advertising effectiveness

[0541] While the ad is being displayed, the device uses a camera and an emotion engine to collect user reaction data, which is then sent to a server to measure the effectiveness of the ad.

[0542] 7. Analysis of advertising effectiveness and model update

[0543] The server analyzes the advertising effectiveness based on the response data sent from the terminal and updates the advertising model to reflect the results in the next advertisement distribution.

[0544] Specific examples

[0545] When a user wears smart glasses and visits a clothing store, the camera in the glasses tracks the user's gaze, and if the user stays in a particular section for a long time, the emotion engine analyzes the user's facial expressions of interest or joy. Based on this information, the server displays information about similar products and discount coupons on the smart glasses' display in real time.

[0546] Prompt Sentence Examples

[0547] "Get 20% off your favorite clothes today. Check out our new collection."

[0548] The above is a specific embodiment of the invention. This system allows users to receive optimized advertisements in real time, maximizing the effectiveness of the advertisements.

[0549] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0550] Step 1:

[0551] Collecting viewer data via devices

[0552] The device uses the camera installed in the smart glasses to collect the user's video data in real time, including the user's facial image, eye movements, and facial expressions. This video data is input and output as raw data captured by the camera.

[0553] Step 2:

[0554] Audience data preprocessing and facial recognition

[0555] The device preprocesses the collected video data using OpenCV to detect and track faces. Specifically, it analyzes the video data frame by frame to recognize the face, and then tracks the position of the face. The input for this process is the camera video data, and the output is data including the position information of the face.

[0556] Step 3:

[0557] Sending face data to the server

[0558] The device sends the detected and tracked face data to the server. The sent data includes face images, their location information, and changes over time. The input is data including face location information, and the output is a transmission completion status to the server.

[0559] Step 4:

[0560] Analysis of viewer data by the server

[0561] The server receives the facial data sent from the device and uses emotion_engine to estimate the viewer's attributes such as age, gender, and emotion. Specifically, it applies a facial recognition algorithm to the received data as input and performs analysis using an age estimation model, gender estimation model, and emotion estimation model. The input is facial data, and the output is the analysis results (e.g., "woman in her 20s," "smiling," etc.).

[0562] Step 5:

[0563] Selecting the right ads

[0564] The server selects suitable advertisements based on the analyzed viewer data. Here, it references the product database and the advertising material database provided by the advertiser to list advertisements that match the analysis results. The input is the analysis results of the viewer data, and the output is a list of selected advertisement candidates.

[0565] Step 6:

[0566] Auto-generated ads

[0567] The server automatically generates personalized advertising content using an AI algorithm based on the ad list obtained in the ad selection step. Specifically, it uses a generative AI model to combine advertising materials to create optimal advertising content. The input is the list of ad candidates, and the output is the generated advertising content.

[0568] Step 7:

[0569] Sending advertising content to devices

[0570] The server transmits automatically generated advertising content to the terminal, where the input is the generated advertising content and the output is a transmission completion status to the terminal.

[0571] Step 8:

[0572] Display of advertisements by device

[0573] The terminal receives the advertising content sent from the server and displays it on the smart glasses display in real time, where specific advertising content and discount coupons are displayed to the user. The input is the received advertising content, and the output is the advertisement displayed to the user.

[0574] Step 9:

[0575] Collecting user responses while ads are displayed

[0576] While the ad is being displayed, the device uses a camera and emotion engine to collect user response data (such as gaze, facial expressions, and visual characteristics). Specific operations include recording attention to the ad and changes in emotion. The input is the video data of the ad being displayed, and the output is user response data.

[0577] Step 10:

[0578] Sending reaction data to the server

[0579] The terminal sends the collected user reaction data to the server. The input is the user reaction data, and the output is the transmission completion status to the server.

[0580] Step 11:

[0581] Analysis of advertising effectiveness and model updating

[0582] The server analyzes the effectiveness of the advertisement based on the response data sent from the device and updates the advertising model to reflect this in the next ad delivery. Specifically, it uses an analysis algorithm to evaluate gaze time and changes in emotions. The input is the user's response data, and the output is an updated advertising model.

[0583] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0584] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0585] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0586] [Second embodiment]

[0587] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0588] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0589] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0590] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0591] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0592] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0593] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0594] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0597] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0598] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0599] The present invention relates to a system for displaying advertisements optimized for viewers in real time. The program processing of the entire system will be explained below in natural language.

[0600] Collection and analysis of user data

[0601] Subject: Device

[0602] The device uses a camera to collect video data of the viewer, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[0603] Subject: Server

[0604] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses a facial recognition model to analyze the image data and identify attributes such as "female in her 20s" or "smiling." The server also references data collected in the past to determine whether the viewer is the same person. This also makes it possible to identify repeat viewers.

[0605] Ad selection and automatic generation

[0606] Subject: Server

[0607] The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces." It then searches a database of advertising materials provided by the advertiser to create a list of the most suitable candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[0608] Ad display and effectiveness measurement

[0609] Subject: Device

[0610] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[0611] Subject: Server

[0612] While the advertisement is being displayed, the device collects viewer response data, which is then sent to a server to measure the effectiveness of the advertisement. The server then uses this data to analyze the success and impact of the advertisement and update the advertising model for further optimization.

[0613] Specific examples

[0614] Signage in a shopping mall

[0615] Subject: Server

[0616] The server collects and analyzes viewer data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[0617] Digital signage in public transport

[0618] Subject: Server

[0619] The server collects and analyzes viewer data from terminals installed on public transport station platforms. For example, if the analysis results indicate "men in their 30s," it selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response data is collected and the effectiveness of the advertisements is measured.

[0620] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[0621] The processing flow will be explained below.

[0622] Step 1:

[0623] Subject: Device

[0624] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[0625] Step 2:

[0626] Subject: Device

[0627] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[0628] Step 3:

[0629] Subject: Device

[0630] The terminal transmits the detected and tracked face data to the server.

[0631] Step 4:

[0632] Subject: Server

[0633] The server receives the face data sent from the terminal.

[0634] Step 5:

[0635] Subject: Server

[0636] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[0637] Step 6:

[0638] Subject: Server

[0639] The server references previously collected data and checks it against a database to identify whether the viewer is a repeat visitor.

[0640] Step 7:

[0641] Subject: Server

[0642] The server selects appropriate advertisements based on the analyzed viewer data.

[0643] Step 8:

[0644] Subject: Server

[0645] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[0646] Step 9:

[0647] Subject: Server

[0648] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[0649] Step 10:

[0650] Subject: Server

[0651] The server transmits the automatically generated advertising content to the terminal.

[0652] Step 11:

[0653] Subject: Device

[0654] The terminal receives the advertisement content transmitted from the server.

[0655] Step 12:

[0656] Subject: Device

[0657] The terminal displays the received advertising content on the display in real time, for example, an advertisement for a new fashion item.

[0658] Step 13:

[0659] Subject: Device

[0660] While the ad is being displayed, the device uses its camera to collect viewer response data, including eye tracking and facial expression analysis.

[0661] Step 14:

[0662] Subject: Device

[0663] The terminal transmits the collected viewer reaction data to the server.

[0664] Step 15:

[0665] Subject: Server

[0666] The server receives the response data sent from the terminal.

[0667] Step 16:

[0668] Subject: Server

[0669] The server analyzes the effectiveness of the advertisement based on the reaction data, specifically evaluating the length of time the viewer's gaze remains on the screen and changes in facial expression, and quantifies the effectiveness of the advertisement.

[0670] Step 17:

[0671] Subject: Server

[0672] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[0673] This allows viewers to receive ads that are optimized for them in real time, maximizing advertising effectiveness.

[0674] Example 1

[0675] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Conventional advertising display systems have had difficulty effectively delivering advertisements to viewers. In particular, they have had issues with being unable to select the most appropriate advertisement based on the viewer's attributes and display it in real time. Furthermore, there are limitations to accurately measuring advertising effectiveness and reflecting this in future ad distribution. Furthermore, there are insufficient methods for collecting data to analyze how ads affect viewers.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0678] In this invention, the server includes means for preprocessing viewer data, detecting and tracking face data, means for estimating the viewer's age, gender, and emotion based on the face data, and means for selecting optimal advertisements based on the estimated viewer attributes. This makes it possible to display advertisements optimized for the viewer in real time, accurately measure the effectiveness of the advertisements, and reflect this in the next distribution.

[0679] A "terminal" is a device installed to collect data from viewers and transmit the collected data to a server.

[0680] "Viewer data" refers to viewer video data and other related data collected by a terminal via a camera or the like.

[0681] "Preprocessing" refers to processing such as image resizing and noise removal that is performed on collected video data.

[0682] "Face data" refers to data of the viewer's face regions detected and tracked from the pre-processed video data.

[0683] "Means for estimating age, gender, and emotion of a viewer based on facial data" refers to the processes and models within the server that analyze received facial data and estimate the age, gender, and emotional attributes of the target viewer.

[0684] "Viewer demographics" are data that refer to specific characteristics of viewers, such as their age, gender, and emotions.

[0685] The "means for selecting an advertisement" refers to the mechanisms and algorithms within the server for selecting the most suitable advertisement based on the estimated viewer attributes.

[0686] "Advertising materials" are material data such as images, text, and videos used to generate advertisements.

[0687] "Advertising content" refers to advertising content that is automatically generated based on selected advertising materials.

[0688] "Response data" refers to data such as the viewer's gaze direction and changes in facial expression that are collected while the advertisement is being displayed.

[0689] "Means for analyzing advertising effectiveness" refers to the processes and algorithms within the server that analyze the effectiveness and influence of displayed advertisements based on collected response data.

[0690] The present invention relates to a system for displaying advertisements optimized for viewers in real time, which operates through multiple steps involving terminals, servers, and users.

[0691] 1. Collection of User Data

[0692] Subject: Device

[0693] The device collects viewer video data using a camera. The camera is a commonly used hardware device, such as a USB or built-in camera. The video data is pre-processed using an image processing library such as OpenCV, which results in the detection and tracking of viewer facial data. This facial data is then sent to the server in real time.

[0694] 2. Data Preprocessing and Analysis

[0695] Subject: Device

[0696] The device performs preprocessing on the collected video data. This preprocessing includes image resizing and noise reduction. It uses libraries such as OpenCV to adjust the image quality of the video data and detects the viewer's face using a face detection model (e.g., Haar Cascades or MTCNN). The device then extracts the detected face area and sends the data to the server.

[0697] 3. Estimation of viewer attributes

[0698] Subject: Server

[0699] The server receives the facial data sent from the device. Based on the received data, it uses a facial recognition model (e.g., Dlib or FaceNet) to estimate the viewer's age, gender, and emotion. For example, it identifies attributes such as "female in her 20s" or "smiling." In this process, it extracts features from the facial data and uses a pre-trained model to make an estimation.

[0700] 4. Ad selection and automatic generation

[0701] Subject: Server

[0702] The server selects the most suitable advertisement based on the analysis results. For example, it selects advertisements for fashion brands or cosmetics based on the attributes "women in their 20s" and "smiling face." The server searches an advertising material database (e.g., cloud storage service) and automatically generates advertising content using deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The advertising content is generated in formats such as HTML5 or MP4.

[0703] 5. Submission of advertising content

[0704] Subject: Server

[0705] The server sends the generated advertising content to the device using real-time communication technology such as WebSocket. The server identifies the device's IP address and sends the advertising data to the endpoint.

[0706] 6. Display of advertisements

[0707] Subject: Device

[0708] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[0709] 7. Collecting viewer response data

[0710] Subject: Device

[0711] While the ad is being displayed, the device uses a camera to collect viewer response data, capturing gaze direction and facial expression changes in real time for visual analysis, which is then sent back to the server.

[0712] 8. Analysis of advertising effectiveness and feedback

[0713] Subject: Server

[0714] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. For example, if the viewer's gaze is frequently directed at the advertisement while viewing it, the advertisement is deemed to be highly successful. Based on this data, the server evaluates the effectiveness and influence of the advertisement and updates the model to help with future advertisement selection.

[0715] Specific examples

[0716] Signage in a shopping mall

[0717] Terminals installed at the entrance to the shopping mall collect viewer data. If the analysis results indicate a "woman in her early twenties," the server selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. The generated advertisements are sent to the terminals and displayed on displays within the mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[0718] Digital signage in public transport

[0719] Terminals installed on public transport station platforms collect viewer data. For example, if the analysis results show "men in their 30s," the server selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. The generated advertisements are sent to the terminals and displayed on digital signage on the station platforms. Viewer response data is collected during display and sent to the server, where the advertising effectiveness is measured.

[0720] Prompt Sentence Examples

[0721] "Select an ad for a tech gadget for a man in his 30s and generate and display an ad for the latest smartphone."

[0722] keyword

[0723] Generative AI model, prompt sentence

[0724] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0725] Step 1:

[0726] Subject: Device

[0727] The device uses a camera to collect video data of the viewer. This video data is captured from the camera in real time. Specifically, it stores frame data continuously acquired from the camera in memory and performs image preprocessing using the OpenCV library. Preprocessing includes image resizing and noise removal. The input is raw video data, and the output is preprocessed video data.

[0728] Step 2:

[0729] Subject: Device

[0730] The device detects and tracks the viewer's facial data from the preprocessed video data. Specifically, it identifies the facial region using a face detection model such as OpenCV, Haar Cascades, or MTCNN. The input is the preprocessed video data, and the output is the viewer's facial region data. This data is sent to the server for further processing.

[0731] Step 3:

[0732] Subject: Server

[0733] The server receives the face region data sent from the device. Based on the received data, it uses a face recognition model such as Dlib or FaceNet to estimate the viewer's age, gender, and emotion. The input is the face region data, and the output is the estimated viewer attributes (e.g., "female in her 20s" and "smiling"). Specifically, it extracts features and analyzes them using a pre-trained model.

[0734] Step 4:

[0735] Subject: Server

[0736] The server selects the most suitable advertisement based on the viewer attributes obtained as a result of the analysis. As a specific example, it searches an advertising material database to select advertisements for fashion brands and cosmetics for the attributes "women in their 20s" and "smiling faces." The input is the estimated viewer attributes, and the output is the most suitable advertising material.

[0737] Step 5:

[0738] Subject: Server

[0739] The server automatically generates advertising content based on the selected advertising materials. This process uses deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The input is the selected advertising materials, and the output is the generated advertising content. Specifically, the server creates advertisements in HTML5 or MP4 format based on the material data.

[0740] Step 6:

[0741] Subject: Server

[0742] The server transmits the generated advertising content to the device using real-time communication technology such as WebSocket. The input is the generated advertising content, and the output is data transmission to the device.

[0743] Step 7:

[0744] Subject: Device

[0745] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, an advertisement for a new fashion item is displayed on a display device connected to the terminal. The input is the received advertising content, and the output is the display on which the advertisement is displayed.

[0746] Step 8:

[0747] Subject: Device

[0748] While the ad is being displayed, the device uses a camera to collect viewer response data. Specifically, it captures changes in gaze direction and facial expressions. The input is the displayed ad content, and the output is viewer response data. This data is then sent back to the server.

[0749] Step 9:

[0750] Subject: Server

[0751] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. Specifically, if the viewer's gaze is frequently directed at the advertisement, the success of the advertisement is evaluated highly. The input is the viewer response data, and the output is the analysis result of the advertisement's effectiveness. The server updates the advertising model based on this analysis result and reflects it in subsequent advertisement selections.

[0752] (Application example 1)

[0753] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] Conventional advertising display systems did not fully utilize viewer information, making it difficult to select the optimal advertisement to maximize advertising effectiveness. Furthermore, they lacked the technology to collect viewer response information in real time and immediately reflect it in advertising strategies. As a result, advertisements that were not appealing to viewers were displayed, reducing advertising effectiveness.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0756] In this invention, the server includes means for collecting viewer data from the terminal, means for analyzing the viewer data and estimating age, gender, and emotion, means for selecting an appropriate advertisement based on the analyzed viewer data, means for automatically generating an advertisement based on the selected advertising material, means for transmitting the automatically generated advertisement to the terminal, means for displaying the advertisement on the terminal, means for collecting viewer responses while the advertisement is being displayed, means for analyzing the effectiveness of the advertisement based on the collected response data, means for collecting and analyzing video of the viewer using a camera mounted on the smart glasses, and means for displaying the advertisement on the display of the smart glasses. This enables real-time display of personalized advertisements tailored to the characteristics of the viewer and rapid feedback of the effectiveness of the advertisement.

[0757] "Terminal" refers to a device used to collect viewer data and display advertisements, including smartphones, tablets, smart glasses, etc.

[0758] "Viewer data" refers to data that includes information such as the viewer's age, gender, and emotions, and is collected through cameras and sensors.

[0759] "Analysis" is the process of estimating attributes such as age, gender, and emotions based on collected viewer data.

[0760] "Ad selection" is the process of selecting the most suitable advertisement based on analyzed viewer data.

[0761] "Advertising materials" are materials such as databases, images, videos, and text used to generate advertisements.

[0762] "Automatic generation" is the process of automatically creating advertising content based on advertising materials after an advertisement has been selected.

[0763] "Advertising effectiveness" is an indicator that evaluates the impact that an advertisement has on viewers, and is analyzed based on viewer response data.

[0764] "Smart glasses" are wearable devices equipped with a camera and a display that can collect video data from viewers and display advertisements.

[0765] A "camera" is a device that collects video data from viewers and is installed in smart glasses or terminals.

[0766] A "display" is a display device that displays the generated advertisement to the viewer, and is installed in a terminal or smart glasses.

[0767] "Response data" refers to data such as the viewer's gaze, facial expressions, and movements while the advertisement is being displayed.

[0768] This invention relates to a system for displaying advertisements optimized for viewers in real time and measuring their effectiveness. This system is composed of the following components:

[0769] System Configuration

[0770] 1. Hardware

[0771] Devices: Includes smartphones, tablets, smart glasses, etc.

[0772] Camera: A device installed in smart glasses or a device that collects video data of the viewer.

[0773] Display: A device mounted on the smart glasses or device that displays the generated advertisements.

[0774] 2. Software

[0775] Facial recognition technology: For example, OpenCV is used.

[0776] Data analysis platform: For example, AWS Lambda is used.

[0777] Ad selection algorithm: For example, TensorFlow is used.

[0778] System processing procedure

[0779] Collection and analysis of user data

[0780] The device collects the viewer's video data, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[0781] Ad selection and automatic generation

[0782] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotions. To do this, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling." The server then references previously collected data to determine whether the viewer is the same person. The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on the attributes "female in her 20s" and "smiling." It then searches a database of advertising materials provided by the advertiser and lists the most suitable advertising candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[0783] Ad display and effectiveness measurement

[0784] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be displayed on the display. While the advertisement is being displayed, the device collects viewer response data. The collected response data is sent to the server, where the effectiveness of the advertisement is measured. The server uses this data to analyze the success and influence of the advertisement and update the advertising model for further optimization.

[0785] Specific examples

[0786] Signage in a shopping mall

[0787] When a user is wearing smart glasses while walking through a shopping mall, the glasses collect the user's facial data in real time and send it to a server. Based on this data, the server determines that the user is a woman in her early twenties and selects advertisements for fashion brands and accessories. Advertisements for new collections are automatically generated and instantly displayed on the smart glasses' display. The user's reactions are also collected again, and the effectiveness of the advertisements is measured.

[0788] Digital signage in public transport

[0789] Terminals installed on public transport station platforms collect the viewer's facial data and send it to a server. If the user is identified as a "male in his 30s," ads for technology gadgets and sports equipment are selected, and the latest smartphone ads are automatically generated. These ads are then displayed on digital signage on the station platforms. Viewer response data is collected in the same way, and the effectiveness of the ads is measured.

[0790] Prompt Sentence Examples

[0791] "Woman in her 20s, smiling. Generate an ad for new fashion items. Also consider the option to show an ad for your summer accessories collection instead."

[0792] "Based on facial recognition data, we've identified her as a woman in her 20s. Then generate ads for a fashion brand that would be suitable for her. Also, create content that reflects positive emotions, taking into account smiling faces."

[0793] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[0794] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0795] Step 1:

[0796] User Data Collection

[0797] Input: Video data from a camera mounted on smart glasses

[0798] Processing: The device uses the camera to collect the user's video data. The video data is captured in real time and pre-processed (resolution adjustment, noise reduction).

[0799] Output: Pre-processed video data

[0800] Step 2:

[0801] Face Detection and Tracking

[0802] Input: Preprocessed video data

[0803] Processing: The device uses OpenCV to detect faces in the video data and perform face tracking. The face detection algorithm identifies the position of the face within the video frame.

[0804] Output: Face data (face position, facial feature points, etc.)

[0805] Step 3:

[0806] Data transmission

[0807] Input: Face data

[0808] Processing: The device sends the detected face data to the server in real time using the secure HTTP(S) protocol.

[0809] Output: Face data sent to the server

[0810] Step 4:

[0811] Face Recognition and Attribute Estimation

[0812] Input: Face data sent to the server

[0813] Processing: The server analyzes the facial data using facial recognition technology (e.g., a facial recognition model) to estimate age, gender, and emotion. Analysis includes extracting facial features and estimating attributes using a classifier.

[0814] Output: Viewer attribute data (e.g., "Female in her 20s," "Smiling," etc.)

[0815] Step 5:

[0816] Repeater Identification

[0817] Input: Viewer demographic data

[0818] Processing: The server refers to a database of previously collected data and compares it with the viewer's facial features to identify whether they are repeat visitors. For this purpose, it uses an approximate nearest neighbor search algorithm.

[0819] Output: Repeater identification result

[0820] Step 6:

[0821] Ad selection

[0822] Input: Viewer attribute data, repeater identification results

[0823] Processing: The server selects the optimal advertisement using an advertisement selection algorithm (for example, an algorithm using TensorFlow) based on the viewer attribute data and the results of repeat customer identification.

[0824] Output: Ad candidate list

[0825] Step 7:

[0826] Auto-generated ads

[0827] Input: Ad candidate list

[0828] Processing: The server automatically generates optimal advertising content from a selected list of ad candidates based on the advertising material database provided by the advertiser. For generation, a generative AI model is used.

[0829] Output: Generated ad content

[0830] Step 8:

[0831] Sending Ads

[0832] Input: Generated ad content

[0833] Processing: The server sends the generated advertising content to the device using secure HTTP(S) as the communication protocol.

[0834] Output: Ad content sent to the device

[0835] Step 9:

[0836] Displaying ads

[0837] Input: Ad content sent to the device

[0838] Processing: The device displays the received advertising content on the display in real time. The advertisement is displayed to the user using the display of the smart glasses.

[0839] Output: The ad shown to the user

[0840] Step 10:

[0841] Collecting viewer responses

[0842] Input: The ad shown to the user

[0843] Processing: The device collects user response data (eye gaze, facial expressions, movements, etc.) while the ad is being displayed. The device uses cameras and sensors to capture response data in real time.

[0844] Output: Collected reaction data

[0845] Step 11:

[0846] Sending reaction data

[0847] Input: Collected reaction data

[0848] Processing: The device sends the collected reaction data to the server using secure HTTP(S).

[0849] Output: Response data sent to the server

[0850] Step 12:

[0851] Measuring advertising effectiveness

[0852] Input: Reaction data sent to the server

[0853] Processing: The server analyzes the effectiveness of the ads based on the collected response data, using statistical analysis and machine learning models.

[0854] Output: Advertising effectiveness data

[0855] Step 13:

[0856] Updated advertising model

[0857] Input: Advertising effectiveness data

[0858] Processing: The server updates the advertising model based on the advertising effectiveness data. The updated model is reflected in the next ad delivery.

[0859] Output: Updated ad model

[0860] The above are the specific processing steps for carrying out the present invention.

[0861] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0862] The present invention combines a system that displays advertisements optimized for each viewer in real time with an emotion engine that recognizes the user's emotions. Below, the program processing of this entire system is explained in natural language.

[0863] Collection and analysis of user data

[0864] Subject: Device

[0865] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[0866] Subject: Device

[0867] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[0868] Subject: Device

[0869] The terminal transmits the detected and tracked face data to the server.

[0870] Subject: Server

[0871] The server receives the face data sent from the terminal.

[0872] Subject: Server

[0873] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[0874] Subject: Server

[0875] The server uses an emotion engine to analyze the user's facial expressions, voice, posture, and other characteristics to estimate their emotions. For example, it can recognize emotions such as smile, surprise, and interest.

[0876] Ad selection and automatic generation

[0877] Subject: Server

[0878] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces."

[0879] Subject: Server

[0880] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[0881] Subject: Server

[0882] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[0883] Subject: Server

[0884] The automatically generated advertising content is transmitted to the terminal.

[0885] Ad display and effectiveness measurement

[0886] Subject: Device

[0887] The device receives the advertising content sent from the server and displays it on the display in real time, for example, advertising new fashion items.

[0888] Subject: Device

[0889] While the ad is being displayed, the device uses its camera and emotion engine to collect viewer response data, including eye tracking and facial expression analysis, to measure whether the viewer is paying attention to the ad.

[0890] Subject: Device

[0891] The terminal transmits the collected viewer reaction data and emotion data to the server.

[0892] Subject: Server

[0893] The server receives the reaction data and emotion data transmitted from the terminal.

[0894] Subject: Server

[0895] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. Specifically, it evaluates the length of time the viewer's gaze remains on the screen and changes in emotions, and quantifies the effectiveness of the advertisement.

[0896] Subject: Server

[0897] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[0898] Specific examples

[0899] Signage in a shopping mall

[0900] Subject: Server

[0901] The server collects and analyzes viewer and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate a "woman in her early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. Meanwhile, viewer response and emotional data are collected again, and the effectiveness of the advertisements is measured.

[0902] Digital signage in public transport

[0903] Subject: Server

[0904] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a "male in his 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response and emotional data are collected, and the effectiveness of the advertisements is measured.

[0905] The above is a specific embodiment for implementing the invention of a system incorporating an emotion engine, which allows advertisements optimized for viewers to be provided in real time, maximizing the effectiveness of the advertisements.

[0906] The processing flow will be explained below.

[0907] Step 1:

[0908] Subject: Device

[0909] The device uses a camera to collect real-time video data of viewers, including facial images, eye movements, and facial expressions.

[0910] Step 2:

[0911] Subject: Device

[0912] The device pre-processes the collected video data and performs face detection and tracking for facial recognition, specifically by using a specific algorithm to detect the position of the face and track its movement.

[0913] Step 3:

[0914] Subject: Device

[0915] The device sends face data to the server, including face position information and tracking data.

[0916] Step 4:

[0917] Subject: Server

[0918] The server receives the face data sent from the terminal.

[0919] Step 5:

[0920] Subject: Server

[0921] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Specifically, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling."

[0922] Step 6:

[0923] Subject: Server

[0924] The server uses an emotion engine to analyze the viewer's facial expressions, voice, posture, and other characteristics to estimate their emotions in real time, for example, determining whether they are smiling, surprised, or interested.

[0925] Step 7:

[0926] Subject: Server

[0927] The server references previously collected viewer data and checks it against a database to identify repeat viewers.

[0928] Step 8:

[0929] Subject: Server

[0930] The server then selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands or cosmetics that match the attributes of "women in their 20s" and "smiling faces."

[0931] Step 9:

[0932] Subject: Server

[0933] The server searches an advertising material database provided by the advertiser and lists suitable advertisements as candidates.

[0934] Step 10:

[0935] Subject: Server

[0936] The server then uses AI and algorithms to automatically generate personalized advertising content based on the listed advertising materials, a process that involves combining images, text, and videos.

[0937] Step 11:

[0938] Subject: Server

[0939] The automatically generated advertising content is transmitted to the terminal.

[0940] Step 12:

[0941] Subject: Device

[0942] The terminal receives the advertisement content transmitted from the server.

[0943] Step 13:

[0944] Subject: Device

[0945] The terminal displays the received advertising content on the display in real time. For example, an advertisement for "new fashion items" is displayed on the display.

[0946] Step 14:

[0947] Subject: Device

[0948] While the ad is being displayed, the device uses a camera and emotion engine to collect viewer response data, specifically tracking eye movements and analyzing facial expressions to record changes in viewer attention and emotions.

[0949] Step 15:

[0950] Subject: Device

[0951] The terminal transmits the collected viewer reaction data and emotion data to the server.

[0952] Step 16:

[0953] Subject: Server

[0954] The server receives the reaction data and emotion data transmitted from the terminal.

[0955] Step 17:

[0956] Subject: Server

[0957] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. For example, it quantifies the time the viewer pays attention to the advertisement and changes in their emotions to evaluate the success and impact of the advertisement.

[0958] Step 18:

[0959] Subject: Server

[0960] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[0961] That's all. This system, combined with an emotion engine, makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[0962] Example 2

[0963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0964] Conventional advertising display systems have difficulty delivering personalized ads to viewers in real time, making it difficult to maximize advertising effectiveness. They also lack the means to accurately analyze viewers' emotions and reactions and dynamically optimize ads based on that analysis. This has prevented advertisers from delivering optimal ads to viewers, resulting in reduced advertising effectiveness.

[0965] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving viewer data transmitted from a terminal, means for estimating the viewer's age, gender, and emotion using face recognition technology, means for analyzing the user's facial expression, voice, and posture using an emotion engine to estimate emotion, means for selecting an optimal advertisement based on the analyzed viewer data and emotion data, means for referencing an advertising material database and listing appropriate advertisement candidates, means for automatically generating personalized advertising content using a generative AI model, and means for receiving response data and emotion data transmitted from the terminal and analyzing advertising effectiveness. This makes it possible to provide viewers with optimized advertisements in real time, maximizing advertising effectiveness.

[0966] A "terminal" is a hardware device for collecting and processing viewer video data.

[0967] A "server" is a computer system that receives data sent from a terminal, analyzes it, and generates advertisements.

[0968] "Viewer data" refers to data that includes information about the viewer, such as an image of the viewer's face and eye movements.

[0969] "Facial recognition technology" is a technology for detecting human faces from video data and estimating their attributes.

[0970] The "emotion engine" is software that analyzes and estimates emotions from a user's facial expressions, voice, and posture.

[0971] The "advertising material database" is a database for managing various advertising materials provided by advertisers.

[0972] A "generative AI model" is an artificial intelligence model that automatically generates advertising content based on input data.

[0973] "Response data" refers to data related to viewers' reactions to advertisements, such as eye tracking and facial expression analysis.

[0974] "Advertising effectiveness" refers to the effectiveness of an advertisement as evaluated based on viewer reactions and emotional data.

[0975] An "advertising model" is a set of algorithms or rules used to maximize the effectiveness of advertising.

[0976] This invention is a system that provides viewers with advertisements optimized for them in real time, and combines it with an emotion engine that recognizes the user's emotions. This system is composed of multiple elements, including a terminal, a server, and an emotion engine, and these elements work together to maximize the effectiveness of advertisements.

[0977] Collection and analysis of user data

[0978] Subject: Device

[0979] The device is equipped with a camera to collect video data of viewers. The video data collected using the camera includes images of the viewer's face and eye movements. The device uses libraries such as OpenCV to detect and track faces from the collected video data. The detected and tracked face data is sent to the server in JSON format or protocol buffers.

[0980] Subject: Server

[0981] The server receives the facial data sent from the device. Based on the received facial data, it uses facial recognition technology such as dlib to estimate the viewer's age, gender, and emotion. The server then uses an emotion engine (for example, a model trained with TensorFlow or PyTorch) to analyze the user's facial expression, voice, and posture to estimate their emotion. Specifically, it recognizes emotions such as smile, surprise, and interest.

[0982] Ad selection and automatic generation

[0983] Subject: Server

[0984] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetic products based on attributes such as "women in their 20s" and "smiling faces." It references a database of advertising materials provided by the advertiser to create a list of suitable advertisement candidates, and then automatically generates personalized advertising content using a generative AI model (such as GPT-3). The generated advertising content is then sent to the device.

[0985] Ad display and effectiveness measurement

[0986] Subject: Device

[0987] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be shown on the display. While the advertisement is being displayed, the device uses a camera and an emotion engine to collect viewer reaction data. Specifically, it uses eye-tracking technology to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions.

[0988] Subject: Server

[0989] The server receives the reaction and emotion data sent from the device and performs detailed analysis. It evaluates factors such as gaze duration and changes in facial expression to quantify advertising effectiveness. The advertising model is updated based on the analysis results and reflected in the next ad delivery. This maximizes advertising effectiveness and enables more effective ad delivery.

[0990] Specific examples

[0991] Signage in a shopping mall

[0992] Subject: Server

[0993] The server collects and analyzes viewer data and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays within the shopping mall. While the advertisements are displayed, the terminals collect viewer response data and emotional data, which are then sent back to the server to measure the effectiveness of the advertisements.

[0994] Example prompt sentence:

[0995] A video shows a woman in her early twenties at the entrance of a shopping mall. She is smiling and seems interested in the new fashion items. Generate the best ad for her.

[0996] Digital signage in public transport

[0997] Subject: Server

[0998] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a target audience of "men in their 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are then sent to the terminals and displayed on digital signage on the station platforms. At the same time, the terminals collect viewer response and emotional data, which are then sent to the server to measure the effectiveness of the advertisements.

[0999] Example prompt sentence:

[1000] A video shows a man in his 30s on a train platform. He seems interested in technology gadgets. Generate an ad for the latest smartphone that is perfect for him.

[1001] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[1002] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1003] Step 1:

[1004] Video data collection

[1005] Subject: Device

[1006] The device uses a camera to collect real-time video data of the viewer, including facial images and eye movements. Specifically, the device captures the camera's video stream and captures multiple frames per second. The captured frames are stored in a buffer and sent to the next processing step.

[1007] Step 2:

[1008] Face Detection and Tracking

[1009] Subject: Device

[1010] The device detects and tracks faces from the collected video data. It uses OpenCV's face recognition algorithm to detect faces. Specifically, it detects the position of a face in each frame and tracks the same face in the next frame. It uses the acquired video frame as input and generates coordinate data of the detected face as output.

[1011] Step 3:

[1012] Sending data

[1013] Subject: Device

[1014] The device sends the detected and tracked face data to the server. The data is sent in JSON format and is encoded before being sent to the server. It uses the detected face coordinate data as input and generates encoded JSON data as output, which is sent to the server.

[1015] Step 4:

[1016] Receiving data

[1017] Subject: Server

[1018] The server receives the face data sent from the device. Specifically, the server receives an HTTP request and parses the JSON data. It uses the received JSON data as input and obtains parsed face coordinate data as output.

[1019] Step 5:

[1020] Face Recognition and Attribute Estimation

[1021] Subject: Server

[1022] The server uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses dlib's facial recognition model to analyze the viewer's facial features. It uses the received facial coordinate data as input and generates attribute data such as age, gender, and emotion as output.

[1023] Step 6:

[1024] Emotion Analysis

[1025] Subject: Server

[1026] The server uses an emotion engine to analyze the user's facial expressions, voice, and posture to estimate their emotions. Emotion analysis is performed using a model using TensorFlow and PyTorch. Specifically, it analyzes changes in facial expressions, tone of voice, and posture to estimate emotions such as "smile" or "surprise." It uses facial feature data as input and generates emotion data as output.

[1027] Step 7:

[1028] Ad selection

[1029] Subject: Server

[1030] The server selects the optimal advertisement based on the analyzed viewer data and emotional data. It searches the database for an appropriate advertisement according to the viewer's attributes. It uses attribute data such as age, gender, and emotion as input and generates the selected advertisement data as output.

[1031] Step 8:

[1032] Listing advertising materials

[1033] Subject: Server

[1034] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates. Using the selected advertisement data as input, the server generates the listed advertisement material data as output.

[1035] Step 9:

[1036] Auto-generated personalized ads

[1037] Subject: Server

[1038] The server uses a generative AI model to automatically generate personalized advertising content based on the listed advertising materials. It generates advertising text and images using a generative AI model (such as GPT-3). It uses the listed advertising material data as input and obtains generated advertising content as output.

[1039] Step 10:

[1040] Sending advertising content

[1041] Subject: Server

[1042] The server sends the automatically generated advertising content to the terminal. Specifically, the server sends the generated advertising content to the terminal as an HTTP response. The server uses the generated advertising content as input and sends encoded data to the terminal as output.

[1043] Step 11:

[1044] Displaying ads

[1045] Subject: Device

[1046] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, the terminal displays advertising text and images on the display. The terminal uses the received advertising content as input and obtains the displayed advertisement as output.

[1047] Step 12:

[1048] Collecting viewer response data

[1049] Subject: Device

[1050] While the advertisement is being displayed, the device uses a camera and emotion engine to collect viewer response data. Eye-tracking technology is used to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions. The input is the video data during the advertisement display, and the output is the collected viewer response data.

[1051] Step 13:

[1052] Sending reaction data

[1053] Subject: Device

[1054] The device sends the collected viewer reaction data and emotion data to the server. Specifically, it encodes the collected data in JSON format and sends it as an HTTP request. It uses the collected reaction data as input and sends the encoded data to the server as output.

[1055] Step 14:

[1056] Receiving and analyzing data

[1057] Subject: Server

[1058] The server receives the reaction data and emotion data sent from the device and performs detailed analysis. It evaluates gaze duration and changes in facial expressions to quantify the effectiveness of the advertisement. It uses the received reaction data as input and obtains the quantified advertising effectiveness as output.

[1059] Step 15:

[1060] Updated advertising model

[1061] Subject: Server

[1062] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery. Specifically, the server reflects the newly obtained advertising effectiveness data as learning data in the model. The server uses the quantified advertising effectiveness data as input and obtains an updated advertising model as output.

[1063] This concludes the detailed explanation of the processing flow of the program for this system. These processing steps enable personalized advertisements to be provided to viewers in real time, maximizing their effectiveness.

[1064] (Application example 2)

[1065] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1066] Conventional advertising systems have difficulty delivering personalized ads that reflect the emotions and interests of individual users, making it difficult to maximize the effectiveness of advertising. Furthermore, while there is a need to analyze user behavior and emotional data in real time and apply the results immediately, there has been a lack of technology to solve this problem.

[1067] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1068] In this invention, the server includes means for collecting viewer data from terminals, means for analyzing the viewer data to estimate age, gender, and emotion, means for selecting appropriate advertisements based on the analyzed viewer data, means for automatically generating advertisements based on the selected advertising materials, means for transmitting the automatically generated advertisements to the terminals, means for displaying advertisements on the terminals, means for collecting viewer responses while the advertisements are being displayed, means for analyzing the effectiveness of the advertisements based on the collected response data, and means for analyzing user behavior, gaze data, and emotions and providing information about in-store products and discount coupons in real time. This allows advertisements optimized for individual users to be provided in real time, maximizing the effectiveness of the advertisements.

[1069] A "terminal" is a device that processes information and has functions such as collecting viewer data and displaying advertisements.

[1070] "Viewer data" refers to data that includes information such as the viewer's facial image, eye movements, and facial expressions, and is used to analyze emotions and attributes.

[1071] "Emotion" refers to the psychological state of the viewer that can be inferred from their facial expressions and gestures, including, for example, smile, surprise, interest, etc.

[1072] "Advertising materials" refers to content such as images, text, and videos used in advertising distribution.

[1073] "Auto-generating ads" refers to the process of automatically creating appropriate advertising content based on analyzed viewer data.

[1074] "Real-time" means that data collection, analysis, ad generation and delivery occur almost simultaneously.

[1075] "Behavioral data" refers to data including users' movements within a store, their line of sight, and the length of time they spend there.

[1076] A "discount coupon" is an electronic or paper voucher offering a special discount on the purchase of a product.

[1077] A "server" is a central computer system for receiving, analyzing, and processing data sent from terminals.

[1078] "Advertising effectiveness" refers to the results of evaluating viewers' reactions and purchasing intentions after viewing an advertisement.

[1079] To implement this invention, the following system configuration and processes are required: The system functions through cooperation between the terminal, the server, and the user.

[1080] System configuration

[1081] 1. Terminal

[1082] Hardware: Smart glasses (including camera, display, and communication module)

[1083] Software: OpenCV (for face recognition), emotion_engine (for emotion analysis)

[1084] 2. Server

[1085] Software: Data receiving module, data analysis module (viewer data analysis, emotion estimation), ad selection module, ad generation module, ad transmission module

[1086] 3. Users

[1087] An individual wearing smart glasses and moving around in a physical store

[1088] Program processing

[1089] 1. Collecting viewer data via devices

[1090] The device uses the camera installed in the smart glasses to collect real-time video data of the user, including facial images and eye movements (eye tracking).

[1091] 2. Preprocessing and analysis of viewer data

[1092] The device preprocesses the collected video data using OpenCV, detects and tracks faces, and transmits the detected and tracked face data to the server.

[1093] 3. Data analysis and ad selection by the server

[1094] The server receives the facial data sent from the device and uses the emotion_engine to estimate the viewer's attributes, such as age, gender, and emotions. Based on the analysis results, the server selects the most suitable advertisement. Specifically, it selects products and discount coupons that are likely to interest the viewer.

[1095] 4. Auto-generated ads

[1096] The server references a database of advertising materials provided by advertisers and uses AI algorithms to automatically generate personalized advertising content, which is then sent to the device.

[1097] 5. Display of advertisements by device

[1098] The terminal receives the advertising content sent from the server and displays it on the display of the smart glasses in real time.

[1099] 6. Measuring advertising effectiveness

[1100] While the ad is being displayed, the device uses a camera and an emotion engine to collect user reaction data, which is then sent to a server to measure the effectiveness of the ad.

[1101] 7. Analysis of advertising effectiveness and model update

[1102] The server analyzes the advertising effectiveness based on the response data sent from the terminal and updates the advertising model to reflect the results in the next advertisement distribution.

[1103] Specific examples

[1104] When a user wears smart glasses and visits a clothing store, the camera in the glasses tracks the user's gaze, and if the user stays in a particular section for a long time, the emotion engine analyzes the user's facial expressions of interest or joy. Based on this information, the server displays information about similar products and discount coupons on the smart glasses' display in real time.

[1105] Prompt Sentence Examples

[1106] "Get 20% off your favorite clothes today. Check out our new collection."

[1107] The above is a specific embodiment of the invention. This system allows users to receive optimized advertisements in real time, maximizing the effectiveness of the advertisements.

[1108] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1109] Step 1:

[1110] Collecting viewer data via devices

[1111] The device uses the camera installed in the smart glasses to collect the user's video data in real time, including the user's facial image, eye movements, and facial expressions. This video data is input and output as raw data captured by the camera.

[1112] Step 2:

[1113] Audience data preprocessing and facial recognition

[1114] The device preprocesses the collected video data using OpenCV to detect and track faces. Specifically, it analyzes the video data frame by frame to recognize the face, and then tracks the position of the face. The input for this process is the camera video data, and the output is data including the position information of the face.

[1115] Step 3:

[1116] Sending face data to the server

[1117] The device sends the detected and tracked face data to the server. The sent data includes face images, their location information, and changes over time. The input is data including face location information, and the output is a transmission completion status to the server.

[1118] Step 4:

[1119] Analysis of viewer data by the server

[1120] The server receives the facial data sent from the device and uses emotion_engine to estimate the viewer's attributes such as age, gender, and emotion. Specifically, it applies a facial recognition algorithm to the received data as input and performs analysis using an age estimation model, gender estimation model, and emotion estimation model. The input is facial data, and the output is the analysis results (e.g., "woman in her 20s," "smiling," etc.).

[1121] Step 5:

[1122] Selecting the right ads

[1123] The server selects suitable advertisements based on the analyzed viewer data. Here, it references the product database and the advertising material database provided by the advertiser to list advertisements that match the analysis results. The input is the analysis results of the viewer data, and the output is a list of selected advertisement candidates.

[1124] Step 6:

[1125] Auto-generated ads

[1126] The server automatically generates personalized advertising content using an AI algorithm based on the ad list obtained in the ad selection step. Specifically, it uses a generative AI model to combine advertising materials to create optimal advertising content. The input is the list of ad candidates, and the output is the generated advertising content.

[1127] Step 7:

[1128] Sending advertising content to devices

[1129] The server transmits automatically generated advertising content to the terminal, where the input is the generated advertising content and the output is a transmission completion status to the terminal.

[1130] Step 8:

[1131] Display of advertisements by device

[1132] The terminal receives the advertising content sent from the server and displays it on the smart glasses display in real time, where specific advertising content and discount coupons are displayed to the user. The input is the received advertising content, and the output is the advertisement displayed to the user.

[1133] Step 9:

[1134] Collecting user responses while ads are displayed

[1135] While the ad is being displayed, the device uses a camera and emotion engine to collect user response data (such as gaze, facial expressions, and visual characteristics). Specific operations include recording attention to the ad and changes in emotion. The input is the video data of the ad being displayed, and the output is user response data.

[1136] Step 10:

[1137] Sending reaction data to the server

[1138] The terminal sends the collected user reaction data to the server. The input is the user reaction data, and the output is the transmission completion status to the server.

[1139] Step 11:

[1140] Analysis of advertising effectiveness and model updating

[1141] The server analyzes the effectiveness of the advertisement based on the response data sent from the device and updates the advertising model to reflect this in the next ad delivery. Specifically, it uses an analysis algorithm to evaluate gaze time and changes in emotions. The input is the user's response data, and the output is an updated advertising model.

[1142] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1143] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1144] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1145] [Third embodiment]

[1146] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1147] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1148] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1149] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1150] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1151] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1152] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1153] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1154] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1155] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1156] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1157] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1158] The present invention relates to a system for displaying advertisements optimized for viewers in real time. The program processing of the entire system will be explained below in natural language.

[1159] Collection and analysis of user data

[1160] Subject: Device

[1161] The device uses a camera to collect video data of the viewer, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[1162] Subject: Server

[1163] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses a facial recognition model to analyze the image data and identify attributes such as "female in her 20s" or "smiling." The server also references data collected in the past to determine whether the viewer is the same person. This also makes it possible to identify repeat viewers.

[1164] Ad selection and automatic generation

[1165] Subject: Server

[1166] The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces." It then searches a database of advertising materials provided by the advertiser to create a list of the most suitable candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[1167] Ad display and effectiveness measurement

[1168] Subject: Device

[1169] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[1170] Subject: Server

[1171] While the advertisement is being displayed, the device collects viewer response data, which is then sent to a server to measure the effectiveness of the advertisement. The server then uses this data to analyze the success and impact of the advertisement and update the advertising model for further optimization.

[1172] Specific examples

[1173] Signage in a shopping mall

[1174] Subject: Server

[1175] The server collects and analyzes viewer data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[1176] Digital signage in public transport

[1177] Subject: Server

[1178] The server collects and analyzes viewer data from terminals installed on public transport station platforms. For example, if the analysis results indicate "men in their 30s," it selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response data is collected and the effectiveness of the advertisements is measured.

[1179] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[1180] The processing flow will be explained below.

[1181] Step 1:

[1182] Subject: Device

[1183] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[1184] Step 2:

[1185] Subject: Device

[1186] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[1187] Step 3:

[1188] Subject: Device

[1189] The terminal transmits the detected and tracked face data to the server.

[1190] Step 4:

[1191] Subject: Server

[1192] The server receives the face data sent from the terminal.

[1193] Step 5:

[1194] Subject: Server

[1195] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[1196] Step 6:

[1197] Subject: Server

[1198] The server references previously collected data and checks it against a database to identify whether the viewer is a repeat visitor.

[1199] Step 7:

[1200] Subject: Server

[1201] The server selects appropriate advertisements based on the analyzed viewer data.

[1202] Step 8:

[1203] Subject: Server

[1204] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[1205] Step 9:

[1206] Subject: Server

[1207] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[1208] Step 10:

[1209] Subject: Server

[1210] The server transmits the automatically generated advertising content to the terminal.

[1211] Step 11:

[1212] Subject: Device

[1213] The terminal receives the advertisement content transmitted from the server.

[1214] Step 12:

[1215] Subject: Device

[1216] The terminal displays the received advertising content on the display in real time, for example, an advertisement for a new fashion item.

[1217] Step 13:

[1218] Subject: Device

[1219] While the ad is being displayed, the device uses its camera to collect viewer response data, including eye tracking and facial expression analysis.

[1220] Step 14:

[1221] Subject: Device

[1222] The terminal transmits the collected viewer reaction data to the server.

[1223] Step 15:

[1224] Subject: Server

[1225] The server receives the response data sent from the terminal.

[1226] Step 16:

[1227] Subject: Server

[1228] The server analyzes the effectiveness of the advertisement based on the reaction data, specifically evaluating the length of time the viewer's gaze remains on the screen and changes in facial expression, and quantifies the effectiveness of the advertisement.

[1229] Step 17:

[1230] Subject: Server

[1231] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[1232] This allows viewers to receive ads that are optimized for them in real time, maximizing advertising effectiveness.

[1233] Example 1

[1234] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1235] Conventional advertising display systems have had difficulty effectively delivering advertisements to viewers. In particular, they have had issues with being unable to select the most appropriate advertisement based on the viewer's attributes and display it in real time. Furthermore, there are limitations to accurately measuring advertising effectiveness and reflecting this in future ad distribution. Furthermore, there are insufficient methods for collecting data to analyze how ads affect viewers.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1237] In this invention, the server includes means for preprocessing viewer data, detecting and tracking face data, means for estimating the viewer's age, gender, and emotion based on the face data, and means for selecting optimal advertisements based on the estimated viewer attributes. This makes it possible to display advertisements optimized for the viewer in real time, accurately measure the effectiveness of the advertisements, and reflect this in the next distribution.

[1238] A "terminal" is a device installed to collect data from viewers and transmit the collected data to a server.

[1239] "Viewer data" refers to viewer video data and other related data collected by a terminal via a camera or the like.

[1240] "Preprocessing" refers to processing such as image resizing and noise removal that is performed on collected video data.

[1241] "Face data" refers to data of the viewer's face regions detected and tracked from the pre-processed video data.

[1242] "Means for estimating age, gender, and emotion of a viewer based on facial data" refers to the processes and models within the server that analyze received facial data and estimate the age, gender, and emotional attributes of the target viewer.

[1243] "Viewer demographics" are data that refer to specific characteristics of viewers, such as their age, gender, and emotions.

[1244] The "means for selecting an advertisement" refers to the mechanisms and algorithms within the server for selecting the most suitable advertisement based on the estimated viewer attributes.

[1245] "Advertising materials" are material data such as images, text, and videos used to generate advertisements.

[1246] "Advertising content" refers to advertising content that is automatically generated based on selected advertising materials.

[1247] "Response data" refers to data such as the viewer's gaze direction and changes in facial expression that are collected while the advertisement is being displayed.

[1248] "Means for analyzing advertising effectiveness" refers to the processes and algorithms within the server that analyze the effectiveness and influence of displayed advertisements based on collected response data.

[1249] The present invention relates to a system for displaying advertisements optimized for viewers in real time, which operates through multiple steps involving terminals, servers, and users.

[1250] 1. Collection of User Data

[1251] Subject: Device

[1252] The device collects viewer video data using a camera. The camera is a commonly used hardware device, such as a USB or built-in camera. The video data is pre-processed using an image processing library such as OpenCV, which results in the detection and tracking of viewer facial data. This facial data is then sent to the server in real time.

[1253] 2. Data Preprocessing and Analysis

[1254] Subject: Device

[1255] The device performs preprocessing on the collected video data. This preprocessing includes image resizing and noise reduction. It uses libraries such as OpenCV to adjust the image quality of the video data and detects the viewer's face using a face detection model (e.g., Haar Cascades or MTCNN). The device then extracts the detected face area and sends the data to the server.

[1256] 3. Estimation of viewer attributes

[1257] Subject: Server

[1258] The server receives the facial data sent from the device. Based on the received data, it uses a facial recognition model (e.g., Dlib or FaceNet) to estimate the viewer's age, gender, and emotion. For example, it identifies attributes such as "female in her 20s" or "smiling." In this process, it extracts features from the facial data and uses a pre-trained model to make an estimation.

[1259] 4. Ad selection and automatic generation

[1260] Subject: Server

[1261] The server selects the most suitable advertisement based on the analysis results. For example, it selects advertisements for fashion brands or cosmetics based on the attributes "women in their 20s" and "smiling face." The server searches an advertising material database (e.g., cloud storage service) and automatically generates advertising content using deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The advertising content is generated in formats such as HTML5 or MP4.

[1262] 5. Submission of advertising content

[1263] Subject: Server

[1264] The server sends the generated advertising content to the device using real-time communication technology such as WebSocket. The server identifies the device's IP address and sends the advertising data to the endpoint.

[1265] 6. Display of advertisements

[1266] Subject: Device

[1267] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[1268] 7. Collecting viewer response data

[1269] Subject: Device

[1270] While the ad is being displayed, the device uses a camera to collect viewer response data, capturing gaze direction and facial expression changes in real time for visual analysis, which is then sent back to the server.

[1271] 8. Analysis of advertising effectiveness and feedback

[1272] Subject: Server

[1273] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. For example, if the viewer's gaze is frequently directed at the advertisement while viewing it, the advertisement is deemed to be highly successful. Based on this data, the server evaluates the effectiveness and influence of the advertisement and updates the model to help with future advertisement selection.

[1274] Specific examples

[1275] Signage in a shopping mall

[1276] Terminals installed at the entrance to the shopping mall collect viewer data. If the analysis results indicate a "woman in her early twenties," the server selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. The generated advertisements are sent to the terminals and displayed on displays within the mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[1277] Digital signage in public transport

[1278] Terminals installed on public transport station platforms collect viewer data. For example, if the analysis results show "men in their 30s," the server selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. The generated advertisements are sent to the terminals and displayed on digital signage on the station platforms. Viewer response data is collected during display and sent to the server, where the advertising effectiveness is measured.

[1279] Prompt Sentence Examples

[1280] "Select an ad for a tech gadget for a man in his 30s and generate and display an ad for the latest smartphone."

[1281] keyword

[1282] Generative AI model, prompt sentence

[1283] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1284] Step 1:

[1285] Subject: Device

[1286] The device uses a camera to collect video data of the viewer. This video data is captured from the camera in real time. Specifically, it stores frame data continuously acquired from the camera in memory and performs image preprocessing using the OpenCV library. Preprocessing includes image resizing and noise removal. The input is raw video data, and the output is preprocessed video data.

[1287] Step 2:

[1288] Subject: Device

[1289] The device detects and tracks the viewer's facial data from the preprocessed video data. Specifically, it identifies the facial region using a face detection model such as OpenCV, Haar Cascades, or MTCNN. The input is the preprocessed video data, and the output is the viewer's facial region data. This data is sent to the server for further processing.

[1290] Step 3:

[1291] Subject: Server

[1292] The server receives the face region data sent from the device. Based on the received data, it uses a face recognition model such as Dlib or FaceNet to estimate the viewer's age, gender, and emotion. The input is the face region data, and the output is the estimated viewer attributes (e.g., "female in her 20s" and "smiling"). Specifically, it extracts features and analyzes them using a pre-trained model.

[1293] Step 4:

[1294] Subject: Server

[1295] The server selects the most suitable advertisement based on the viewer attributes obtained as a result of the analysis. As a specific example, it searches an advertising material database to select advertisements for fashion brands and cosmetics for the attributes "women in their 20s" and "smiling faces." The input is the estimated viewer attributes, and the output is the most suitable advertising material.

[1296] Step 5:

[1297] Subject: Server

[1298] The server automatically generates advertising content based on the selected advertising materials. This process uses deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The input is the selected advertising materials, and the output is the generated advertising content. Specifically, the server creates advertisements in HTML5 or MP4 format based on the material data.

[1299] Step 6:

[1300] Subject: Server

[1301] The server transmits the generated advertising content to the device using real-time communication technology such as WebSocket. The input is the generated advertising content, and the output is data transmission to the device.

[1302] Step 7:

[1303] Subject: Device

[1304] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, an advertisement for a new fashion item is displayed on a display device connected to the terminal. The input is the received advertising content, and the output is the display on which the advertisement is displayed.

[1305] Step 8:

[1306] Subject: Device

[1307] While the ad is being displayed, the device uses a camera to collect viewer response data. Specifically, it captures changes in gaze direction and facial expressions. The input is the displayed ad content, and the output is viewer response data. This data is then sent back to the server.

[1308] Step 9:

[1309] Subject: Server

[1310] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. Specifically, if the viewer's gaze is frequently directed at the advertisement, the success of the advertisement is evaluated highly. The input is the viewer response data, and the output is the analysis result of the advertisement's effectiveness. The server updates the advertising model based on this analysis result and reflects it in subsequent advertisement selections.

[1311] (Application example 1)

[1312] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1313] Conventional advertising display systems did not fully utilize viewer information, making it difficult to select the optimal advertisement to maximize advertising effectiveness. Furthermore, they lacked the technology to collect viewer response information in real time and immediately reflect it in advertising strategies. As a result, advertisements that were not appealing to viewers were displayed, reducing advertising effectiveness.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1315] In this invention, the server includes means for collecting viewer data from the terminal, means for analyzing the viewer data and estimating age, gender, and emotion, means for selecting an appropriate advertisement based on the analyzed viewer data, means for automatically generating an advertisement based on the selected advertising material, means for transmitting the automatically generated advertisement to the terminal, means for displaying the advertisement on the terminal, means for collecting viewer responses while the advertisement is being displayed, means for analyzing the effectiveness of the advertisement based on the collected response data, means for collecting and analyzing video of the viewer using a camera mounted on the smart glasses, and means for displaying the advertisement on the display of the smart glasses. This enables real-time display of personalized advertisements tailored to the characteristics of the viewer and rapid feedback of the effectiveness of the advertisement.

[1316] "Terminal" refers to a device used to collect viewer data and display advertisements, including smartphones, tablets, smart glasses, etc.

[1317] "Viewer data" refers to data that includes information such as the viewer's age, gender, and emotions, and is collected through cameras and sensors.

[1318] "Analysis" is the process of estimating attributes such as age, gender, and emotions based on collected viewer data.

[1319] "Ad selection" is the process of selecting the most suitable advertisement based on analyzed viewer data.

[1320] "Advertising materials" are materials such as databases, images, videos, and text used to generate advertisements.

[1321] "Automatic generation" is the process of automatically creating advertising content based on advertising materials after an advertisement has been selected.

[1322] "Advertising effectiveness" is an indicator that evaluates the impact that an advertisement has on viewers, and is analyzed based on viewer response data.

[1323] "Smart glasses" are wearable devices equipped with a camera and a display that can collect video data from viewers and display advertisements.

[1324] A "camera" is a device that collects video data from viewers and is installed in smart glasses or terminals.

[1325] A "display" is a display device that displays the generated advertisement to the viewer, and is installed in a terminal or smart glasses.

[1326] "Response data" refers to data such as the viewer's gaze, facial expressions, and movements while the advertisement is being displayed.

[1327] This invention relates to a system for displaying advertisements optimized for viewers in real time and measuring their effectiveness. This system is composed of the following components:

[1328] System Configuration

[1329] 1. Hardware

[1330] Devices: Includes smartphones, tablets, smart glasses, etc.

[1331] Camera: A device installed in smart glasses or a device that collects video data of the viewer.

[1332] Display: A device mounted on the smart glasses or device that displays the generated advertisements.

[1333] 2. Software

[1334] Facial recognition technology: For example, OpenCV is used.

[1335] Data analysis platform: For example, AWS Lambda is used.

[1336] Ad selection algorithm: For example, TensorFlow is used.

[1337] System processing procedure

[1338] Collection and analysis of user data

[1339] The device collects the viewer's video data, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[1340] Ad selection and automatic generation

[1341] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotions. To do this, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling." The server then references previously collected data to determine whether the viewer is the same person. The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on the attributes "female in her 20s" and "smiling." It then searches a database of advertising materials provided by the advertiser and lists the most suitable advertising candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[1342] Ad display and effectiveness measurement

[1343] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be displayed on the display. While the advertisement is being displayed, the device collects viewer response data. The collected response data is sent to the server, where the effectiveness of the advertisement is measured. The server uses this data to analyze the success and influence of the advertisement and update the advertising model for further optimization.

[1344] Specific examples

[1345] Signage in a shopping mall

[1346] When a user is wearing smart glasses while walking through a shopping mall, the glasses collect the user's facial data in real time and send it to a server. Based on this data, the server determines that the user is a woman in her early twenties and selects advertisements for fashion brands and accessories. Advertisements for new collections are automatically generated and instantly displayed on the smart glasses' display. The user's reactions are also collected again, and the effectiveness of the advertisements is measured.

[1347] Digital signage in public transport

[1348] Terminals installed on public transport station platforms collect the viewer's facial data and send it to a server. If the user is identified as a "male in his 30s," ads for technology gadgets and sports equipment are selected, and the latest smartphone ads are automatically generated. These ads are then displayed on digital signage on the station platforms. Viewer response data is collected in the same way, and the effectiveness of the ads is measured.

[1349] Prompt Sentence Examples

[1350] "Woman in her 20s, smiling. Generate an ad for new fashion items. Also consider the option to show an ad for your summer accessories collection instead."

[1351] "Based on facial recognition data, we've identified her as a woman in her 20s. Then generate ads for a fashion brand that would be suitable for her. Also, create content that reflects positive emotions, taking into account smiling faces."

[1352] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[1353] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1354] Step 1:

[1355] User Data Collection

[1356] Input: Video data from a camera mounted on smart glasses

[1357] Processing: The device uses the camera to collect the user's video data. The video data is captured in real time and pre-processed (resolution adjustment, noise reduction).

[1358] Output: Pre-processed video data

[1359] Step 2:

[1360] Face Detection and Tracking

[1361] Input: Preprocessed video data

[1362] Processing: The device uses OpenCV to detect faces in the video data and perform face tracking. The face detection algorithm identifies the position of the face within the video frame.

[1363] Output: Face data (face position, facial feature points, etc.)

[1364] Step 3:

[1365] Data transmission

[1366] Input: Face data

[1367] Processing: The device sends the detected face data to the server in real time using the secure HTTP(S) protocol.

[1368] Output: Face data sent to the server

[1369] Step 4:

[1370] Face Recognition and Attribute Estimation

[1371] Input: Face data sent to the server

[1372] Processing: The server analyzes the facial data using facial recognition technology (e.g., a facial recognition model) to estimate age, gender, and emotion. Analysis includes extracting facial features and estimating attributes using a classifier.

[1373] Output: Viewer attribute data (e.g., "Female in her 20s," "Smiling," etc.)

[1374] Step 5:

[1375] Repeater Identification

[1376] Input: Viewer demographic data

[1377] Processing: The server refers to a database of previously collected data and compares it with the viewer's facial features to identify whether they are repeat visitors. For this purpose, it uses an approximate nearest neighbor search algorithm.

[1378] Output: Repeater identification result

[1379] Step 6:

[1380] Ad selection

[1381] Input: Viewer attribute data, repeater identification results

[1382] Processing: The server selects the optimal advertisement using an advertisement selection algorithm (for example, an algorithm using TensorFlow) based on the viewer attribute data and the results of repeat customer identification.

[1383] Output: Ad candidate list

[1384] Step 7:

[1385] Auto-generated ads

[1386] Input: Ad candidate list

[1387] Processing: The server automatically generates optimal advertising content from a selected list of ad candidates based on the advertising material database provided by the advertiser. For generation, a generative AI model is used.

[1388] Output: Generated ad content

[1389] Step 8:

[1390] Sending Ads

[1391] Input: Generated ad content

[1392] Processing: The server sends the generated advertising content to the device using secure HTTP(S) as the communication protocol.

[1393] Output: Ad content sent to the device

[1394] Step 9:

[1395] Displaying ads

[1396] Input: Ad content sent to the device

[1397] Processing: The device displays the received advertising content on the display in real time. The advertisement is displayed to the user using the display of the smart glasses.

[1398] Output: The ad shown to the user

[1399] Step 10:

[1400] Collecting viewer responses

[1401] Input: The ad shown to the user

[1402] Processing: The device collects user response data (eye gaze, facial expressions, movements, etc.) while the ad is being displayed. The device uses cameras and sensors to capture response data in real time.

[1403] Output: Collected reaction data

[1404] Step 11:

[1405] Sending reaction data

[1406] Input: Collected reaction data

[1407] Processing: The device sends the collected reaction data to the server using secure HTTP(S).

[1408] Output: Response data sent to the server

[1409] Step 12:

[1410] Measuring advertising effectiveness

[1411] Input: Reaction data sent to the server

[1412] Processing: The server analyzes the effectiveness of the ads based on the collected response data, using statistical analysis and machine learning models.

[1413] Output: Advertising effectiveness data

[1414] Step 13:

[1415] Updated advertising model

[1416] Input: Advertising effectiveness data

[1417] Processing: The server updates the advertising model based on the advertising effectiveness data. The updated model is reflected in the next ad delivery.

[1418] Output: Updated ad model

[1419] The above are the specific processing steps for carrying out the present invention.

[1420] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1421] The present invention combines a system that displays advertisements optimized for each viewer in real time with an emotion engine that recognizes the user's emotions. Below, the program processing of this entire system is explained in natural language.

[1422] Collection and analysis of user data

[1423] Subject: Device

[1424] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[1425] Subject: Device

[1426] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[1427] Subject: Device

[1428] The terminal transmits the detected and tracked face data to the server.

[1429] Subject: Server

[1430] The server receives the face data sent from the terminal.

[1431] Subject: Server

[1432] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[1433] Subject: Server

[1434] The server uses an emotion engine to analyze the user's facial expressions, voice, posture, and other characteristics to estimate their emotions. For example, it can recognize emotions such as smile, surprise, and interest.

[1435] Ad selection and automatic generation

[1436] Subject: Server

[1437] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces."

[1438] Subject: Server

[1439] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[1440] Subject: Server

[1441] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[1442] Subject: Server

[1443] The automatically generated advertising content is transmitted to the terminal.

[1444] Ad display and effectiveness measurement

[1445] Subject: Device

[1446] The device receives the advertising content sent from the server and displays it on the display in real time, for example, advertising new fashion items.

[1447] Subject: Device

[1448] While the ad is being displayed, the device uses its camera and emotion engine to collect viewer response data, including eye tracking and facial expression analysis, to measure whether the viewer is paying attention to the ad.

[1449] Subject: Device

[1450] The terminal transmits the collected viewer reaction data and emotion data to the server.

[1451] Subject: Server

[1452] The server receives the reaction data and emotion data transmitted from the terminal.

[1453] Subject: Server

[1454] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. Specifically, it evaluates the length of time the viewer's gaze remains on the screen and changes in emotions, and quantifies the effectiveness of the advertisement.

[1455] Subject: Server

[1456] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[1457] Specific examples

[1458] Signage in a shopping mall

[1459] Subject: Server

[1460] The server collects and analyzes viewer and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate a "woman in her early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. Meanwhile, viewer response and emotional data are collected again, and the effectiveness of the advertisements is measured.

[1461] Digital signage in public transport

[1462] Subject: Server

[1463] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a "male in his 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response and emotional data are collected, and the effectiveness of the advertisements is measured.

[1464] The above is a specific embodiment for implementing the invention of a system incorporating an emotion engine, which allows advertisements optimized for viewers to be provided in real time, maximizing the effectiveness of the advertisements.

[1465] The processing flow will be explained below.

[1466] Step 1:

[1467] Subject: Device

[1468] The device uses a camera to collect real-time video data of viewers, including facial images, eye movements, and facial expressions.

[1469] Step 2:

[1470] Subject: Device

[1471] The device pre-processes the collected video data and performs face detection and tracking for facial recognition, specifically by using a specific algorithm to detect the position of the face and track its movement.

[1472] Step 3:

[1473] Subject: Device

[1474] The device sends face data to the server, including face position information and tracking data.

[1475] Step 4:

[1476] Subject: Server

[1477] The server receives the face data sent from the terminal.

[1478] Step 5:

[1479] Subject: Server

[1480] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Specifically, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling."

[1481] Step 6:

[1482] Subject: Server

[1483] The server uses an emotion engine to analyze the viewer's facial expressions, voice, posture, and other characteristics to estimate their emotions in real time, for example, determining whether they are smiling, surprised, or interested.

[1484] Step 7:

[1485] Subject: Server

[1486] The server references previously collected viewer data and checks it against a database to identify repeat viewers.

[1487] Step 8:

[1488] Subject: Server

[1489] The server then selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands or cosmetics that match the attributes of "women in their 20s" and "smiling faces."

[1490] Step 9:

[1491] Subject: Server

[1492] The server searches an advertising material database provided by the advertiser and lists suitable advertisements as candidates.

[1493] Step 10:

[1494] Subject: Server

[1495] The server then uses AI and algorithms to automatically generate personalized advertising content based on the listed advertising materials, a process that involves combining images, text, and videos.

[1496] Step 11:

[1497] Subject: Server

[1498] The automatically generated advertising content is transmitted to the terminal.

[1499] Step 12:

[1500] Subject: Device

[1501] The terminal receives the advertisement content transmitted from the server.

[1502] Step 13:

[1503] Subject: Device

[1504] The terminal displays the received advertising content on the display in real time. For example, an advertisement for "new fashion items" is displayed on the display.

[1505] Step 14:

[1506] Subject: Device

[1507] While the ad is being displayed, the device uses a camera and emotion engine to collect viewer response data, specifically tracking eye movements and analyzing facial expressions to record changes in viewer attention and emotions.

[1508] Step 15:

[1509] Subject: Device

[1510] The terminal transmits the collected viewer reaction data and emotion data to the server.

[1511] Step 16:

[1512] Subject: Server

[1513] The server receives the reaction data and emotion data transmitted from the terminal.

[1514] Step 17:

[1515] Subject: Server

[1516] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. For example, it quantifies the time the viewer pays attention to the advertisement and changes in their emotions to evaluate the success and impact of the advertisement.

[1517] Step 18:

[1518] Subject: Server

[1519] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[1520] That's all. This system, combined with an emotion engine, makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[1521] Example 2

[1522] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1523] Conventional advertising display systems have difficulty delivering personalized ads to viewers in real time, making it difficult to maximize advertising effectiveness. They also lack the means to accurately analyze viewers' emotions and reactions and dynamically optimize ads based on that analysis. This has prevented advertisers from delivering optimal ads to viewers, resulting in reduced advertising effectiveness.

[1524] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving viewer data transmitted from a terminal, means for estimating the viewer's age, gender, and emotion using face recognition technology, means for analyzing the user's facial expression, voice, and posture using an emotion engine to estimate emotion, means for selecting an optimal advertisement based on the analyzed viewer data and emotion data, means for referencing an advertising material database and listing appropriate advertisement candidates, means for automatically generating personalized advertising content using a generative AI model, and means for receiving response data and emotion data transmitted from the terminal and analyzing advertising effectiveness. This makes it possible to provide viewers with optimized advertisements in real time, maximizing advertising effectiveness.

[1525] A "terminal" is a hardware device for collecting and processing viewer video data.

[1526] A "server" is a computer system that receives data sent from a terminal, analyzes it, and generates advertisements.

[1527] "Viewer data" refers to data that includes information about the viewer, such as an image of the viewer's face and eye movements.

[1528] "Facial recognition technology" is a technology for detecting human faces from video data and estimating their attributes.

[1529] The "emotion engine" is software that analyzes and estimates emotions from a user's facial expressions, voice, and posture.

[1530] The "advertising material database" is a database for managing various advertising materials provided by advertisers.

[1531] A "generative AI model" is an artificial intelligence model that automatically generates advertising content based on input data.

[1532] "Response data" refers to data related to viewers' reactions to advertisements, such as eye tracking and facial expression analysis.

[1533] "Advertising effectiveness" refers to the effectiveness of an advertisement as evaluated based on viewer reactions and emotional data.

[1534] An "advertising model" is a set of algorithms or rules used to maximize the effectiveness of advertising.

[1535] This invention is a system that provides viewers with advertisements optimized for them in real time, and combines it with an emotion engine that recognizes the user's emotions. This system is composed of multiple elements, including a terminal, a server, and an emotion engine, and these elements work together to maximize the effectiveness of advertisements.

[1536] Collection and analysis of user data

[1537] Subject: Device

[1538] The device is equipped with a camera to collect video data of viewers. The video data collected using the camera includes images of the viewer's face and eye movements. The device uses libraries such as OpenCV to detect and track faces from the collected video data. The detected and tracked face data is sent to the server in JSON format or protocol buffers.

[1539] Subject: Server

[1540] The server receives the facial data sent from the device. Based on the received facial data, it uses facial recognition technology such as dlib to estimate the viewer's age, gender, and emotion. The server then uses an emotion engine (for example, a model trained with TensorFlow or PyTorch) to analyze the user's facial expression, voice, and posture to estimate their emotion. Specifically, it recognizes emotions such as smile, surprise, and interest.

[1541] Ad selection and automatic generation

[1542] Subject: Server

[1543] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetic products based on attributes such as "women in their 20s" and "smiling faces." It references a database of advertising materials provided by the advertiser to create a list of suitable advertisement candidates, and then automatically generates personalized advertising content using a generative AI model (such as GPT-3). The generated advertising content is then sent to the device.

[1544] Ad display and effectiveness measurement

[1545] Subject: Device

[1546] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be shown on the display. While the advertisement is being displayed, the device uses a camera and an emotion engine to collect viewer reaction data. Specifically, it uses eye-tracking technology to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions.

[1547] Subject: Server

[1548] The server receives the reaction and emotion data sent from the device and performs detailed analysis. It evaluates factors such as gaze duration and changes in facial expression to quantify advertising effectiveness. The advertising model is updated based on the analysis results and reflected in the next ad delivery. This maximizes advertising effectiveness and enables more effective ad delivery.

[1549] Specific examples

[1550] Signage in a shopping mall

[1551] Subject: Server

[1552] The server collects and analyzes viewer data and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays within the shopping mall. While the advertisements are displayed, the terminals collect viewer response data and emotional data, which are then sent back to the server to measure the effectiveness of the advertisements.

[1553] Example prompt sentence:

[1554] A video shows a woman in her early twenties at the entrance of a shopping mall. She is smiling and seems interested in the new fashion items. Generate the best ad for her.

[1555] Digital signage in public transport

[1556] Subject: Server

[1557] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a target audience of "men in their 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are then sent to the terminals and displayed on digital signage on the station platforms. At the same time, the terminals collect viewer response and emotional data, which are then sent to the server to measure the effectiveness of the advertisements.

[1558] Example prompt sentence:

[1559] A video shows a man in his 30s on a train platform. He seems interested in technology gadgets. Generate an ad for the latest smartphone that is perfect for him.

[1560] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[1561] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1562] Step 1:

[1563] Video data collection

[1564] Subject: Device

[1565] The device uses a camera to collect real-time video data of the viewer, including facial images and eye movements. Specifically, the device captures the camera's video stream and captures multiple frames per second. The captured frames are stored in a buffer and sent to the next processing step.

[1566] Step 2:

[1567] Face Detection and Tracking

[1568] Subject: Device

[1569] The device detects and tracks faces from the collected video data. It uses OpenCV's face recognition algorithm to detect faces. Specifically, it detects the position of a face in each frame and tracks the same face in the next frame. It uses the acquired video frame as input and generates coordinate data of the detected face as output.

[1570] Step 3:

[1571] Sending data

[1572] Subject: Device

[1573] The device sends the detected and tracked face data to the server. The data is sent in JSON format and is encoded before being sent to the server. It uses the detected face coordinate data as input and generates encoded JSON data as output, which is sent to the server.

[1574] Step 4:

[1575] Receiving data

[1576] Subject: Server

[1577] The server receives the face data sent from the device. Specifically, the server receives an HTTP request and parses the JSON data. It uses the received JSON data as input and obtains parsed face coordinate data as output.

[1578] Step 5:

[1579] Face Recognition and Attribute Estimation

[1580] Subject: Server

[1581] The server uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses dlib's facial recognition model to analyze the viewer's facial features. It uses the received facial coordinate data as input and generates attribute data such as age, gender, and emotion as output.

[1582] Step 6:

[1583] Emotion Analysis

[1584] Subject: Server

[1585] The server uses an emotion engine to analyze the user's facial expressions, voice, and posture to estimate their emotions. Emotion analysis is performed using a model using TensorFlow and PyTorch. Specifically, it analyzes changes in facial expressions, tone of voice, and posture to estimate emotions such as "smile" or "surprise." It uses facial feature data as input and generates emotion data as output.

[1586] Step 7:

[1587] Ad selection

[1588] Subject: Server

[1589] The server selects the optimal advertisement based on the analyzed viewer data and emotional data. It searches the database for an appropriate advertisement according to the viewer's attributes. It uses attribute data such as age, gender, and emotion as input and generates the selected advertisement data as output.

[1590] Step 8:

[1591] Listing advertising materials

[1592] Subject: Server

[1593] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates. Using the selected advertisement data as input, the server generates the listed advertisement material data as output.

[1594] Step 9:

[1595] Auto-generated personalized ads

[1596] Subject: Server

[1597] The server uses a generative AI model to automatically generate personalized advertising content based on the listed advertising materials. It generates advertising text and images using a generative AI model (such as GPT-3). It uses the listed advertising material data as input and obtains generated advertising content as output.

[1598] Step 10:

[1599] Sending advertising content

[1600] Subject: Server

[1601] The server sends the automatically generated advertising content to the terminal. Specifically, the server sends the generated advertising content to the terminal as an HTTP response. The server uses the generated advertising content as input and sends encoded data to the terminal as output.

[1602] Step 11:

[1603] Displaying ads

[1604] Subject: Device

[1605] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, the terminal displays advertising text and images on the display. The terminal uses the received advertising content as input and obtains the displayed advertisement as output.

[1606] Step 12:

[1607] Collecting viewer response data

[1608] Subject: Device

[1609] While the advertisement is being displayed, the device uses a camera and emotion engine to collect viewer response data. Eye-tracking technology is used to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions. The input is the video data during the advertisement display, and the output is the collected viewer response data.

[1610] Step 13:

[1611] Sending reaction data

[1612] Subject: Device

[1613] The device sends the collected viewer reaction data and emotion data to the server. Specifically, it encodes the collected data in JSON format and sends it as an HTTP request. It uses the collected reaction data as input and sends the encoded data to the server as output.

[1614] Step 14:

[1615] Receiving and analyzing data

[1616] Subject: Server

[1617] The server receives the reaction data and emotion data sent from the device and performs detailed analysis. It evaluates gaze duration and changes in facial expressions to quantify the effectiveness of the advertisement. It uses the received reaction data as input and obtains the quantified advertising effectiveness as output.

[1618] Step 15:

[1619] Updated advertising model

[1620] Subject: Server

[1621] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery. Specifically, the server reflects the newly obtained advertising effectiveness data as learning data in the model. The server uses the quantified advertising effectiveness data as input and obtains an updated advertising model as output.

[1622] This concludes the detailed explanation of the processing flow of the program for this system. These processing steps enable personalized advertisements to be provided to viewers in real time, maximizing their effectiveness.

[1623] (Application example 2)

[1624] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1625] Conventional advertising systems have difficulty delivering personalized ads that reflect the emotions and interests of individual users, making it difficult to maximize the effectiveness of advertising. Furthermore, while there is a need to analyze user behavior and emotional data in real time and apply the results immediately, there has been a lack of technology to solve this problem.

[1626] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1627] In this invention, the server includes means for collecting viewer data from terminals, means for analyzing the viewer data to estimate age, gender, and emotion, means for selecting appropriate advertisements based on the analyzed viewer data, means for automatically generating advertisements based on the selected advertising materials, means for transmitting the automatically generated advertisements to the terminals, means for displaying advertisements on the terminals, means for collecting viewer responses while the advertisements are being displayed, means for analyzing the effectiveness of the advertisements based on the collected response data, and means for analyzing user behavior, gaze data, and emotions and providing information about in-store products and discount coupons in real time. This allows advertisements optimized for individual users to be provided in real time, maximizing the effectiveness of the advertisements.

[1628] A "terminal" is a device that processes information and has functions such as collecting viewer data and displaying advertisements.

[1629] "Viewer data" refers to data that includes information such as the viewer's facial image, eye movements, and facial expressions, and is used to analyze emotions and attributes.

[1630] "Emotion" refers to the psychological state of the viewer that can be inferred from their facial expressions and gestures, including, for example, smile, surprise, interest, etc.

[1631] "Advertising materials" refers to content such as images, text, and videos used in advertising distribution.

[1632] "Auto-generating ads" refers to the process of automatically creating appropriate advertising content based on analyzed viewer data.

[1633] "Real-time" means that data collection, analysis, ad generation and delivery occur almost simultaneously.

[1634] "Behavioral data" refers to data including users' movements within a store, their line of sight, and the length of time they spend there.

[1635] A "discount coupon" is an electronic or paper voucher offering a special discount on the purchase of a product.

[1636] A "server" is a central computer system for receiving, analyzing, and processing data sent from terminals.

[1637] "Advertising effectiveness" refers to the results of evaluating viewers' reactions and purchasing intentions after viewing an advertisement.

[1638] To implement this invention, the following system configuration and processes are required: The system functions through cooperation between the terminal, the server, and the user.

[1639] System configuration

[1640] 1. Terminal

[1641] Hardware: Smart glasses (including camera, display, and communication module)

[1642] Software: OpenCV (for face recognition), emotion_engine (for emotion analysis)

[1643] 2. Server

[1644] Software: Data receiving module, data analysis module (viewer data analysis, emotion estimation), ad selection module, ad generation module, ad transmission module

[1645] 3. Users

[1646] An individual wearing smart glasses and moving around in a physical store

[1647] Program processing

[1648] 1. Collecting viewer data via devices

[1649] The device uses the camera installed in the smart glasses to collect real-time video data of the user, including facial images and eye movements (eye tracking).

[1650] 2. Preprocessing and analysis of viewer data

[1651] The device preprocesses the collected video data using OpenCV, detects and tracks faces, and transmits the detected and tracked face data to the server.

[1652] 3. Data analysis and ad selection by the server

[1653] The server receives the facial data sent from the device and uses the emotion_engine to estimate the viewer's attributes, such as age, gender, and emotions. Based on the analysis results, the server selects the most suitable advertisement. Specifically, it selects products and discount coupons that are likely to interest the viewer.

[1654] 4. Auto-generated ads

[1655] The server references a database of advertising materials provided by advertisers and uses AI algorithms to automatically generate personalized advertising content, which is then sent to the device.

[1656] 5. Display of advertisements by device

[1657] The terminal receives the advertising content sent from the server and displays it on the display of the smart glasses in real time.

[1658] 6. Measuring advertising effectiveness

[1659] While the ad is being displayed, the device uses a camera and an emotion engine to collect user reaction data, which is then sent to a server to measure the effectiveness of the ad.

[1660] 7. Analysis of advertising effectiveness and model update

[1661] The server analyzes the advertising effectiveness based on the response data sent from the terminal and updates the advertising model to reflect the results in the next advertisement distribution.

[1662] Specific examples

[1663] When a user wears smart glasses and visits a clothing store, the camera in the glasses tracks the user's gaze, and if the user stays in a particular section for a long time, the emotion engine analyzes the user's facial expressions of interest or joy. Based on this information, the server displays information about similar products and discount coupons on the smart glasses' display in real time.

[1664] Prompt Sentence Examples

[1665] "Get 20% off your favorite clothes today. Check out our new collection."

[1666] The above is a specific embodiment of the invention. This system allows users to receive optimized advertisements in real time, maximizing the effectiveness of the advertisements.

[1667] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1668] Step 1:

[1669] Collecting viewer data via devices

[1670] The device uses the camera installed in the smart glasses to collect the user's video data in real time, including the user's facial image, eye movements, and facial expressions. This video data is input and output as raw data captured by the camera.

[1671] Step 2:

[1672] Audience data preprocessing and facial recognition

[1673] The device preprocesses the collected video data using OpenCV to detect and track faces. Specifically, it analyzes the video data frame by frame to recognize the face, and then tracks the position of the face. The input for this process is the camera video data, and the output is data including the position information of the face.

[1674] Step 3:

[1675] Sending face data to the server

[1676] The device sends the detected and tracked face data to the server. The sent data includes face images, their location information, and changes over time. The input is data including face location information, and the output is a transmission completion status to the server.

[1677] Step 4:

[1678] Analysis of viewer data by the server

[1679] The server receives the facial data sent from the device and uses emotion_engine to estimate the viewer's attributes such as age, gender, and emotion. Specifically, it applies a facial recognition algorithm to the received data as input and performs analysis using an age estimation model, gender estimation model, and emotion estimation model. The input is facial data, and the output is the analysis results (e.g., "woman in her 20s," "smiling," etc.).

[1680] Step 5:

[1681] Selecting the right ads

[1682] The server selects suitable advertisements based on the analyzed viewer data. Here, it references the product database and the advertising material database provided by the advertiser to list advertisements that match the analysis results. The input is the analysis results of the viewer data, and the output is a list of selected advertisement candidates.

[1683] Step 6:

[1684] Auto-generated ads

[1685] The server automatically generates personalized advertising content using an AI algorithm based on the ad list obtained in the ad selection step. Specifically, it uses a generative AI model to combine advertising materials to create optimal advertising content. The input is the list of ad candidates, and the output is the generated advertising content.

[1686] Step 7:

[1687] Sending advertising content to devices

[1688] The server transmits automatically generated advertising content to the terminal, where the input is the generated advertising content and the output is a transmission completion status to the terminal.

[1689] Step 8:

[1690] Display of advertisements by device

[1691] The terminal receives the advertising content sent from the server and displays it on the smart glasses display in real time, where specific advertising content and discount coupons are displayed to the user. The input is the received advertising content, and the output is the advertisement displayed to the user.

[1692] Step 9:

[1693] Collecting user responses while ads are displayed

[1694] While the ad is being displayed, the device uses a camera and emotion engine to collect user response data (such as gaze, facial expressions, and visual characteristics). Specific operations include recording attention to the ad and changes in emotion. The input is the video data of the ad being displayed, and the output is user response data.

[1695] Step 10:

[1696] Sending reaction data to the server

[1697] The terminal sends the collected user reaction data to the server. The input is the user reaction data, and the output is the transmission completion status to the server.

[1698] Step 11:

[1699] Analysis of advertising effectiveness and model updating

[1700] The server analyzes the effectiveness of the advertisement based on the response data sent from the device and updates the advertising model to reflect this in the next ad delivery. Specifically, it uses an analysis algorithm to evaluate gaze time and changes in emotions. The input is the user's response data, and the output is an updated advertising model.

[1701] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1702] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1703] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1704] [Fourth embodiment]

[1705] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1706] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1707] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1708] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1709] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1710] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1711] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1712] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1713] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1714] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1715] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1716] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1717] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1718] The present invention relates to a system for displaying advertisements optimized for viewers in real time. The program processing of the entire system will be explained below in natural language.

[1719] Collection and analysis of user data

[1720] Subject: Device

[1721] The device uses a camera to collect video data of the viewer, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[1722] Subject: Server

[1723] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses a facial recognition model to analyze the image data and identify attributes such as "female in her 20s" or "smiling." The server also references data collected in the past to determine whether the viewer is the same person. This also makes it possible to identify repeat viewers.

[1724] Ad selection and automatic generation

[1725] Subject: Server

[1726] The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces." It then searches a database of advertising materials provided by the advertiser to create a list of the most suitable candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[1727] Ad display and effectiveness measurement

[1728] Subject: Device

[1729] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[1730] Subject: Server

[1731] While the advertisement is being displayed, the device collects viewer response data, which is then sent to a server to measure the effectiveness of the advertisement. The server then uses this data to analyze the success and impact of the advertisement and update the advertising model for further optimization.

[1732] Specific examples

[1733] Signage in a shopping mall

[1734] Subject: Server

[1735] The server collects and analyzes viewer data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[1736] Digital signage in public transport

[1737] Subject: Server

[1738] The server collects and analyzes viewer data from terminals installed on public transport station platforms. For example, if the analysis results indicate "men in their 30s," it selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response data is collected and the effectiveness of the advertisements is measured.

[1739] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[1740] The processing flow will be explained below.

[1741] Step 1:

[1742] Subject: Device

[1743] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[1744] Step 2:

[1745] Subject: Device

[1746] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[1747] Step 3:

[1748] Subject: Device

[1749] The terminal transmits the detected and tracked face data to the server.

[1750] Step 4:

[1751] Subject: Server

[1752] The server receives the face data sent from the terminal.

[1753] Step 5:

[1754] Subject: Server

[1755] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[1756] Step 6:

[1757] Subject: Server

[1758] The server references previously collected data and checks it against a database to identify whether the viewer is a repeat visitor.

[1759] Step 7:

[1760] Subject: Server

[1761] The server selects appropriate advertisements based on the analyzed viewer data.

[1762] Step 8:

[1763] Subject: Server

[1764] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[1765] Step 9:

[1766] Subject: Server

[1767] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[1768] Step 10:

[1769] Subject: Server

[1770] The server transmits the automatically generated advertising content to the terminal.

[1771] Step 11:

[1772] Subject: Device

[1773] The terminal receives the advertisement content transmitted from the server.

[1774] Step 12:

[1775] Subject: Device

[1776] The terminal displays the received advertising content on the display in real time, for example, an advertisement for a new fashion item.

[1777] Step 13:

[1778] Subject: Device

[1779] While the ad is being displayed, the device uses its camera to collect viewer response data, including eye tracking and facial expression analysis.

[1780] Step 14:

[1781] Subject: Device

[1782] The terminal transmits the collected viewer reaction data to the server.

[1783] Step 15:

[1784] Subject: Server

[1785] The server receives the response data sent from the terminal.

[1786] Step 16:

[1787] Subject: Server

[1788] The server analyzes the effectiveness of the advertisement based on the reaction data, specifically evaluating the length of time the viewer's gaze remains on the screen and changes in facial expression, and quantifies the effectiveness of the advertisement.

[1789] Step 17:

[1790] Subject: Server

[1791] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[1792] This allows viewers to receive ads that are optimized for them in real time, maximizing advertising effectiveness.

[1793] Example 1

[1794] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1795] Conventional advertising display systems have had difficulty effectively delivering advertisements to viewers. In particular, they have had issues with being unable to select the most appropriate advertisement based on the viewer's attributes and display it in real time. Furthermore, there are limitations to accurately measuring advertising effectiveness and reflecting this in future ad distribution. Furthermore, there are insufficient methods for collecting data to analyze how ads affect viewers.

[1796] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1797] In this invention, the server includes means for preprocessing viewer data, detecting and tracking face data, means for estimating the viewer's age, gender, and emotion based on the face data, and means for selecting optimal advertisements based on the estimated viewer attributes. This makes it possible to display advertisements optimized for the viewer in real time, accurately measure the effectiveness of the advertisements, and reflect this in the next distribution.

[1798] A "terminal" is a device installed to collect data from viewers and transmit the collected data to a server.

[1799] "Viewer data" refers to viewer video data and other related data collected by a terminal via a camera or the like.

[1800] "Preprocessing" refers to processing such as image resizing and noise removal that is performed on collected video data.

[1801] "Face data" refers to data of the viewer's face regions detected and tracked from the pre-processed video data.

[1802] "Means for estimating age, gender, and emotion of a viewer based on facial data" refers to the processes and models within the server that analyze received facial data and estimate the age, gender, and emotional attributes of the target viewer.

[1803] "Viewer demographics" are data that refer to specific characteristics of viewers, such as their age, gender, and emotions.

[1804] The "means for selecting an advertisement" refers to the mechanisms and algorithms within the server for selecting the most suitable advertisement based on the estimated viewer attributes.

[1805] "Advertising materials" are material data such as images, text, and videos used to generate advertisements.

[1806] "Advertising content" refers to advertising content that is automatically generated based on selected advertising materials.

[1807] "Response data" refers to data such as the viewer's gaze direction and changes in facial expression that are collected while the advertisement is being displayed.

[1808] "Means for analyzing advertising effectiveness" refers to the processes and algorithms within the server that analyze the effectiveness and influence of displayed advertisements based on collected response data.

[1809] The present invention relates to a system for displaying advertisements optimized for viewers in real time, which operates through multiple steps involving terminals, servers, and users.

[1810] 1. Collection of User Data

[1811] Subject: Device

[1812] The device collects viewer video data using a camera. The camera is a commonly used hardware device, such as a USB or built-in camera. The video data is pre-processed using an image processing library such as OpenCV, which results in the detection and tracking of viewer facial data. This facial data is then sent to the server in real time.

[1813] 2. Data Preprocessing and Analysis

[1814] Subject: Device

[1815] The device performs preprocessing on the collected video data. This preprocessing includes image resizing and noise reduction. It uses libraries such as OpenCV to adjust the image quality of the video data and detects the viewer's face using a face detection model (e.g., Haar Cascades or MTCNN). The device then extracts the detected face area and sends the data to the server.

[1816] 3. Estimation of viewer attributes

[1817] Subject: Server

[1818] The server receives the facial data sent from the device. Based on the received data, it uses a facial recognition model (e.g., Dlib or FaceNet) to estimate the viewer's age, gender, and emotion. For example, it identifies attributes such as "female in her 20s" or "smiling." In this process, it extracts features from the facial data and uses a pre-trained model to make an estimation.

[1819] 4. Ad selection and automatic generation

[1820] Subject: Server

[1821] The server selects the most suitable advertisement based on the analysis results. For example, it selects advertisements for fashion brands or cosmetics based on the attributes "women in their 20s" and "smiling face." The server searches an advertising material database (e.g., cloud storage service) and automatically generates advertising content using deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The advertising content is generated in formats such as HTML5 or MP4.

[1822] 5. Submission of advertising content

[1823] Subject: Server

[1824] The server sends the generated advertising content to the device using real-time communication technology such as WebSocket. The server identifies the device's IP address and sends the advertising data to the endpoint.

[1825] 6. Display of advertisements

[1826] Subject: Device

[1827] The terminal receives the advertising content sent from the server and displays it on the display in real time, for example, an advertisement for a new fashion item.

[1828] 7. Collecting viewer response data

[1829] Subject: Device

[1830] While the ad is being displayed, the device uses a camera to collect viewer response data, capturing gaze direction and facial expression changes in real time for visual analysis, which is then sent back to the server.

[1831] 8. Analysis of advertising effectiveness and feedback

[1832] Subject: Server

[1833] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. For example, if the viewer's gaze is frequently directed at the advertisement while viewing it, the advertisement is deemed to be highly successful. Based on this data, the server evaluates the effectiveness and influence of the advertisement and updates the model to help with future advertisement selection.

[1834] Specific examples

[1835] Signage in a shopping mall

[1836] Terminals installed at the entrance to the shopping mall collect viewer data. If the analysis results indicate a "woman in her early twenties," the server selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. The generated advertisements are sent to the terminals and displayed on displays within the mall. During this time, viewer response data is collected again, and the effectiveness of the advertisements is measured.

[1837] Digital signage in public transport

[1838] Terminals installed on public transport station platforms collect viewer data. For example, if the analysis results show "men in their 30s," the server selects advertisements for technology gadgets and sports equipment and automatically generates the latest smartphone advertisements. The generated advertisements are sent to the terminals and displayed on digital signage on the station platforms. Viewer response data is collected during display and sent to the server, where the advertising effectiveness is measured.

[1839] Prompt Sentence Examples

[1840] "Select an ad for a tech gadget for a man in his 30s and generate and display an ad for the latest smartphone."

[1841] keyword

[1842] Generative AI model, prompt sentence

[1843] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1844] Step 1:

[1845] Subject: Device

[1846] The device uses a camera to collect video data of the viewer. This video data is captured from the camera in real time. Specifically, it stores frame data continuously acquired from the camera in memory and performs image preprocessing using the OpenCV library. Preprocessing includes image resizing and noise removal. The input is raw video data, and the output is preprocessed video data.

[1847] Step 2:

[1848] Subject: Device

[1849] The device detects and tracks the viewer's facial data from the preprocessed video data. Specifically, it identifies the facial region using a face detection model such as OpenCV, Haar Cascades, or MTCNN. The input is the preprocessed video data, and the output is the viewer's facial region data. This data is sent to the server for further processing.

[1850] Step 3:

[1851] Subject: Server

[1852] The server receives the face region data sent from the device. Based on the received data, it uses a face recognition model such as Dlib or FaceNet to estimate the viewer's age, gender, and emotion. The input is the face region data, and the output is the estimated viewer attributes (e.g., "female in her 20s" and "smiling"). Specifically, it extracts features and analyzes them using a pre-trained model.

[1853] Step 4:

[1854] Subject: Server

[1855] The server selects the most suitable advertisement based on the viewer attributes obtained as a result of the analysis. As a specific example, it searches an advertising material database to select advertisements for fashion brands and cosmetics for the attributes "women in their 20s" and "smiling faces." The input is the estimated viewer attributes, and the output is the most suitable advertising material.

[1856] Step 5:

[1857] Subject: Server

[1858] The server automatically generates advertising content based on the selected advertising materials. This process uses deep learning models (e.g., GPT-3 or Transformer models) or rule-based algorithms. The input is the selected advertising materials, and the output is the generated advertising content. Specifically, the server creates advertisements in HTML5 or MP4 format based on the material data.

[1859] Step 6:

[1860] Subject: Server

[1861] The server transmits the generated advertising content to the device using real-time communication technology such as WebSocket. The input is the generated advertising content, and the output is data transmission to the device.

[1862] Step 7:

[1863] Subject: Device

[1864] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, an advertisement for a new fashion item is displayed on a display device connected to the terminal. The input is the received advertising content, and the output is the display on which the advertisement is displayed.

[1865] Step 8:

[1866] Subject: Device

[1867] While the ad is being displayed, the device uses a camera to collect viewer response data. Specifically, it captures changes in gaze direction and facial expressions. The input is the displayed ad content, and the output is viewer response data. This data is then sent back to the server.

[1868] Step 9:

[1869] Subject: Server

[1870] The server receives viewer response data sent from the device and analyzes the effectiveness of the advertisement. Specifically, if the viewer's gaze is frequently directed at the advertisement, the success of the advertisement is evaluated highly. The input is the viewer response data, and the output is the analysis result of the advertisement's effectiveness. The server updates the advertising model based on this analysis result and reflects it in subsequent advertisement selections.

[1871] (Application example 1)

[1872] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1873] Conventional advertising display systems did not fully utilize viewer information, making it difficult to select the optimal advertisement to maximize advertising effectiveness. Furthermore, they lacked the technology to collect viewer response information in real time and immediately reflect it in advertising strategies. As a result, advertisements that were not appealing to viewers were displayed, reducing advertising effectiveness.

[1874] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1875] In this invention, the server includes means for collecting viewer data from the terminal, means for analyzing the viewer data and estimating age, gender, and emotion, means for selecting an appropriate advertisement based on the analyzed viewer data, means for automatically generating an advertisement based on the selected advertising material, means for transmitting the automatically generated advertisement to the terminal, means for displaying the advertisement on the terminal, means for collecting viewer responses while the advertisement is being displayed, means for analyzing the effectiveness of the advertisement based on the collected response data, means for collecting and analyzing video of the viewer using a camera mounted on the smart glasses, and means for displaying the advertisement on the display of the smart glasses. This enables real-time display of personalized advertisements tailored to the characteristics of the viewer and rapid feedback of the effectiveness of the advertisement.

[1876] "Terminal" refers to a device used to collect viewer data and display advertisements, including smartphones, tablets, smart glasses, etc.

[1877] "Viewer data" refers to data that includes information such as the viewer's age, gender, and emotions, and is collected through cameras and sensors.

[1878] "Analysis" is the process of estimating attributes such as age, gender, and emotions based on collected viewer data.

[1879] "Ad selection" is the process of selecting the most suitable advertisement based on analyzed viewer data.

[1880] "Advertising materials" are materials such as databases, images, videos, and text used to generate advertisements.

[1881] "Automatic generation" is the process of automatically creating advertising content based on advertising materials after an advertisement has been selected.

[1882] "Advertising effectiveness" is an indicator that evaluates the impact that an advertisement has on viewers, and is analyzed based on viewer response data.

[1883] "Smart glasses" are wearable devices equipped with a camera and a display that can collect video data from viewers and display advertisements.

[1884] A "camera" is a device that collects video data from viewers and is installed in smart glasses or terminals.

[1885] A "display" is a display device that displays the generated advertisement to the viewer, and is installed in a terminal or smart glasses.

[1886] "Response data" refers to data such as the viewer's gaze, facial expressions, and movements while the advertisement is being displayed.

[1887] This invention relates to a system for displaying advertisements optimized for viewers in real time and measuring their effectiveness. This system is composed of the following components:

[1888] System Configuration

[1889] 1. Hardware

[1890] Devices: Includes smartphones, tablets, smart glasses, etc.

[1891] Camera: A device installed in smart glasses or a device that collects video data of the viewer.

[1892] Display: A device mounted on the smart glasses or device that displays the generated advertisements.

[1893] 2. Software

[1894] Facial recognition technology: For example, OpenCV is used.

[1895] Data analysis platform: For example, AWS Lambda is used.

[1896] Ad selection algorithm: For example, TensorFlow is used.

[1897] System processing procedure

[1898] Collection and analysis of user data

[1899] The device collects the viewer's video data, which is then pre-processed to detect and track the viewer's face, and the detected face data is sent to the server in real time.

[1900] Ad selection and automatic generation

[1901] The server receives the facial data sent from the device and uses facial recognition technology to estimate the viewer's age, gender, and emotions. To do this, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling." The server then references previously collected data to determine whether the viewer is the same person. The server selects the most suitable advertisement based on the analyzed viewer data. For example, it selects advertisements for fashion brands and cosmetics based on the attributes "female in her 20s" and "smiling." It then searches a database of advertising materials provided by the advertiser and lists the most suitable advertising candidates. Based on the selected advertising materials, advertising content is automatically generated using AI and algorithms. The generated advertising content is then sent to the device.

[1902] Ad display and effectiveness measurement

[1903] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be displayed on the display. While the advertisement is being displayed, the device collects viewer response data. The collected response data is sent to the server, where the effectiveness of the advertisement is measured. The server uses this data to analyze the success and influence of the advertisement and update the advertising model for further optimization.

[1904] Specific examples

[1905] Signage in a shopping mall

[1906] When a user is wearing smart glasses while walking through a shopping mall, the glasses collect the user's facial data in real time and send it to a server. Based on this data, the server determines that the user is a woman in her early twenties and selects advertisements for fashion brands and accessories. Advertisements for new collections are automatically generated and instantly displayed on the smart glasses' display. The user's reactions are also collected again, and the effectiveness of the advertisements is measured.

[1907] Digital signage in public transport

[1908] Terminals installed on public transport station platforms collect the viewer's facial data and send it to a server. If the user is identified as a "male in his 30s," ads for technology gadgets and sports equipment are selected, and the latest smartphone ads are automatically generated. These ads are then displayed on digital signage on the station platforms. Viewer response data is collected in the same way, and the effectiveness of the ads is measured.

[1909] Prompt Sentence Examples

[1910] "Woman in her 20s, smiling. Generate an ad for new fashion items. Also consider the option to show an ad for your summer accessories collection instead."

[1911] "Based on facial recognition data, we've identified her as a woman in her 20s. Then generate ads for a fashion brand that would be suitable for her. Also, create content that reflects positive emotions, taking into account smiling faces."

[1912] The above is a specific embodiment for carrying out the present invention, which makes it possible to provide advertisements optimized for viewers in real time and maximize the effectiveness of the advertisements.

[1913] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1914] Step 1:

[1915] User Data Collection

[1916] Input: Video data from a camera mounted on smart glasses

[1917] Processing: The device uses the camera to collect the user's video data. The video data is captured in real time and pre-processed (resolution adjustment, noise reduction).

[1918] Output: Pre-processed video data

[1919] Step 2:

[1920] Face Detection and Tracking

[1921] Input: Preprocessed video data

[1922] Processing: The device uses OpenCV to detect faces in the video data and perform face tracking. The face detection algorithm identifies the position of the face within the video frame.

[1923] Output: Face data (face position, facial feature points, etc.)

[1924] Step 3:

[1925] Data transmission

[1926] Input: Face data

[1927] Processing: The device sends the detected face data to the server in real time using the secure HTTP(S) protocol.

[1928] Output: Face data sent to the server

[1929] Step 4:

[1930] Face Recognition and Attribute Estimation

[1931] Input: Face data sent to the server

[1932] Processing: The server analyzes the facial data using facial recognition technology (e.g., a facial recognition model) to estimate age, gender, and emotion. Analysis includes extracting facial features and estimating attributes using a classifier.

[1933] Output: Viewer attribute data (e.g., "Female in her 20s," "Smiling," etc.)

[1934] Step 5:

[1935] Repeater Identification

[1936] Input: Viewer demographic data

[1937] Processing: The server refers to a database of previously collected data and compares it with the viewer's facial features to identify whether they are repeat visitors. For this purpose, it uses an approximate nearest neighbor search algorithm.

[1938] Output: Repeater identification result

[1939] Step 6:

[1940] Ad selection

[1941] Input: Viewer attribute data, repeater identification results

[1942] Processing: The server selects the optimal advertisement using an advertisement selection algorithm (for example, an algorithm using TensorFlow) based on the viewer attribute data and the results of repeat customer identification.

[1943] Output: Ad candidate list

[1944] Step 7:

[1945] Auto-generated ads

[1946] Input: Ad candidate list

[1947] Processing: The server automatically generates optimal advertising content from a selected list of ad candidates based on the advertising material database provided by the advertiser. For generation, a generative AI model is used.

[1948] Output: Generated ad content

[1949] Step 8:

[1950] Sending Ads

[1951] Input: Generated ad content

[1952] Processing: The server sends the generated advertising content to the device using secure HTTP(S) as the communication protocol.

[1953] Output: Ad content sent to the device

[1954] Step 9:

[1955] Displaying ads

[1956] Input: Ad content sent to the device

[1957] Processing: The device displays the received advertising content on the display in real time. The advertisement is displayed to the user using the display of the smart glasses.

[1958] Output: The ad shown to the user

[1959] Step 10:

[1960] Collecting viewer responses

[1961] Input: The ad shown to the user

[1962] Processing: The device collects user response data (eye gaze, facial expressions, movements, etc.) while the ad is being displayed. The device uses cameras and sensors to capture response data in real time.

[1963] Output: Collected reaction data

[1964] Step 11:

[1965] Sending reaction data

[1966] Input: Collected reaction data

[1967] Processing: The device sends the collected reaction data to the server using secure HTTP(S).

[1968] Output: Response data sent to the server

[1969] Step 12:

[1970] Measuring advertising effectiveness

[1971] Input: Reaction data sent to the server

[1972] Processing: The server analyzes the effectiveness of the ads based on the collected response data, using statistical analysis and machine learning models.

[1973] Output: Advertising effectiveness data

[1974] Step 13:

[1975] Updated advertising model

[1976] Input: Advertising effectiveness data

[1977] Processing: The server updates the advertising model based on the advertising effectiveness data. The updated model is reflected in the next ad delivery.

[1978] Output: Updated ad model

[1979] The above are the specific processing steps for carrying out the present invention.

[1980] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1981] The present invention combines a system that displays advertisements optimized for each viewer in real time with an emotion engine that recognizes the user's emotions. Below, the program processing of this entire system is explained in natural language.

[1982] Collection and analysis of user data

[1983] Subject: Device

[1984] The device uses a camera to collect real-time video data of viewers, including facial images and eye movements.

[1985] Subject: Device

[1986] The device pre-processes the collected video data and performs face detection and tracking for facial recognition.

[1987] Subject: Device

[1988] The terminal transmits the detected and tracked face data to the server.

[1989] Subject: Server

[1990] The server receives the face data sent from the terminal.

[1991] Subject: Server

[1992] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Using a facial recognition model, the server identifies attributes such as "female in her 20s" and "smiling."

[1993] Subject: Server

[1994] The server uses an emotion engine to analyze the user's facial expressions, voice, posture, and other characteristics to estimate their emotions. For example, it can recognize emotions such as smile, surprise, and interest.

[1995] Ad selection and automatic generation

[1996] Subject: Server

[1997] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetics based on attributes such as "women in their 20s" and "smiling faces."

[1998] Subject: Server

[1999] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates.

[2000] Subject: Server

[2001] The server automatically generates personalized advertising content using AI and algorithms based on the listed advertising materials.

[2002] Subject: Server

[2003] The automatically generated advertising content is transmitted to the terminal.

[2004] Ad display and effectiveness measurement

[2005] Subject: Device

[2006] The device receives the advertising content sent from the server and displays it on the display in real time, for example, advertising new fashion items.

[2007] Subject: Device

[2008] While the ad is being displayed, the device uses its camera and emotion engine to collect viewer response data, including eye tracking and facial expression analysis, to measure whether the viewer is paying attention to the ad.

[2009] Subject: Device

[2010] The terminal transmits the collected viewer reaction data and emotion data to the server.

[2011] Subject: Server

[2012] The server receives the reaction data and emotion data transmitted from the terminal.

[2013] Subject: Server

[2014] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. Specifically, it evaluates the length of time the viewer's gaze remains on the screen and changes in emotions, and quantifies the effectiveness of the advertisement.

[2015] Subject: Server

[2016] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[2017] Specific examples

[2018] Signage in a shopping mall

[2019] Subject: Server

[2020] The server collects and analyzes viewer and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate a "woman in her early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays inside the shopping mall. Meanwhile, viewer response and emotional data are collected again, and the effectiveness of the advertisements is measured.

[2021] Digital signage in public transport

[2022] Subject: Server

[2023] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a "male in his 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are sent to the terminals and displayed on digital signage on the station platforms. Similarly, viewer response and emotional data are collected, and the effectiveness of the advertisements is measured.

[2024] The above is a specific embodiment for implementing the invention of a system incorporating an emotion engine, which allows advertisements optimized for viewers to be provided in real time, maximizing the effectiveness of the advertisements.

[2025] The processing flow will be explained below.

[2026] Step 1:

[2027] Subject: Device

[2028] The device uses a camera to collect real-time video data of viewers, including facial images, eye movements, and facial expressions.

[2029] Step 2:

[2030] Subject: Device

[2031] The device pre-processes the collected video data and performs face detection and tracking for facial recognition, specifically by using a specific algorithm to detect the position of the face and track its movement.

[2032] Step 3:

[2033] Subject: Device

[2034] The device sends face data to the server, including face position information and tracking data.

[2035] Step 4:

[2036] Subject: Server

[2037] The server receives the face data sent from the terminal.

[2038] Step 5:

[2039] Subject: Server

[2040] The server uses facial recognition technology to estimate the viewer's attributes, such as age, gender, and emotion. Specifically, it uses a facial recognition model to analyze image data and identify attributes such as "female in her 20s" and "smiling."

[2041] Step 6:

[2042] Subject: Server

[2043] The server uses an emotion engine to analyze the viewer's facial expressions, voice, posture, and other characteristics to estimate their emotions in real time, for example, determining whether they are smiling, surprised, or interested.

[2044] Step 7:

[2045] Subject: Server

[2046] The server references previously collected viewer data and checks it against a database to identify repeat viewers.

[2047] Step 8:

[2048] Subject: Server

[2049] The server then selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands or cosmetics that match the attributes of "women in their 20s" and "smiling faces."

[2050] Step 9:

[2051] Subject: Server

[2052] The server searches an advertising material database provided by the advertiser and lists suitable advertisements as candidates.

[2053] Step 10:

[2054] Subject: Server

[2055] The server then uses AI and algorithms to automatically generate personalized advertising content based on the listed advertising materials, a process that involves combining images, text, and videos.

[2056] Step 11:

[2057] Subject: Server

[2058] The automatically generated advertising content is transmitted to the terminal.

[2059] Step 12:

[2060] Subject: Device

[2061] The terminal receives the advertisement content transmitted from the server.

[2062] Step 13:

[2063] Subject: Device

[2064] The terminal displays the received advertising content on the display in real time. For example, an advertisement for "new fashion items" is displayed on the display.

[2065] Step 14:

[2066] Subject: Device

[2067] While the ad is being displayed, the device uses a camera and emotion engine to collect viewer response data, specifically tracking eye movements and analyzing facial expressions to record changes in viewer attention and emotions.

[2068] Step 15:

[2069] Subject: Device

[2070] The terminal transmits the collected viewer reaction data and emotion data to the server.

[2071] Step 16:

[2072] Subject: Server

[2073] The server receives the reaction data and emotion data transmitted from the terminal.

[2074] Step 17:

[2075] Subject: Server

[2076] The server analyzes the effectiveness of the advertisement based on the reaction data and emotional data. For example, it quantifies the time the viewer pays attention to the advertisement and changes in their emotions to evaluate the success and impact of the advertisement.

[2077] Step 18:

[2078] Subject: Server

[2079] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery.

[2080] That's all. This system, combined with an emotion engine, makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[2081] Example 2

[2082] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2083] Conventional advertising display systems have difficulty delivering personalized ads to viewers in real time, making it difficult to maximize advertising effectiveness. They also lack the means to accurately analyze viewers' emotions and reactions and dynamically optimize ads based on that analysis. This has prevented advertisers from delivering optimal ads to viewers, resulting in reduced advertising effectiveness.

[2084] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving viewer data transmitted from a terminal, means for estimating the viewer's age, gender, and emotion using face recognition technology, means for analyzing the user's facial expression, voice, and posture using an emotion engine to estimate emotion, means for selecting an optimal advertisement based on the analyzed viewer data and emotion data, means for referencing an advertising material database and listing appropriate advertisement candidates, means for automatically generating personalized advertising content using a generative AI model, and means for receiving response data and emotion data transmitted from the terminal and analyzing advertising effectiveness. This makes it possible to provide viewers with optimized advertisements in real time, maximizing advertising effectiveness.

[2085] A "terminal" is a hardware device for collecting and processing viewer video data.

[2086] A "server" is a computer system that receives data sent from a terminal, analyzes it, and generates advertisements.

[2087] "Viewer data" refers to data that includes information about the viewer, such as an image of the viewer's face and eye movements.

[2088] "Facial recognition technology" is a technology for detecting human faces from video data and estimating their attributes.

[2089] The "emotion engine" is software that analyzes and estimates emotions from a user's facial expressions, voice, and posture.

[2090] The "advertising material database" is a database for managing various advertising materials provided by advertisers.

[2091] A "generative AI model" is an artificial intelligence model that automatically generates advertising content based on input data.

[2092] "Response data" refers to data related to viewers' reactions to advertisements, such as eye tracking and facial expression analysis.

[2093] "Advertising effectiveness" refers to the effectiveness of an advertisement as evaluated based on viewer reactions and emotional data.

[2094] An "advertising model" is a set of algorithms or rules used to maximize the effectiveness of advertising.

[2095] This invention is a system that provides viewers with advertisements optimized for them in real time, and combines it with an emotion engine that recognizes the user's emotions. This system is composed of multiple elements, including a terminal, a server, and an emotion engine, and these elements work together to maximize the effectiveness of advertisements.

[2096] Collection and analysis of user data

[2097] Subject: Device

[2098] The device is equipped with a camera to collect video data of viewers. The video data collected using the camera includes images of the viewer's face and eye movements. The device uses libraries such as OpenCV to detect and track faces from the collected video data. The detected and tracked face data is sent to the server in JSON format or protocol buffers.

[2099] Subject: Server

[2100] The server receives the facial data sent from the device. Based on the received facial data, it uses facial recognition technology such as dlib to estimate the viewer's age, gender, and emotion. The server then uses an emotion engine (for example, a model trained with TensorFlow or PyTorch) to analyze the user's facial expression, voice, and posture to estimate their emotion. Specifically, it recognizes emotions such as smile, surprise, and interest.

[2101] Ad selection and automatic generation

[2102] Subject: Server

[2103] The server selects the most suitable advertisement based on the analyzed viewer data and emotional data. For example, it selects advertisements for fashion brands and cosmetic products based on attributes such as "women in their 20s" and "smiling faces." It references a database of advertising materials provided by the advertiser to create a list of suitable advertisement candidates, and then automatically generates personalized advertising content using a generative AI model (such as GPT-3). The generated advertising content is then sent to the device.

[2104] Ad display and effectiveness measurement

[2105] Subject: Device

[2106] The device receives advertising content sent from the server and displays it on the display in real time. For example, an advertisement for a new fashion item may be shown on the display. While the advertisement is being displayed, the device uses a camera and an emotion engine to collect viewer reaction data. Specifically, it uses eye-tracking technology to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions.

[2107] Subject: Server

[2108] The server receives the reaction and emotion data sent from the device and performs detailed analysis. It evaluates factors such as gaze duration and changes in facial expression to quantify advertising effectiveness. The advertising model is updated based on the analysis results and reflected in the next ad delivery. This maximizes advertising effectiveness and enables more effective ad delivery.

[2109] Specific examples

[2110] Signage in a shopping mall

[2111] Subject: Server

[2112] The server collects and analyzes viewer data and emotional data from terminals installed at the entrances of shopping malls. For example, if the analysis results indicate "women in their early twenties," it selects advertisements for fashion brands and accessories and automatically generates advertisements for new collections. These advertisements are sent to the terminals and displayed on displays within the shopping mall. While the advertisements are displayed, the terminals collect viewer response data and emotional data, which are then sent back to the server to measure the effectiveness of the advertisements.

[2113] Example prompt sentence:

[2114] A video shows a woman in her early twenties at the entrance of a shopping mall. She is smiling and seems interested in the new fashion items. Generate the best ad for her.

[2115] Digital signage in public transport

[2116] Subject: Server

[2117] The server collects and analyzes viewer and emotional data from terminals installed on public transport station platforms. For example, if the analysis results indicate a target audience of "men in their 30s," it will select advertisements for technology gadgets and sports equipment and automatically generate the latest smartphone advertisements. These advertisements are then sent to the terminals and displayed on digital signage on the station platforms. At the same time, the terminals collect viewer response and emotional data, which are then sent to the server to measure the effectiveness of the advertisements.

[2118] Example prompt sentence:

[2119] A video shows a man in his 30s on a train platform. He seems interested in technology gadgets. Generate an ad for the latest smartphone that is perfect for him.

[2120] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide advertisements optimized for viewers in real time, maximizing the effectiveness of the advertisements.

[2121] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2122] Step 1:

[2123] Video data collection

[2124] Subject: Device

[2125] The device uses a camera to collect real-time video data of the viewer, including facial images and eye movements. Specifically, the device captures the camera's video stream and captures multiple frames per second. The captured frames are stored in a buffer and sent to the next processing step.

[2126] Step 2:

[2127] Face Detection and Tracking

[2128] Subject: Device

[2129] The device detects and tracks faces from the collected video data. It uses OpenCV's face recognition algorithm to detect faces. Specifically, it detects the position of a face in each frame and tracks the same face in the next frame. It uses the acquired video frame as input and generates coordinate data of the detected face as output.

[2130] Step 3:

[2131] Sending data

[2132] Subject: Device

[2133] The device sends the detected and tracked face data to the server. The data is sent in JSON format and is encoded before being sent to the server. It uses the detected face coordinate data as input and generates encoded JSON data as output, which is sent to the server.

[2134] Step 4:

[2135] Receiving data

[2136] Subject: Server

[2137] The server receives the face data sent from the device. Specifically, the server receives an HTTP request and parses the JSON data. It uses the received JSON data as input and obtains parsed face coordinate data as output.

[2138] Step 5:

[2139] Face Recognition and Attribute Estimation

[2140] Subject: Server

[2141] The server uses facial recognition technology to estimate the viewer's age, gender, and emotion. It uses dlib's facial recognition model to analyze the viewer's facial features. It uses the received facial coordinate data as input and generates attribute data such as age, gender, and emotion as output.

[2142] Step 6:

[2143] Emotion Analysis

[2144] Subject: Server

[2145] The server uses an emotion engine to analyze the user's facial expressions, voice, and posture to estimate their emotions. Emotion analysis is performed using a model using TensorFlow and PyTorch. Specifically, it analyzes changes in facial expressions, tone of voice, and posture to estimate emotions such as "smile" or "surprise." It uses facial feature data as input and generates emotion data as output.

[2146] Step 7:

[2147] Ad selection

[2148] Subject: Server

[2149] The server selects the optimal advertisement based on the analyzed viewer data and emotional data. It searches the database for an appropriate advertisement according to the viewer's attributes. It uses attribute data such as age, gender, and emotion as input and generates the selected advertisement data as output.

[2150] Step 8:

[2151] Listing advertising materials

[2152] Subject: Server

[2153] The server refers to an advertisement material database provided by the advertiser and lists suitable advertisements as candidates. Using the selected advertisement data as input, the server generates the listed advertisement material data as output.

[2154] Step 9:

[2155] Auto-generated personalized ads

[2156] Subject: Server

[2157] The server uses a generative AI model to automatically generate personalized advertising content based on the listed advertising materials. It generates advertising text and images using a generative AI model (such as GPT-3). It uses the listed advertising material data as input and obtains generated advertising content as output.

[2158] Step 10:

[2159] Sending advertising content

[2160] Subject: Server

[2161] The server sends the automatically generated advertising content to the terminal. Specifically, the server sends the generated advertising content to the terminal as an HTTP response. The server uses the generated advertising content as input and sends encoded data to the terminal as output.

[2162] Step 11:

[2163] Displaying ads

[2164] Subject: Device

[2165] The terminal receives the advertising content sent from the server and displays it on the display in real time. Specifically, the terminal displays advertising text and images on the display. The terminal uses the received advertising content as input and obtains the displayed advertisement as output.

[2166] Step 12:

[2167] Collecting viewer response data

[2168] Subject: Device

[2169] While the advertisement is being displayed, the device uses a camera and emotion engine to collect viewer response data. Eye-tracking technology is used to measure whether the viewer is paying attention to the advertisement and simultaneously analyzes changes in facial expressions. The input is the video data during the advertisement display, and the output is the collected viewer response data.

[2170] Step 13:

[2171] Sending reaction data

[2172] Subject: Device

[2173] The device sends the collected viewer reaction data and emotion data to the server. Specifically, it encodes the collected data in JSON format and sends it as an HTTP request. It uses the collected reaction data as input and sends the encoded data to the server as output.

[2174] Step 14:

[2175] Receiving and analyzing data

[2176] Subject: Server

[2177] The server receives the reaction data and emotion data sent from the device and performs detailed analysis. It evaluates gaze duration and changes in facial expressions to quantify the effectiveness of the advertisement. It uses the received reaction data as input and obtains the quantified advertising effectiveness as output.

[2178] Step 15:

[2179] Updated advertising model

[2180] Subject: Server

[2181] The server updates the advertising model based on the analysis results and reflects them in the next ad delivery. Specifically, the server reflects the newly obtained advertising effectiveness data as learning data in the model. The server uses the quantified advertising effectiveness data as input and obtains an updated advertising model as output.

[2182] This concludes the detailed explanation of the processing flow of the program for this system. These processing steps enable personalized advertisements to be provided to viewers in real time, maximizing their effectiveness.

[2183] (Application example 2)

[2184] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2185] Conventional advertising systems have difficulty delivering personalized ads that reflect the emotions and interests of individual users, making it difficult to maximize the effectiveness of advertising. Furthermore, while there is a need to analyze user behavior and emotional data in real time and apply the results immediately, there has been a lack of technology to solve this problem.

[2186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2187] In this invention, the server includes means for collecting viewer data from terminals, means for analyzing the viewer data to estimate age, gender, and emotion, means for selecting appropriate advertisements based on the analyzed viewer data, means for automatically generating advertisements based on the selected advertising materials, means for transmitting the automatically generated advertisements to the terminals, means for displaying advertisements on the terminals, means for collecting viewer responses while the advertisements are being displayed, means for analyzing the effectiveness of the advertisements based on the collected response data, and means for analyzing user behavior, gaze data, and emotions and providing information about in-store products and discount coupons in real time. This allows advertisements optimized for individual users to be provided in real time, maximizing the effectiveness of the advertisements.

[2188] A "terminal" is a device that processes information and has functions such as collecting viewer data and displaying advertisements.

[2189] "Viewer data" refers to data that includes information such as the viewer's facial image, eye movements, and facial expressions, and is used to analyze emotions and attributes.

[2190] "Emotion" refers to the psychological state of the viewer that can be inferred from their facial expressions and gestures, including, for example, smile, surprise, interest, etc.

[2191] "Advertising materials" refers to content such as images, text, and videos used in advertising distribution.

[2192] "Auto-generating ads" refers to the process of automatically creating appropriate advertising content based on analyzed viewer data.

[2193] "Real-time" means that data collection, analysis, ad generation and delivery occur almost simultaneously.

[2194] "Behavioral data" refers to data including users' movements within a store, their line of sight, and the length of time they spend there.

[2195] A "discount coupon" is an electronic or paper voucher offering a special discount on the purchase of a product.

[2196] A "server" is a central computer system for receiving, analyzing, and processing data sent from terminals.

[2197] "Advertising effectiveness" refers to the results of evaluating viewers' reactions and purchasing intentions after viewing an advertisement.

[2198] To implement this invention, the following system configuration and processes are required: The system functions through cooperation between the terminal, the server, and the user.

[2199] System configuration

[2200] 1. Terminal

[2201] Hardware: Smart glasses (including camera, display, and communication module)

[2202] Software: OpenCV (for face recognition), emotion_engine (for emotion analysis)

[2203] 2. Server

[2204] Software: Data receiving module, data analysis module (viewer data analysis, emotion estimation), ad selection module, ad generation module, ad transmission module

[2205] 3. Users

[2206] An individual wearing smart glasses and moving around in a physical store

[2207] Program processing

[2208] 1. Collecting viewer data via devices

[2209] The device uses the camera installed in the smart glasses to collect real-time video data of the user, including facial images and eye movements (eye tracking).

[2210] 2. Preprocessing and analysis of viewer data

[2211] The device preprocesses the collected video data using OpenCV, detects and tracks faces, and transmits the detected and tracked face data to the server.

[2212] 3. Data analysis and ad selection by the server

[2213] The server receives the facial data sent from the device and uses the emotion_engine to estimate the viewer's attributes, such as age, gender, and emotions. Based on the analysis results, the server selects the most suitable advertisement. Specifically, it selects products and discount coupons that are likely to interest the viewer.

[2214] 4. Auto-generated ads

[2215] The server references a database of advertising materials provided by advertisers and uses AI algorithms to automatically generate personalized advertising content, which is then sent to the device.

[2216] 5. Display of advertisements by device

[2217] The terminal receives the advertising content sent from the server and displays it on the display of the smart glasses in real time.

[2218] 6. Measuring advertising effectiveness

[2219] While the ad is being displayed, the device uses a camera and an emotion engine to collect user reaction data, which is then sent to a server to measure the effectiveness of the ad.

[2220] 7. Analysis of advertising effectiveness and model update

[2221] The server analyzes the advertising effectiveness based on the response data sent from the terminal and updates the advertising model to reflect the results in the next advertisement distribution.

[2222] Specific examples

[2223] When a user wears smart glasses and visits a clothing store, the camera in the glasses tracks the user's gaze, and if the user stays in a particular section for a long time, the emotion engine analyzes the user's facial expressions of interest or joy. Based on this information, the server displays information about similar products and discount coupons on the smart glasses' display in real time.

[2224] Prompt Sentence Examples

[2225] "Get 20% off your favorite clothes today. Check out our new collection."

[2226] The above is a specific embodiment of the invention. This system allows users to receive optimized advertisements in real time, maximizing the effectiveness of the advertisements.

[2227] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2228] Step 1:

[2229] Collecting viewer data via devices

[2230] The device uses the camera installed in the smart glasses to collect the user's video data in real time, including the user's facial image, eye movements, and facial expressions. This video data is input and output as raw data captured by the camera.

[2231] Step 2:

[2232] Audience data preprocessing and facial recognition

[2233] The device preprocesses the collected video data using OpenCV to detect and track faces. Specifically, it analyzes the video data frame by frame to recognize the face, and then tracks the position of the face. The input for this process is the camera video data, and the output is data including the position information of the face.

[2234] Step 3:

[2235] Sending face data to the server

[2236] The device sends the detected and tracked face data to the server. The sent data includes face images, their location information, and changes over time. The input is data including face location information, and the output is a transmission completion status to the server.

[2237] Step 4:

[2238] Analysis of viewer data by the server

[2239] The server receives the facial data sent from the device and uses emotion_engine to estimate the viewer's attributes such as age, gender, and emotion. Specifically, it applies a facial recognition algorithm to the received data as input and performs analysis using an age estimation mode...

Claims

1. a means for collecting viewer data from the device; A means of analyzing viewer data to estimate age, gender, and emotions; A means of selecting appropriate advertisements based on analyzed viewer data; A means for automatically generating advertisements based on the selected advertising materials; means for transmitting the automatically generated advertisement to the terminal; means for displaying advertisements on the terminal; a means for collecting viewer responses during the display of the advertisement; A system that includes a means for analyzing advertising effectiveness based on collected response data.

2. 10. The system of claim 1, further comprising means for receiving viewer data transmitted from the terminal and identifying repeaters based on the data.

3. 2. The system according to claim 1, further comprising means for updating the advertising model based on the analysis result of the advertising effectiveness and reflecting the updated model in the next advertisement distribution. These claims clearly define the technical scope of the invention and provide criteria for patent examination.

Images