System

Abstract

A system is provided.SOLUTION: A system comprising: emotion sensor means for obtaining emotion data of a user; analysis means for analyzing the emotion data to identify an emotional state of the user; advertisement selection means for selecting an optimized advertisement based on the emotional state; and advertisement delivery means for delivering the advertisement to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current web advertising targeting methods are primarily based on user attributes (gender, age), interests, and behavioral history, but these methods do not necessarily take into account the user's actual emotions or state. As a result, users are often shown ads that they are not interested in, reducing the effectiveness of the ads and even causing users to distrust the ads due to inappropriate targeting. This poses a risk of causing annoyance rather than increasing users' desire to purchase.

[0005] Specifically, the following issues exist:

[0006] 1. Inaccurate ad targeting due to lack of knowledge of users' real-time emotional state.

[0007] 2. Ad delivery that relies solely on past behavioral history makes it difficult to provide ads that reflect a user's current interests and emotions.

[0008] 3. There is a lack of means to measure user preference and optimize advertising based on that. [Means for solving the problem]

[0009] In order to solve the above problems, the present invention provides the following means.

[0010] 1. Emotion Sensor Means:

[0011] To acquire user emotional data, emotion sensors such as wearable devices and cameras are used, allowing the user's pulse and facial expression data to be collected in real time.

[0012] 2. Analysis method:

[0013] The pulse data and facial expression data collected by the emotion sensor are analyzed to identify the user's emotional state. An AI module is used for the analysis, and emotions are identified based on pulse fluctuations and facial expression characteristics.

[0014] 3. Ad selection method:

[0015] The system provides a means for selecting the most suitable advertisement for the user based on the analyzed emotional data. The advertisement selection means checks the database for advertisements corresponding to the emotional state and selects the advertisement that is most suitable for the user's current state.

[0016] 4. Advertising methods:

[0017] The system provides a means for delivering selected advertisements to users. The advertisement delivery means monitors user reactions to displayed advertisements (e.g., gaze retention time, clicks, etc.) and transmits the data to a server.

[0018] 5. Feedback methods:

[0019] The monitored reaction data is used to evaluate the effectiveness of the advertisement and reflected in the ad distribution algorithm from the next time onwards, thereby continuously improving the accuracy of ad distribution.

[0020] By using the above means, advertisements are optimized based on the user's real-time emotional state, and an advertisement distribution system that satisfies both users and advertisers is realized.

[0021] "Emotion sensor means" refers to a device or technology for acquiring emotion data such as pulse data and facial expression data of a user.

[0022] The "analysis means" refers to a technology that analyzes the data acquired by the emotion sensor means and identifies the emotional state of the user.

[0023] "Advertisement selection means" refers to a technology for selecting the most suitable advertisement for a user based on analyzed emotional data.

[0024] "Advertising delivery means" refers to technology for displaying selected advertisements to users and monitoring their responses.

[0025] "Emotional data" refers to information indicating the emotional state of a user obtained from their pulse, facial expression, and other physiological data.

[0026] A "wearable device" refers to a device worn by a user that collects pulse and other physiological data.

[0027] "Camera" refers to an image capture device for capturing a user's facial expressions.

[0028] "AI module" refers to artificial intelligence technology that analyzes collected emotional data and identifies the user's emotional state.

[0029] "Reaction data" refers to data that records the user's reactions to an advertisement, such as gaze duration, clicks, and scrolling.

[0030] The term "database" refers to a system that stores advertisement information that is referenced by the advertisement selection means when selecting advertisements. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0032] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0033] First, the terms used in the following description will be explained.

[0034] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0035] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0036] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0037] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0038] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0039] [First embodiment]

[0040] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0041] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0042] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0043] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0044] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0045] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0046] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0047] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0048] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0049] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0050] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0051] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0052] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components, including emotion sensor means, analysis means, advertisement selection means, and advertisement delivery means. Below, we will explain the details of each component and how they work together.

[0053] 1. Collecting Emotional Data

[0054] User

[0055] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[0056] Terminal

[0057] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[0058] 2. Emotion Data Analysis

[0059] Terminal

[0060] The collected pulse data and facial expression data are analyzed by an AI module, which analyzes pulse fluctuations and facial expression characteristics to identify the user's current emotional state (e.g., relaxed, excited, happy, etc.).

[0061] 3. Sending Emotional Data

[0062] Terminal

[0063] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[0064] 4. Ad optimization

[0065] server

[0066] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[0067] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it will select ads related to relaxation, and if the user is excited, it will select ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[0068] 5. Delivery of advertisements

[0069] server

[0070] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[0071] Terminal

[0072] Based on the received advertising data, the advertisement is displayed to the user. The timing and format of the display depend on the content.

[0073] It monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, scrolling, etc.

[0074] 6. Feedback and algorithm improvements

[0075] Terminal

[0076] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze and click action.

[0077] server

[0078] The received reaction data is analyzed to evaluate the effectiveness of the advertisement.

[0079] The characteristics of ineffective ads are analyzed and reflected in future targeting, for example, determining that certain types of ads are inappropriate for certain emotional states.

[0080] Continually improve ad delivery algorithms based on new emotional and reaction data.

[0081] Specific examples

[0082] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for the relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is sent back to the server and used to select advertisements for future use.

[0083] In this way, an advertising delivery system based on a user's real-time emotional state can attract users' interest with greater accuracy than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[0084] The processing flow will be explained below.

[0085] Step 1:

[0086] The user puts on the wearable device and launches an application (e.g., a VR app) that collects emotion data.

[0087] Step 2:

[0088] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[0089] Step 3:

[0090] The device analyzes the pulse data received and detects fluctuations in the user's heart rate, as increases or decreases in heart rate can indicate changes in mood.

[0091] Step 4:

[0092] The AI ​​module analyzes facial images captured by the device and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.) from their facial features. Changes in facial expression are analyzed as movements of facial parts.

[0093] Step 5:

[0094] The device combines the analysis results of the pulse data and facial expression data to determine the user's final emotional state. For example, if the heart rate is stable and the user is smiling, the device will be classified as "Relaxed."

[0095] Step 6:

[0096] The device then anonymizes the analyzed emotional data and sends it to a server, including the emotional state, device ID, and timestamp.

[0097] Step 7:

[0098] The server stores the received emotion data in a database, and when storing it, it labels the data according to the user's emotional state.

[0099] Step 8:

[0100] The server analyzes the emotional data and compares it with an advertising database to select the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select an advertisement related to relaxation.

[0101] Step 9:

[0102] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[0103] Step 10:

[0104] The device displays advertisements to the user based on the received advertisement data. The timing of the display depends on the application status.

[0105] Step 11:

[0106] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[0107] Step 12:

[0108] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[0109] Step 13:

[0110] The server analyzes the reaction data received and evaluates the effectiveness of the ads, analyzes the characteristics of the ads that were less effective, and generates feedback to discontinue or modify them.

[0111] Step 14:

[0112] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Conventional ad delivery systems have had difficulty delivering optimal ads that reflect the user's emotional state in real time. Furthermore, there were insufficient methods for evaluating the effectiveness of ads and reflecting this in future targeting, making it difficult to simultaneously increase the satisfaction of both users and advertisers.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes an advertisement selection means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the emotional state, a monitoring means for monitoring the user's reaction while the advertisement is being displayed, and an algorithm improvement means for evaluating the effectiveness of the advertisement based on the monitoring results and reflecting the results in subsequent targeting. This makes it possible to optimize advertisements that reflect the user's emotional state in real time and to continuously improve the advertisement distribution algorithm.

[0118] The "emotion sensor means" is a sensor for acquiring the user's emotional data, and specifically is a device including a wearable device for acquiring pulse data and a camera for capturing the user's facial expression.

[0119] The "analysis means" is a means for analyzing the emotional data acquired by the emotion sensor means and identifying the emotional state of the user. Specifically, it performs processing to analyze the emotional state using an AI module.

[0120] The "transmission means" is a means for anonymizing the analyzed emotion data and transmitting it to the server while protecting privacy.

[0121] The "advertisement selection means" is a means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the user's emotional state.

[0122] The "advertisement distribution means" is a means for distributing the selected advertisement to the user.

[0123] The "monitoring means" is a means for monitoring users' reactions while the advertisement is being displayed and collecting the data. Specifically, it is a means for recording gaze retention time, click actions, etc.

[0124] "Algorithm improvement measures" are measures for continuously improving the ad delivery algorithm by evaluating the effectiveness of advertising based on monitoring results and reflecting the evaluation results in future targeting.

[0125] This invention is a system that collects and analyzes real-time user emotional data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components: an emotion sensor means, an analysis means, a transmission means, an advertisement selection means, an advertisement delivery means, a monitoring means, and an algorithm improvement means. Below, we will explain in detail each component and how they work together.

[0126] Collecting Emotional Data

[0127] User

[0128] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[0129] Terminal

[0130] The device uses the in-camera to capture the user's facial expression data in real time, and simultaneously obtains pulse data from the wearable device via Bluetooth.

[0131] Emotional Data Analysis

[0132] Terminal

[0133] The device passes the acquired facial expression and pulse data to an AI module, which uses deep learning frameworks such as TensorFlow and PyTorch.

[0134] The device's AI module analyzes pulse fluctuations and facial features to identify the user's current emotional state, which can be relaxation, excitement, joy, etc.

[0135] Sending emotional data

[0136] Terminal

[0137] The device anonymizes the analyzed emotion data by removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[0138] The anonymized emotion data is sent to a server.

[0139] Ad optimization

[0140] server

[0141] The server stores the received emotion data in a database, which uses a relational database such as MySQL or PostgreSQL.

[0142] Based on the stored emotion data, the server matches it with an advertisement database: if the user is relaxed, it selects advertisements related to relaxation from the advertisement database.

[0143] The server takes into consideration the past ad display history and the user's action history to select the most suitable ad.

[0144] Ad serving

[0145] server

[0146] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[0147] Terminal

[0148] The device will then display the advertisement on the user's screen based on the received advertising data. The timing and format of the display will depend on the content. For example, the advertisements will be designed to appear naturally during the VR experience.

[0149] While the ad is being displayed, the device monitors the user's reactions, specifically recording actions such as gaze retention time, clicks on the ad, and scrolling.

[0150] Feedback and algorithm improvements

[0151] Terminal

[0152] After the ad is displayed, the user's reaction data, including gaze duration and click action, is sent to the server.

[0153] server

[0154] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It also analyzes the characteristics of ads that were less effective and reflects this in future targeting.

[0155] New emotional and reaction data will be used to continuously improve ad serving algorithms, for example by determining that certain types of ads are inappropriate for certain emotional states.

[0156] Specific examples

[0157] For example, if a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The device's AI module analyzes this data and determines that the user is relaxed. The device anonymizes this analysis result and sends it to a server. The server then selects advertisements appropriate for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions, such as gaze duration and clicks. This data is then sent back to the server and used to analyze the effectiveness of the advertisements and improve the algorithm.

[0158] The present invention enables optimal advertisement delivery based on the user's real-time emotional state, improving advertisement accuracy and user satisfaction.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Step 1:

[0161] User provides emotion data

[0162] The user puts on the wearable device and launches the VR application. The front camera captures facial expressions. At the same time, the terminal acquires pulse data from the wearable device via Bluetooth.

[0163] Input: User's facial expression data, pulse data

[0164] Output: Raw data collected

[0165] Step 2:

[0166] The device analyzes the emotional data

[0167] The device passes the acquired facial expression and pulse data to an AI module, which then uses a deep learning framework (such as TensorFlow or PyTorch) to analyze the user's emotional state.

[0168] Input: facial expression data, pulse data

[0169] Data processing / calculation: Analysis and processing using facial expression data and pulse rate data

[0170] Output: User's emotional state (relaxed, excited, happy, etc.)

[0171] Step 3:

[0172] The device sends analysis data

[0173] The device anonymizes the analyzed emotion data and transmits it to a server in a privacy-preserving manner, which involves removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[0174] Input: Parsed emotion data

[0175] Data processing: Data anonymization

[0176] Output: Anonymized emotion data

[0177] Step 4:

[0178] The server receives and stores emotion data.

[0179] The server stores the received emotion data in a database using a relational database (e.g., MySQL or PostgreSQL).

[0180] Input: Anonymized emotion data

[0181] Data processing: data storage processing

[0182] Output: Data stored in the database

[0183] Step 5:

[0184] Server optimizes ads

[0185] The server matches the stored emotional data with the advertisement database. If the user is relaxed, it selects advertisements related to relaxation, taking into account the user's past ad viewing history and user action history.

[0186] Input: Stored emotion data

[0187] Data processing / calculation: Matching of emotion data with advertising database

[0188] Output: Optimized advertising data

[0189] Step 6:

[0190] The server sends the advertisement

[0191] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[0192] Input: Optimized Ad Data

[0193] Data processing: Generation of transmission data

[0194] Output: Data sent to the terminal

[0195] Step 7:

[0196] The device displays the ad and monitors the reaction

[0197] The device displays the advertisement on the user's screen based on the received advertisement data, and monitors the user's reactions (e.g., gaze retention time, clicks, scrolling, etc.) while the advertisement is displayed.

[0198] Input: Transmission data (advertising ID, display timing, content URL)

[0199] Data processing: Displaying advertisements, monitoring user reactions

[0200] Output: Reaction data (gaze duration, click actions, etc.)

[0201] Step 8:

[0202] The device sends reaction data

[0203] The terminal transmits reaction data after the advertisement is displayed to the server.

[0204] Input: Reaction data

[0205] Data processing: Preparing data for transmission

[0206] Output: Reaction data to the server

[0207] Step 9:

[0208] The server analyzes the feedback and improves the algorithm

[0209] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It reanalyzes the characteristics of ads that were ineffective and reflects them in future targeting. It also continuously improves the ad delivery algorithm based on new emotion and reaction data.

[0210] Input: Reaction data

[0211] Data processing / calculation: Analysis of reaction data, updating of algorithms

[0212] Output: Improved ad serving algorithm

[0213] (Application example 1)

[0214] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0215] Conventional ad delivery systems deliver ads without considering the user's emotional state, making it difficult to attract users' attention. They also lacked a means to understand the effectiveness of ads in real time and reflect this in future ad optimizations. Furthermore, there was no mechanism for continuously improving the ad delivery algorithm based on user reactions, making effective targeting impossible.

[0216] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0217] In this invention, the server includes an emotion data analysis means, a transmission means for transmitting user reaction data after displaying an advertisement to the server and evaluating the effectiveness of the advertisement, and an algorithm improvement means for reselecting advertisements using an improved advertisement distribution algorithm, thereby enabling effective advertisement distribution based on the user's emotional state and continuous optimization of advertisements based on the reaction data.

[0218] An "emotion sensor means" is a device or apparatus used to acquire emotion data of a user.

[0219] The "analysis means" refers to a device or software that analyzes the acquired emotion data and identifies the user's emotional state.

[0220] An "advertising selector" is a device or software for selecting an optimized advertisement based on the identified emotional state.

[0221] The "monitoring means" refers to a device or software for monitoring and recording user reaction data to advertisements.

[0222] The "transmission means" refers to a device or software that transmits user reaction data after an advertisement is displayed to a server and evaluates the effectiveness of the advertisement.

[0223] "Algorithm improvement means" refers to devices or software that continuously improve ad delivery algorithms based on analyzed reaction data.

[0224] The "advertising distribution means" refers to a device or software for distributing selected advertisements to users.

[0225] A "wearable device" is a device that is worn on a user's body and is used to acquire biometric information such as pulse data.

[0226] A "camera" is a photographing device for capturing the user's facial expression.

[0227] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

[0228] 1. System Overview

[0229] This system collects and analyzes user emotional data and delivers optimal advertisements based on the results. The system consists of an emotion sensor, analysis means, advertisement selection means, monitoring means, transmission means, algorithm improvement means, and advertisement delivery means.

[0230] 2. Collecting Emotional Data

[0231] The user wears the wearable device and runs a specific application. The device captures the user's facial expressions using the front camera and receives pulse data via Bluetooth. These emotion data are collected in real time by the device.

[0232] 3. Emotion Data Analysis

[0233] The device analyzes the collected pulse data and facial expression data using an AI module. The software used includes an emotion recognition model using Keras and facial expression capture using OpenCV. The AI ​​module analyzes pulse fluctuations and facial expression characteristics to identify the user's emotional state (e.g., relaxed, excited, happy, etc.).

[0234] 4. Sending Emotional Data

[0235] The device sends anonymized analysis results to a server, including data such as device ID and timestamp to protect privacy.

[0236] 5. Ad optimization

[0237] The server stores the received emotional data in a database and matches it with the advertisement database to select an advertisement that best suits the emotional state. This process also takes into account the user's past ad viewing history and user action history.

[0238] 6. Delivery of advertisements

[0239] The server sends the optimized ad data to the device, which then displays the ad to the user and monitors the user's reaction during viewing. The software used includes a web view and an external browser for displaying the ad.

[0240] 7. Feedback and algorithm improvements

[0241] The device sends user reaction data (such as gaze duration, clicks, and scrolling) after the ad is displayed to the server. The server analyzes the received reaction data and evaluates the effectiveness of the ad. It analyzes the characteristics of ads that were less effective and reflects this in future targeting. It continuously improves the ad delivery algorithm based on new emotional and reaction data.

[0242] Adding specific examples

[0243] When a user launches a smartphone app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while viewing them. The reaction data is sent back to the server and used to select advertisements for future use.

[0244] Prompt Sentence Examples

[0245] "Generate an ad to show me when I'm relaxing. The dataset contains facial expression data and pulse rate data, and the information has been analyzed using a Keras model."

[0246] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0247] Step 1:

[0248] The user puts on the wearable device and launches the smartphone application. The wearable device acquires pulse data and transmits it to the device via Bluetooth. The device's front camera also captures the user's facial expressions and obtains facial expression data. This data is collected in real time.

[0249] Input: User's pulse data, facial expression data

[0250] Output: Real-time pulse data, facial expression data

[0251] Specific operation: The user launches a smartphone application and acquires emotion data using the wearable device and the device's in-camera.

[0252] Step 2:

[0253] The device analyzes the acquired pulse data and facial expression data using an AI module. The pulse data is analyzed for fluctuation patterns, and the facial expression data is converted to grayscale, the facial area is cut out and resized, and then analyzed using an emotion recognition model. The AI ​​module identifies the user's emotional state (relaxed, excited, happy, etc.).

[0254] Input: Real-time pulse data, facial expression data

[0255] Output: Parsed emotional state data

[0256] Specific operation: The device's AI module analyzes pulse data and facial expression data to identify emotional states such as relaxation, excitement, and joy.

[0257] Step 3:

[0258] The device anonymizes the analyzed emotion data and sends it to the server, along with other data such as the device ID and timestamp. The emotion data is anonymized to protect privacy.

[0259] Input: Parsed emotional state data

[0260] Output: Anonymized emotion data sent to the server

[0261] Specific operation: The device anonymizes the analysis results and sends them to the server.

[0262] Step 4:

[0263] The server stores the received emotion data in a database and matches it with the advertisement database. Taking into account the user's past ad display history and action history, the server selects advertisements that best fit the user's emotional state.

[0264] Input: Anonymized emotion data sent to the server

[0265] Output: Optimized advertising data

[0266] Specific operation: The server stores the emotion data in a database and matches it with the advertisement database to select the most suitable advertisement.

[0267] Step 5:

[0268] The server then sends the optimized advertising data to the device, which includes the advertising ID, display timing, and URL of the advertising content.

[0269] Input: Optimized Ad Data

[0270] Output: Advertising data sent to the device

[0271] Specific operation: The server sends the selected advertising data to the terminal.

[0272] Step 6:

[0273] The device displays advertisements to the user based on the received advertising data. The timing and format of the advertisements depend on the content. The device also monitors the user's reactions (e.g., gaze retention time, clicks, scrolling) while the advertisements are displayed.

[0274] Input: Advertising data sent to the device

[0275] Output: Ads shown to users and their reaction data

[0276] Specific operation: The device displays advertisements and monitors the user's reactions.

[0277] Step 7:

[0278] The device sends user reaction data after the advertisement is displayed to the server, including gaze duration, click actions, etc.

[0279] Input: User reaction data

[0280] Output: Reaction data sent to the server

[0281] Specific operation: The device sends the user's reaction data to the server.

[0282] Step 8:

[0283] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It analyzes the characteristics of ineffective ads and reflects them in future targeting. It continuously improves the ad delivery algorithm based on new emotion and reaction data.

[0284] Input: Reaction data sent to the server

[0285] Output: Improved ad serving algorithm

[0286] What it does: The server analyzes the reaction data, evaluates the effectiveness of the ads, and improves the algorithm.

[0287] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0288] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes an emotion sensor means, an analysis means, an emotion engine, an advertisement selection means, and an advertisement delivery means. Below, we will explain the details of each component and how they work together.

[0289] 1. Collecting Emotional Data

[0290] User

[0291] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[0292] Terminal

[0293] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[0294] 2. Emotion Data Analysis

[0295] Terminal

[0296] The collected pulse data and facial expression data are sent to the emotion engine, which uses machine learning algorithms to analyze the data and identify the user's emotional state (e.g., joy, sadness, surprise, etc.), allowing the user's emotional state to be updated in real time.

[0297] Emotion Engine

[0298] Based on the received data, the user's emotional state is analyzed and the results are returned to the device. The emotion engine also accumulates past data and continues to learn, improving the accuracy of its analysis.

[0299] 3. Sending Emotional Data

[0300] Terminal

[0301] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[0302] 4. Ad optimization

[0303] server

[0304] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[0305] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it selects ads related to relaxation, and if the user is excited, it selects ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[0306] 5. Delivery of advertisements

[0307] server

[0308] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[0309] Terminal

[0310] Based on the received advertising data, advertisements are displayed to the user. The timing and format of the display depend on the content.

[0311] The system monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, and scrolling of the screen.

[0312] 6. Feedback and algorithm improvements

[0313] Terminal

[0314] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze, click actions, scroll position, etc.

[0315] server

[0316] Analyze the reaction data received to evaluate the effectiveness of the ads, analyze the characteristics of the ads that were less effective, and generate feedback to discontinue or modify them.

[0317] Improve and update ad delivery algorithms based on new emotional and reaction data, which continuously improves the accuracy and effectiveness of ad delivery.

[0318] Specific examples

[0319] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect that the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[0320] In this way, the use of an emotion engine makes it possible to realize an advertising delivery system based on the user's real-time emotional state, which can attract user interest with greater precision than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[0321] The processing flow will be explained below.

[0322] Step 1:

[0323] A user puts on a wearable device and launches an application (e.g., a VR app) that collects emotion data. The application captures facial expressions using the front camera.

[0324] Step 2:

[0325] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[0326] Step 3:

[0327] The device sends the received pulse data to the emotion engine, which analyzes the pulse fluctuations to detect changes in the user's heart rate. For example, an increase in heart rate indicates stress or excitement.

[0328] Step 4:

[0329] The device sends the captured facial image to the emotion engine, which analyzes the facial image, extracts facial features, and identifies the user's emotional state. For example, a smile indicates "joy."

[0330] Step 5:

[0331] The emotion engine integrates both the pulse data and facial expression data to determine the user's final emotional state (e.g., "relaxed," "excited," etc.), and the analysis results are returned to the device.

[0332] Step 6:

[0333] The device then anonymizes the analyzed emotional data, protecting privacy, and transmits it to a server, including the emotional state, device ID, and timestamp.

[0334] Step 7:

[0335] The server stores the received emotional data in a database, and labels it according to the emotional state.

[0336] Step 8:

[0337] The server analyzes the emotional data, compares it with an advertisement database, and selects the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select a relaxation-related advertisement.

[0338] Step 9:

[0339] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[0340] Step 10:

[0341] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the application's context.

[0342] Step 11:

[0343] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[0344] Step 12:

[0345] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[0346] Step 13:

[0347] The server analyzes the reaction data received and evaluates the effectiveness of the advertisement. The characteristics of advertisements that were less effective are analyzed and reflected in future advertisement selection.

[0348] Step 14:

[0349] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[0350] Example 2

[0351] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] Conventional ad delivery systems were unable to accurately reflect the user's real-time emotional state, making it difficult to deliver appropriate ads. Furthermore, they lacked a mechanism for evaluating the effectiveness of ads and providing feedback for future ad delivery, making it difficult to adequately optimize ads. As a result, ads failed to attract user interest and were less effective.

[0353] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0354] In this invention, the server includes sensor means for acquiring user emotional data, analysis means for analyzing the emotional data to identify the user's emotional state, transmission means for anonymizing the identified emotional state and transmitting it to the server, advertisement selection means for selecting an optimized advertisement based on the emotional state and past behavioral history, advertisement distribution means for delivering the selected advertisement to the user, means for monitoring user reaction data during advertisement delivery, and feedback means for transmitting the monitored data to the server and improving the advertisement delivery algorithm. This enables appropriate advertisement delivery based on the user's real-time emotional state, and enables continuous feedback and optimization to improve advertisement effectiveness.

[0355] The term "sensor means" refers to a device for acquiring emotional data of a user.

[0356] The term "analysis means" refers to a device that has the function of analyzing acquired emotion data and identifying the user's emotional state.

[0357] "Transmitting means" refers to a device that has the function of anonymizing the identified emotional state and transmitting it to a server.

[0358] "Advertisement selection means" refers to a device that has the function of selecting an optimized advertisement based on emotional state and past behavioral history.

[0359] "Advertisement distribution means" refers to a device that has the function of distributing selected advertisements to users.

[0360] The term "monitoring means" refers to a device that has the function of monitoring user reaction data during advertisement distribution.

[0361] "Feedback means" refers to a device that has the function of transmitting monitored data to a server and generating feedback to improve the ad delivery algorithm.

[0362] "Emotion data" refers to data that indicates the user's emotional state, and includes facial expressions, pulse rate, and other biological information.

[0363] "Emotional State" refers to the user's current psychological and physiological state as identified by analytical means.

[0364] "Reaction data" refers to data regarding the reactions and actions of users during advertisement delivery.

[0365] "Anonymization" refers to the act of processing data so that it cannot be used to identify individuals.

[0366] "Server" refers to a computer system that receives emotion data and reaction data via a network and performs analysis and advertisement selection.

[0367] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes emotion sensor means, analysis means, transmission means, advertisement selection means, advertisement delivery means, monitoring means, and feedback means. Below, we will provide details of each component and how they work together, along with specific examples.

[0368] Collecting Emotional Data

[0369] User

[0370] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[0371] Terminal

[0372] The device collects pulse and facial expression data in real time from the front camera and the wearable device. Pulse data is transmitted to the device via Bluetooth, and facial expression data is captured by the front camera.

[0373] Emotional Data Analysis

[0374] Terminal

[0375] The device temporarily stores the collected pulse data and facial expression data locally and sends it to the emotion engine.

[0376] Emotion Engine

[0377] The emotion engine uses machine learning algorithms (e.g., neural networks) to analyze the data and identify the user's emotional state, which may include happiness, sadness, surprise, relaxation, etc., and returns the analysis results to the device.

[0378] Sending emotional data

[0379] Terminal

[0380] After receiving the analysis results from the emotion engine, the device anonymizes the emotion data and sends it to the server along with the device ID and timestamp to protect privacy.

[0381] Ad optimization

[0382] server

[0383] The server stores the received emotion data in a database and runs an algorithm to select the most appropriate ad based on the emotion data, past ad display history, and user action history.

[0384] Ad serving

[0385] server

[0386] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[0387] Terminal

[0388] The device displays advertisements to the user based on the received advertising data. The timing and format of the display depend on the content. The device also monitors the user's reactions while the advertisement is being displayed (for example, how long the user's gaze remains on the advertisement, whether or not the user clicks, whether the user scrolls the screen, etc.).

[0389] Feedback and algorithm improvements

[0390] Terminal

[0391] After the ad is displayed, the device sends the user's reaction data to the server, including gaze duration, click actions, scroll position, etc.

[0392] server

[0393] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback to discontinue or modify them, and updates and improves the ad delivery algorithm based on the new emotion and reaction data.

[0394] Implementing specific examples

[0395] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect when the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[0396] Prompt Sentence Examples

[0397] An example prompt for a generative AI model might look something like this:

[0398] "Select the best ad based on the following emotional state data:

[0399] Emotional state: Relaxed

[0400] Device ID: 12345XYZ

[0401] Timestamp: 2023-10-09 10:00:00

[0402] Advertising history: Hot spring trips, yoga classes

[0403] When this prompt sentence is input into a generative AI model, the optimal advertisement is selected based on the emotional data.

[0404] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0405] Step 1: Collecting emotion data

[0406] User

[0407] The user puts on the wearable device and launches the VR application, which starts capturing the user's facial expressions through the front camera.

[0408] Terminal

[0409] The terminal receives pulse data from the wearable device via Bluetooth, and simultaneously captures the user's facial expression data in real time using the front camera.

[0410] input

[0411] Pulse data and facial expression data

[0412] output

[0413] Raw data temporarily stored in local data storage

[0414] Step 2: Analyze the emotion data

[0415] Terminal

[0416] The device sends the stored pulse data and facial expression data to the emotion engine.

[0417] Emotion Engine

[0418] The emotion engine uses machine learning algorithms to analyze data and identify the user's emotional state. For example, it uses a neural network to analyze facial expression data and pulse rate data to identify an emotional state such as "relaxed."

[0419] input

[0420] Collected pulse data and facial expression data

[0421] Data Processing

[0422] Data analysis using machine learning algorithms

[0423] output

[0424] The user's emotional state (e.g., "Relaxed")

[0425] Step 3: Sending emotion data

[0426] Terminal

[0427] The device anonymizes the emotional state obtained from the emotion engine, and sends it to the server with a device ID and a timestamp.

[0428] input

[0429] Analyzed emotional state, device ID, timestamp

[0430] Data Processing

[0431] Data anonymization

[0432] output

[0433] Anonymized emotion data (e.g., "Relaxed, Device ID: 12345XYZ, Timestamp: 2023-10-09 10:00:00")

[0434] Step 4: Optimize your ads

[0435] server

[0436] The server stores the received emotional data in a database, and then selects the most appropriate ad based on the emotional data, past ad display history, and user action history.

[0437] input

[0438] Anonymized emotional data, past ad viewing history, and user action history

[0439] Data Calculation

[0440] Implementing ad selection algorithms using generative AI models

[0441] output

[0442] Optimized advertising data (e.g., "hot spring trip advertisement")

[0443] Step 5: Serving Ads

[0444] server

[0445] The server then sends the selected advertising data to the device, including the advertising ID, display timing, and the URL of the advertising content.

[0446] Terminal

[0447] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the content.

[0448] input

[0449] Optimized Advertising Data

[0450] output

[0451] Ad content displayed to users

[0452] Step 6: Monitor while your ad is showing

[0453] Terminal

[0454] While the ad is displayed, the device monitors the user's reactions (such as gaze retention, ad clicks, and scrolling).

[0455] input

[0456] User gaze data, click data, scroll data

[0457] output

[0458] User reaction data stored as log files

[0459] Step 7: Feedback and algorithm refinement

[0460] Terminal

[0461] After the advertisement is displayed, the terminal transmits the user's reaction data to the server.

[0462] server

[0463] The server analyzes the received reaction data and evaluates the effectiveness of the advertisement. It analyzes the characteristics of ineffective advertisements and generates feedback for future advertisements. It also updates and improves the ad delivery algorithm based on the new emotional and reaction data.

[0464] input

[0465] User reaction data

[0466] Data Calculation

[0467] Evaluating advertising effectiveness using analytical algorithms and creating new advertising selection algorithms

[0468] output

[0469] Updated ad serving algorithm

[0470] Through the above processing steps, this system is able to deliver highly accurate advertisements and provide continuous feedback on their effectiveness based on the user's real-time emotional state.

[0471] (Application example 2)

[0472] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0473] Conventional ad delivery systems have had the challenge of being difficult to target accurately based on user interests and states. In particular, ad delivery utilizing emotional data has been rare, making it impossible to provide ads that match the user's emotional state in real time. This has led to problems such as a decline in user engagement and difficulty for advertisers in measuring effectiveness.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0475] In this invention, the server includes a transmission means for anonymizing a user's emotional data and transmitting it to the server while protecting their privacy, a feedback means for improving an ad delivery algorithm based on the emotional data and a history of past ad display, and a means for acquiring pulse data and facial expression data in real time from the emotional data, the in-camera, and the wearable device, thereby enabling highly targeted ad delivery based on the user's real-time emotional state.

[0476] The "sensor means" is a device for acquiring emotional data of the user.

[0477] The "analysis means" is a device that analyzes the acquired emotional data and identifies the emotional state of the user.

[0478] The "selection means" is a device that selects the most suitable advertisement based on the analyzed emotional state.

[0479] The "distribution means" is a device that distributes the selected advertisement to the user.

[0480] The "transmission means" is a device that anonymizes the user's emotional data and transmits it to the server while protecting privacy.

[0481] A "feedback means" is a device that improves the ad delivery algorithm based on emotional data and past ad display history.

[0482] A "wearable device" is a device worn by a user to acquire pulse data.

[0483] The "imaging device" is a camera device for capturing the user's facial expression.

[0484] The "emotional state" is the mental state of the user identified as a result of analyzing the emotion data.

[0485] The "advertising delivery algorithm" is an algorithm for controlling the display of advertisements based on emotional data and past advertisement display history.

[0486] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of the following components: sensor means, analysis means, selection means, delivery means, transmission means, and feedback means.

[0487] 1. Collecting Emotional Data

[0488] A user wears a wearable device and runs a specific application (e.g., an app for a smartphone or smart glasses). This application captures the user's facial expressions using an in-camera or an imaging device. Pulse data is also acquired from the wearable device in real time. This data is then transmitted to a terminal via Bluetooth.

[0489] 2. Emotion Data Analysis

[0490] The device sends the collected pulse data and facial expression data to the analysis means, which incorporates a machine learning algorithm that analyzes the emotional data to identify the user's emotional state (e.g., joy, sadness, surprise, etc.). The analysis results are updated in real time, and the emotional state is returned to the device.

[0491] 3. Sending Emotional Data

[0492] The device then anonymizes the analyzed emotional data and transmits it to a server, protecting privacy. The transmitted data includes the emotional state, device ID, and timestamp.

[0493] 4. Ad optimization

[0494] The server stores the received emotional data in a database. Based on the stored emotional data, it matches it with the advertisement database and selects advertisements that suit the user's emotional state. For example, if the user is relaxed, it will select relaxation-related advertisements, and if the user is excited, it will select exciting advertisements. It also takes into account the user's past ad viewing history and user action history.

[0495] 5. Delivery of advertisements

[0496] The server sends the optimized advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content. The device displays the advertisement to the user based on the received advertising data. While the advertisement is being displayed, the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) are monitored.

[0497] 6. Feedback and algorithm improvements

[0498] The device sends data on the user's reaction after the advertisement is displayed to the server, including the duration of gaze, click actions, scroll position, etc.

[0499] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback for corrections, and improves and updates the ad delivery algorithm based on new emotion and reaction data to continuously improve the accuracy and effectiveness of ad delivery.

[0500] Specific examples

[0501] When a user turns on their smart glasses to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an analysis means, which uses a machine learning algorithm to detect relaxation. The results are returned to the device, which then analyzes the data and sends the analyzed emotional data to the server. The server selects an advertisement suitable for the relaxation state (e.g., a hot spring trip or relaxation service) and sends it to the device. The device displays the advertisement to the user and monitors the user's reaction while viewing it. The reaction data is then sent back to the server and used to select future advertisements. In this way, using an emotion engine enables an advertising delivery system based on the user's real-time emotional state.

[0502] Prompt Sentence Examples

[0503] Collect real-time emotional data while the user is wearing the smart glasses, and if the user's current emotional state is analyzed as "relaxed," select and display appropriate relaxation-related advertisements, such as advertisements for hot spring trips or massage services.

[0504] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0505] Step 1:

[0506] The device uses a camera and a wearable device to collect user emotional data. Specifically, it captures the user's facial expressions using the in-camera and obtains pulse data from the wearable device via Bluetooth.

[0507] Input: User's facial expression data (image) and pulse data

[0508] Output: Captured facial expression data and acquired pulse data

[0509] Step 2:

[0510] The terminal transmits the collected facial expression data and pulse rate data to the analysis means, which uses a machine learning algorithm to analyze the emotion data and identify the user's emotional state.

[0511] Input: Captured facial expression data and acquired pulse data

[0512] Data processing / computation: Identifying emotional states using machine learning algorithms

[0513] Output: Identified user emotional state (happy, sad, surprised, etc.)

[0514] Step 3:

[0515] The device anonymizes the analyzed emotion data and transmits it to a server while protecting privacy.

[0516] Input: Identified user emotional state

[0517] Data processing / calculation: Data anonymization

[0518] Output: Anonymized emotional data (emotional state, device ID, timestamp)

[0519] Step 4:

[0520] The server stores the received emotional data in a database, matches the stored emotional data with the advertisement database, and selects advertisements that match the emotional state.

[0521] Input: Anonymized emotion data

[0522] Data processing / calculation: database search and matching processing

[0523] Output: Selected ad data (ad ID, display timing, ad content URL, etc.)

[0524] Step 5:

[0525] The server transmits the selected advertisement data to the terminal, and the terminal displays an advertisement to the user based on the received advertisement data.

[0526] Input: Selected advertising data

[0527] Data processing / calculation: Sending advertising data and displaying advertisements

[0528] Output: Ad content shown to the user

[0529] Step 6:

[0530] The device monitors the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) while the advertisement is displayed.

[0531] Input: User reaction data

[0532] Data processing / calculation: Reaction data capture and recording

[0533] Output: Captured reaction data

[0534] Step 7:

[0535] The device sends user reaction data after the ad is displayed to the server. The server analyzes the received reaction data, evaluates the effectiveness of the ad, and generates feedback to help select future ads.

[0536] Input: Captured reaction data

[0537] Data processing / calculation: Effect analysis and feedback generation

[0538] Output: Effect analysis results and feedback information

[0539] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0540] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0541] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0542] [Second embodiment]

[0543] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0544] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0545] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0546] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0547] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0548] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0549] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0550] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0551] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0552] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0553] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0554] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0555] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components, including emotion sensor means, analysis means, advertisement selection means, and advertisement delivery means. Below, we will explain the details of each component and how they work together.

[0556] 1. Collecting Emotional Data

[0557] User

[0558] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[0559] Terminal

[0560] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[0561] 2. Emotion Data Analysis

[0562] Terminal

[0563] The collected pulse data and facial expression data are analyzed by an AI module, which analyzes pulse fluctuations and facial expression characteristics to identify the user's current emotional state (e.g., relaxed, excited, happy, etc.).

[0564] 3. Sending Emotional Data

[0565] Terminal

[0566] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[0567] 4. Ad optimization

[0568] server

[0569] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[0570] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it will select ads related to relaxation, and if the user is excited, it will select ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[0571] 5. Delivery of advertisements

[0572] server

[0573] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[0574] Terminal

[0575] Based on the received advertising data, the advertisement is displayed to the user. The timing and format of the display depend on the content.

[0576] It monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, scrolling, etc.

[0577] 6. Feedback and algorithm improvements

[0578] Terminal

[0579] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze and click action.

[0580] server

[0581] The received reaction data is analyzed to evaluate the effectiveness of the advertisement.

[0582] The characteristics of ineffective ads are analyzed and reflected in future targeting, for example, determining that certain types of ads are inappropriate for certain emotional states.

[0583] Continually improve ad delivery algorithms based on new emotional and reaction data.

[0584] Specific examples

[0585] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for the relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is sent back to the server and used to select advertisements for future use.

[0586] In this way, an advertising delivery system based on a user's real-time emotional state can attract users' interest with greater accuracy than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[0587] The processing flow will be explained below.

[0588] Step 1:

[0589] The user puts on the wearable device and launches an application (e.g., a VR app) that collects emotion data.

[0590] Step 2:

[0591] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[0592] Step 3:

[0593] The device analyzes the pulse data received and detects fluctuations in the user's heart rate, as increases or decreases in heart rate can indicate changes in mood.

[0594] Step 4:

[0595] The AI ​​module analyzes facial images captured by the device and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.) from their facial features. Changes in facial expression are analyzed as movements of facial parts.

[0596] Step 5:

[0597] The device combines the analysis results of the pulse data and facial expression data to determine the user's final emotional state. For example, if the heart rate is stable and the user is smiling, the device will be classified as "Relaxed."

[0598] Step 6:

[0599] The device then anonymizes the analyzed emotional data and sends it to a server, including the emotional state, device ID, and timestamp.

[0600] Step 7:

[0601] The server stores the received emotion data in a database, and when storing it, it labels the data according to the user's emotional state.

[0602] Step 8:

[0603] The server analyzes the emotional data and compares it with an advertising database to select the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select an advertisement related to relaxation.

[0604] Step 9:

[0605] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[0606] Step 10:

[0607] The device displays advertisements to the user based on the received advertisement data. The timing of the display depends on the application status.

[0608] Step 11:

[0609] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[0610] Step 12:

[0611] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[0612] Step 13:

[0613] The server analyzes the reaction data received and evaluates the effectiveness of the ads, analyzes the characteristics of the ads that were less effective, and generates feedback to discontinue or modify them.

[0614] Step 14:

[0615] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[0616] Example 1

[0617] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] Conventional ad delivery systems have had difficulty delivering optimal ads that reflect the user's emotional state in real time. Furthermore, there were insufficient methods for evaluating the effectiveness of ads and reflecting this in future targeting, making it difficult to simultaneously increase the satisfaction of both users and advertisers.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0620] In this invention, the server includes an advertisement selection means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the emotional state, a monitoring means for monitoring the user's reaction while the advertisement is being displayed, and an algorithm improvement means for evaluating the effectiveness of the advertisement based on the monitoring results and reflecting the results in subsequent targeting. This makes it possible to optimize advertisements that reflect the user's emotional state in real time and to continuously improve the advertisement distribution algorithm.

[0621] The "emotion sensor means" is a sensor for acquiring the user's emotional data, and specifically is a device including a wearable device for acquiring pulse data and a camera for capturing the user's facial expression.

[0622] The "analysis means" is a means for analyzing the emotional data acquired by the emotion sensor means and identifying the emotional state of the user. Specifically, it performs processing to analyze the emotional state using an AI module.

[0623] The "transmission means" is a means for anonymizing the analyzed emotion data and transmitting it to the server while protecting privacy.

[0624] The "advertisement selection means" is a means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the user's emotional state.

[0625] The "advertisement distribution means" is a means for distributing the selected advertisement to the user.

[0626] The "monitoring means" is a means for monitoring users' reactions while the advertisement is being displayed and collecting the data. Specifically, it is a means for recording gaze retention time, click actions, etc.

[0627] "Algorithm improvement measures" are measures for continuously improving the ad delivery algorithm by evaluating the effectiveness of advertising based on monitoring results and reflecting the evaluation results in future targeting.

[0628] This invention is a system that collects and analyzes real-time user emotional data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components: an emotion sensor means, an analysis means, a transmission means, an advertisement selection means, an advertisement delivery means, a monitoring means, and an algorithm improvement means. Below, we will explain in detail each component and how they work together.

[0629] Collecting Emotional Data

[0630] User

[0631] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[0632] Terminal

[0633] The device uses the in-camera to capture the user's facial expression data in real time, and simultaneously obtains pulse data from the wearable device via Bluetooth.

[0634] Emotional Data Analysis

[0635] Terminal

[0636] The device passes the acquired facial expression and pulse data to an AI module, which uses deep learning frameworks such as TensorFlow and PyTorch.

[0637] The device's AI module analyzes pulse fluctuations and facial features to identify the user's current emotional state, which can be relaxation, excitement, joy, etc.

[0638] Sending emotional data

[0639] Terminal

[0640] The device anonymizes the analyzed emotion data by removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[0641] The anonymized emotion data is sent to a server.

[0642] Ad optimization

[0643] server

[0644] The server stores the received emotion data in a database, which uses a relational database such as MySQL or PostgreSQL.

[0645] Based on the stored emotion data, the server matches it with an advertisement database: if the user is relaxed, it selects advertisements related to relaxation from the advertisement database.

[0646] The server takes into consideration the past ad display history and the user's action history to select the most suitable ad.

[0647] Ad serving

[0648] server

[0649] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[0650] Terminal

[0651] The device will then display the advertisement on the user's screen based on the received advertising data. The timing and format of the display will depend on the content. For example, the advertisements will be designed to appear naturally during the VR experience.

[0652] While the ad is being displayed, the device monitors the user's reactions, specifically recording actions such as gaze retention time, clicks on the ad, and scrolling.

[0653] Feedback and algorithm improvements

[0654] Terminal

[0655] After the ad is displayed, the user's reaction data, including gaze duration and click action, is sent to the server.

[0656] server

[0657] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It also analyzes the characteristics of ads that were less effective and reflects this in future targeting.

[0658] New emotional and reaction data will be used to continuously improve ad serving algorithms, for example by determining that certain types of ads are inappropriate for certain emotional states.

[0659] Specific examples

[0660] For example, if a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The device's AI module analyzes this data and determines that the user is relaxed. The device anonymizes this analysis result and sends it to a server. The server then selects advertisements appropriate for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions, such as gaze duration and clicks. This data is then sent back to the server and used to analyze the effectiveness of the advertisements and improve the algorithm.

[0661] The present invention enables optimal advertisement delivery based on the user's real-time emotional state, improving advertisement accuracy and user satisfaction.

[0662] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0663] Step 1:

[0664] User provides emotion data

[0665] The user puts on the wearable device and launches the VR application. The front camera captures facial expressions. At the same time, the terminal acquires pulse data from the wearable device via Bluetooth.

[0666] Input: User's facial expression data, pulse data

[0667] Output: Raw data collected

[0668] Step 2:

[0669] The device analyzes the emotional data

[0670] The device passes the acquired facial expression and pulse data to an AI module, which then uses a deep learning framework (such as TensorFlow or PyTorch) to analyze the user's emotional state.

[0671] Input: facial expression data, pulse data

[0672] Data processing / calculation: Analysis and processing using facial expression data and pulse rate data

[0673] Output: User's emotional state (relaxed, excited, happy, etc.)

[0674] Step 3:

[0675] The device sends analysis data

[0676] The device anonymizes the analyzed emotion data and transmits it to a server in a privacy-preserving manner, which involves removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[0677] Input: Parsed emotion data

[0678] Data processing: Data anonymization

[0679] Output: Anonymized emotion data

[0680] Step 4:

[0681] The server receives and stores emotion data.

[0682] The server stores the received emotion data in a database using a relational database (e.g., MySQL or PostgreSQL).

[0683] Input: Anonymized emotion data

[0684] Data processing: data storage processing

[0685] Output: Data stored in the database

[0686] Step 5:

[0687] Server optimizes ads

[0688] The server matches the stored emotional data with the advertisement database. If the user is relaxed, it selects advertisements related to relaxation, taking into account the user's past ad viewing history and user action history.

[0689] Input: Stored emotion data

[0690] Data processing / calculation: Matching of emotion data with advertising database

[0691] Output: Optimized advertising data

[0692] Step 6:

[0693] The server sends the advertisement

[0694] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[0695] Input: Optimized Ad Data

[0696] Data processing: Generation of transmission data

[0697] Output: Data sent to the terminal

[0698] Step 7:

[0699] The device displays the ad and monitors the reaction

[0700] The device displays the advertisement on the user's screen based on the received advertisement data, and monitors the user's reactions (e.g., gaze retention time, clicks, scrolling, etc.) while the advertisement is displayed.

[0701] Input: Transmission data (advertising ID, display timing, content URL)

[0702] Data processing: Displaying advertisements, monitoring user reactions

[0703] Output: Reaction data (gaze duration, click actions, etc.)

[0704] Step 8:

[0705] The device sends reaction data

[0706] The terminal transmits reaction data after the advertisement is displayed to the server.

[0707] Input: Reaction data

[0708] Data processing: Preparing data for transmission

[0709] Output: Reaction data to the server

[0710] Step 9:

[0711] The server analyzes the feedback and improves the algorithm

[0712] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It reanalyzes the characteristics of ads that were ineffective and reflects them in future targeting. It also continuously improves the ad delivery algorithm based on new emotion and reaction data.

[0713] Input: Reaction data

[0714] Data processing / calculation: Analysis of reaction data, updating of algorithms

[0715] Output: Improved ad serving algorithm

[0716] (Application example 1)

[0717] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0718] Conventional ad delivery systems deliver ads without considering the user's emotional state, making it difficult to attract users' attention. They also lacked a means to understand the effectiveness of ads in real time and reflect this in future ad optimizations. Furthermore, there was no mechanism for continuously improving the ad delivery algorithm based on user reactions, making effective targeting impossible.

[0719] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0720] In this invention, the server includes an emotion data analysis means, a transmission means for transmitting user reaction data after displaying an advertisement to the server and evaluating the effectiveness of the advertisement, and an algorithm improvement means for reselecting advertisements using an improved advertisement distribution algorithm, thereby enabling effective advertisement distribution based on the user's emotional state and continuous optimization of advertisements based on the reaction data.

[0721] An "emotion sensor means" is a device or apparatus used to acquire emotion data of a user.

[0722] The "analysis means" refers to a device or software that analyzes the acquired emotion data and identifies the user's emotional state.

[0723] An "advertising selector" is a device or software for selecting an optimized advertisement based on the identified emotional state.

[0724] The "monitoring means" refers to a device or software for monitoring and recording user reaction data to advertisements.

[0725] The "transmission means" refers to a device or software that transmits user reaction data after an advertisement is displayed to a server and evaluates the effectiveness of the advertisement.

[0726] "Algorithm improvement means" refers to devices or software that continuously improve ad delivery algorithms based on analyzed reaction data.

[0727] The "advertising distribution means" refers to a device or software for distributing selected advertisements to users.

[0728] A "wearable device" is a device that is worn on a user's body and is used to acquire biometric information such as pulse data.

[0729] A "camera" is a photographing device for capturing the user's facial expression.

[0730] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

[0731] 1. System Overview

[0732] This system collects and analyzes user emotional data and delivers optimal advertisements based on the results. The system consists of an emotion sensor, analysis means, advertisement selection means, monitoring means, transmission means, algorithm improvement means, and advertisement delivery means.

[0733] 2. Collecting Emotional Data

[0734] The user wears the wearable device and runs a specific application. The device captures the user's facial expressions using the front camera and receives pulse data via Bluetooth. These emotion data are collected in real time by the device.

[0735] 3. Emotion Data Analysis

[0736] The device analyzes the collected pulse data and facial expression data using an AI module. The software used includes an emotion recognition model using Keras and facial expression capture using OpenCV. The AI ​​module analyzes pulse fluctuations and facial expression characteristics to identify the user's emotional state (e.g., relaxed, excited, happy, etc.).

[0737] 4. Sending Emotional Data

[0738] The device sends anonymized analysis results to a server, including data such as device ID and timestamp to protect privacy.

[0739] 5. Ad optimization

[0740] The server stores the received emotional data in a database and matches it with the advertisement database to select an advertisement that best suits the emotional state. This process also takes into account the user's past ad viewing history and user action history.

[0741] 6. Delivery of advertisements

[0742] The server sends the optimized ad data to the device, which then displays the ad to the user and monitors the user's reaction during viewing. The software used includes a web view and an external browser for displaying the ad.

[0743] 7. Feedback and algorithm improvements

[0744] The device sends user reaction data (such as gaze duration, clicks, and scrolling) after the ad is displayed to the server. The server analyzes the received reaction data and evaluates the effectiveness of the ad. It analyzes the characteristics of ads that were less effective and reflects this in future targeting. It continuously improves the ad delivery algorithm based on new emotional and reaction data.

[0745] Adding specific examples

[0746] When a user launches a smartphone app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while viewing them. The reaction data is sent back to the server and used to select advertisements for future use.

[0747] Prompt Sentence Examples

[0748] "Generate an ad to show me when I'm relaxing. The dataset contains facial expression data and pulse rate data, and the information has been analyzed using a Keras model."

[0749] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0750] Step 1:

[0751] The user puts on the wearable device and launches the smartphone application. The wearable device acquires pulse data and transmits it to the device via Bluetooth. The device's front camera also captures the user's facial expressions and obtains facial expression data. This data is collected in real time.

[0752] Input: User's pulse data, facial expression data

[0753] Output: Real-time pulse data, facial expression data

[0754] Specific operation: The user launches a smartphone application and acquires emotion data using the wearable device and the device's in-camera.

[0755] Step 2:

[0756] The device analyzes the acquired pulse data and facial expression data using an AI module. The pulse data is analyzed for fluctuation patterns, and the facial expression data is converted to grayscale, the facial area is cut out and resized, and then analyzed using an emotion recognition model. The AI ​​module identifies the user's emotional state (relaxed, excited, happy, etc.).

[0757] Input: Real-time pulse data, facial expression data

[0758] Output: Parsed emotional state data

[0759] Specific operation: The device's AI module analyzes pulse data and facial expression data to identify emotional states such as relaxation, excitement, and joy.

[0760] Step 3:

[0761] The device anonymizes the analyzed emotion data and sends it to the server, along with other data such as the device ID and timestamp. The emotion data is anonymized to protect privacy.

[0762] Input: Parsed emotional state data

[0763] Output: Anonymized emotion data sent to the server

[0764] Specific operation: The device anonymizes the analysis results and sends them to the server.

[0765] Step 4:

[0766] The server stores the received emotion data in a database and matches it with the advertisement database. Taking into account the user's past ad display history and action history, the server selects advertisements that best fit the user's emotional state.

[0767] Input: Anonymized emotion data sent to the server

[0768] Output: Optimized advertising data

[0769] Specific operation: The server stores the emotion data in a database and matches it with the advertisement database to select the most suitable advertisement.

[0770] Step 5:

[0771] The server then sends the optimized advertising data to the device, which includes the advertising ID, display timing, and URL of the advertising content.

[0772] Input: Optimized Ad Data

[0773] Output: Advertising data sent to the device

[0774] Specific operation: The server sends the selected advertising data to the terminal.

[0775] Step 6:

[0776] The device displays advertisements to the user based on the received advertising data. The timing and format of the advertisements depend on the content. The device also monitors the user's reactions (e.g., gaze retention time, clicks, scrolling) while the advertisements are displayed.

[0777] Input: Advertising data sent to the device

[0778] Output: Ads shown to users and their reaction data

[0779] Specific operation: The device displays advertisements and monitors the user's reactions.

[0780] Step 7:

[0781] The device sends user reaction data after the advertisement is displayed to the server, including gaze duration, click actions, etc.

[0782] Input: User reaction data

[0783] Output: Reaction data sent to the server

[0784] Specific operation: The device sends the user's reaction data to the server.

[0785] Step 8:

[0786] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It analyzes the characteristics of ineffective ads and reflects them in future targeting. It continuously improves the ad delivery algorithm based on new emotion and reaction data.

[0787] Input: Reaction data sent to the server

[0788] Output: Improved ad serving algorithm

[0789] What it does: The server analyzes the reaction data, evaluates the effectiveness of the ads, and improves the algorithm.

[0790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0791] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes an emotion sensor means, an analysis means, an emotion engine, an advertisement selection means, and an advertisement delivery means. Below, we will explain the details of each component and how they work together.

[0792] 1. Collecting Emotional Data

[0793] User

[0794] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[0795] Terminal

[0796] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[0797] 2. Emotion Data Analysis

[0798] Terminal

[0799] The collected pulse data and facial expression data are sent to the emotion engine, which uses machine learning algorithms to analyze the data and identify the user's emotional state (e.g., joy, sadness, surprise, etc.), allowing the user's emotional state to be updated in real time.

[0800] Emotion Engine

[0801] Based on the received data, the user's emotional state is analyzed and the results are returned to the device. The emotion engine also accumulates past data and continues to learn, improving the accuracy of its analysis.

[0802] 3. Sending Emotional Data

[0803] Terminal

[0804] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[0805] 4. Ad optimization

[0806] server

[0807] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[0808] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it selects ads related to relaxation, and if the user is excited, it selects ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[0809] 5. Delivery of advertisements

[0810] server

[0811] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[0812] Terminal

[0813] Based on the received advertising data, advertisements are displayed to the user. The timing and format of the display depend on the content.

[0814] The system monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, and scrolling of the screen.

[0815] 6. Feedback and algorithm improvements

[0816] Terminal

[0817] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze, click actions, scroll position, etc.

[0818] server

[0819] Analyze the reaction data received to evaluate the effectiveness of the ads, analyze the characteristics of the ads that were less effective, and generate feedback to discontinue or modify them.

[0820] Improve and update ad delivery algorithms based on new emotional and reaction data, which continuously improves the accuracy and effectiveness of ad delivery.

[0821] Specific examples

[0822] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect that the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[0823] In this way, the use of an emotion engine makes it possible to realize an advertising delivery system based on the user's real-time emotional state, which can attract user interest with greater precision than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[0824] The processing flow will be explained below.

[0825] Step 1:

[0826] A user puts on a wearable device and launches an application (e.g., a VR app) that collects emotion data. The application captures facial expressions using the front camera.

[0827] Step 2:

[0828] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[0829] Step 3:

[0830] The device sends the received pulse data to the emotion engine, which analyzes the pulse fluctuations to detect changes in the user's heart rate. For example, an increase in heart rate indicates stress or excitement.

[0831] Step 4:

[0832] The device sends the captured facial image to the emotion engine, which analyzes the facial image, extracts facial features, and identifies the user's emotional state. For example, a smile indicates "joy."

[0833] Step 5:

[0834] The emotion engine integrates both the pulse data and facial expression data to determine the user's final emotional state (e.g., "relaxed," "excited," etc.), and the analysis results are returned to the device.

[0835] Step 6:

[0836] The device then anonymizes the analyzed emotional data, protecting privacy, and transmits it to a server, including the emotional state, device ID, and timestamp.

[0837] Step 7:

[0838] The server stores the received emotional data in a database, and labels it according to the emotional state.

[0839] Step 8:

[0840] The server analyzes the emotional data, compares it with an advertisement database, and selects the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select a relaxation-related advertisement.

[0841] Step 9:

[0842] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[0843] Step 10:

[0844] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the application's context.

[0845] Step 11:

[0846] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[0847] Step 12:

[0848] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[0849] Step 13:

[0850] The server analyzes the reaction data received and evaluates the effectiveness of the advertisement. The characteristics of advertisements that were less effective are analyzed and reflected in future advertisement selection.

[0851] Step 14:

[0852] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[0853] Example 2

[0854] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0855] Conventional ad delivery systems were unable to accurately reflect the user's real-time emotional state, making it difficult to deliver appropriate ads. Furthermore, they lacked a mechanism for evaluating the effectiveness of ads and providing feedback for future ad delivery, making it difficult to adequately optimize ads. As a result, ads failed to attract user interest and were less effective.

[0856] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0857] In this invention, the server includes sensor means for acquiring user emotional data, analysis means for analyzing the emotional data to identify the user's emotional state, transmission means for anonymizing the identified emotional state and transmitting it to the server, advertisement selection means for selecting an optimized advertisement based on the emotional state and past behavioral history, advertisement distribution means for delivering the selected advertisement to the user, means for monitoring user reaction data during advertisement delivery, and feedback means for transmitting the monitored data to the server and improving the advertisement delivery algorithm. This enables appropriate advertisement delivery based on the user's real-time emotional state, and enables continuous feedback and optimization to improve advertisement effectiveness.

[0858] The term "sensor means" refers to a device for acquiring emotional data of a user.

[0859] The term "analysis means" refers to a device that has the function of analyzing acquired emotion data and identifying the user's emotional state.

[0860] "Transmitting means" refers to a device that has the function of anonymizing the identified emotional state and transmitting it to a server.

[0861] "Advertisement selection means" refers to a device that has the function of selecting an optimized advertisement based on emotional state and past behavioral history.

[0862] "Advertisement distribution means" refers to a device that has the function of distributing selected advertisements to users.

[0863] The term "monitoring means" refers to a device that has the function of monitoring user reaction data during advertisement distribution.

[0864] "Feedback means" refers to a device that has the function of transmitting monitored data to a server and generating feedback to improve the ad delivery algorithm.

[0865] "Emotion data" refers to data that indicates the user's emotional state, and includes facial expressions, pulse rate, and other biological information.

[0866] "Emotional State" refers to the user's current psychological and physiological state as identified by analytical means.

[0867] "Reaction data" refers to data regarding the reactions and actions of users during advertisement delivery.

[0868] "Anonymization" refers to the act of processing data so that it cannot be used to identify individuals.

[0869] "Server" refers to a computer system that receives emotion data and reaction data via a network and performs analysis and advertisement selection.

[0870] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes emotion sensor means, analysis means, transmission means, advertisement selection means, advertisement delivery means, monitoring means, and feedback means. Below, we will provide details of each component and how they work together, along with specific examples.

[0871] Collecting Emotional Data

[0872] User

[0873] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[0874] Terminal

[0875] The device collects pulse and facial expression data in real time from the front camera and the wearable device. Pulse data is transmitted to the device via Bluetooth, and facial expression data is captured by the front camera.

[0876] Emotional Data Analysis

[0877] Terminal

[0878] The device temporarily stores the collected pulse data and facial expression data locally and sends it to the emotion engine.

[0879] Emotion Engine

[0880] The emotion engine uses machine learning algorithms (e.g., neural networks) to analyze the data and identify the user's emotional state, which may include happiness, sadness, surprise, relaxation, etc., and returns the analysis results to the device.

[0881] Sending emotional data

[0882] Terminal

[0883] After receiving the analysis results from the emotion engine, the device anonymizes the emotion data and sends it to the server along with the device ID and timestamp to protect privacy.

[0884] Ad optimization

[0885] server

[0886] The server stores the received emotion data in a database and runs an algorithm to select the most appropriate ad based on the emotion data, past ad display history, and user action history.

[0887] Ad serving

[0888] server

[0889] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[0890] Terminal

[0891] The device displays advertisements to the user based on the received advertising data. The timing and format of the display depend on the content. The device also monitors the user's reactions while the advertisement is being displayed (for example, how long the user's gaze remains on the advertisement, whether or not the user clicks, whether the user scrolls the screen, etc.).

[0892] Feedback and algorithm improvements

[0893] Terminal

[0894] After the ad is displayed, the device sends the user's reaction data to the server, including gaze duration, click actions, scroll position, etc.

[0895] server

[0896] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback to discontinue or modify them, and updates and improves the ad delivery algorithm based on the new emotion and reaction data.

[0897] Implementing specific examples

[0898] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect when the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[0899] Prompt Sentence Examples

[0900] An example prompt for a generative AI model might look something like this:

[0901] "Select the best ad based on the following emotional state data:

[0902] Emotional state: Relaxed

[0903] Device ID: 12345XYZ

[0904] Timestamp: 2023-10-09 10:00:00

[0905] Advertising history: Hot spring trips, yoga classes

[0906] When this prompt sentence is input into a generative AI model, the optimal advertisement is selected based on the emotional data.

[0907] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0908] Step 1: Collecting emotion data

[0909] User

[0910] The user puts on the wearable device and launches the VR application, which starts capturing the user's facial expressions through the front camera.

[0911] Terminal

[0912] The terminal receives pulse data from the wearable device via Bluetooth, and simultaneously captures the user's facial expression data in real time using the front camera.

[0913] input

[0914] Pulse data and facial expression data

[0915] output

[0916] Raw data temporarily stored in local data storage

[0917] Step 2: Analyze the emotion data

[0918] Terminal

[0919] The device sends the stored pulse data and facial expression data to the emotion engine.

[0920] Emotion Engine

[0921] The emotion engine uses machine learning algorithms to analyze data and identify the user's emotional state. For example, it uses a neural network to analyze facial expression data and pulse rate data to identify an emotional state such as "relaxed."

[0922] input

[0923] Collected pulse data and facial expression data

[0924] Data Processing

[0925] Data analysis using machine learning algorithms

[0926] output

[0927] The user's emotional state (e.g., "Relaxed")

[0928] Step 3: Sending emotion data

[0929] Terminal

[0930] The device anonymizes the emotional state obtained from the emotion engine, and sends it to the server with a device ID and a timestamp.

[0931] input

[0932] Analyzed emotional state, device ID, timestamp

[0933] Data Processing

[0934] Data anonymization

[0935] output

[0936] Anonymized emotion data (e.g., "Relaxed, Device ID: 12345XYZ, Timestamp: 2023-10-09 10:00:00")

[0937] Step 4: Optimize your ads

[0938] server

[0939] The server stores the received emotional data in a database, and then selects the most appropriate ad based on the emotional data, past ad display history, and user action history.

[0940] input

[0941] Anonymized emotional data, past ad viewing history, and user action history

[0942] Data Calculation

[0943] Implementing ad selection algorithms using generative AI models

[0944] output

[0945] Optimized advertising data (e.g., "hot spring trip advertisement")

[0946] Step 5: Serving Ads

[0947] server

[0948] The server then sends the selected advertising data to the device, including the advertising ID, display timing, and the URL of the advertising content.

[0949] Terminal

[0950] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the content.

[0951] input

[0952] Optimized Advertising Data

[0953] output

[0954] Ad content displayed to users

[0955] Step 6: Monitor while your ad is showing

[0956] Terminal

[0957] While the ad is displayed, the device monitors the user's reactions (such as gaze retention, ad clicks, and scrolling).

[0958] input

[0959] User gaze data, click data, scroll data

[0960] output

[0961] User reaction data stored as log files

[0962] Step 7: Feedback and algorithm refinement

[0963] Terminal

[0964] After the advertisement is displayed, the terminal transmits the user's reaction data to the server.

[0965] server

[0966] The server analyzes the received reaction data and evaluates the effectiveness of the advertisement. It analyzes the characteristics of ineffective advertisements and generates feedback for future advertisements. It also updates and improves the ad delivery algorithm based on the new emotional and reaction data.

[0967] input

[0968] User reaction data

[0969] Data Calculation

[0970] Evaluating advertising effectiveness using analytical algorithms and creating new advertising selection algorithms

[0971] output

[0972] Updated ad serving algorithm

[0973] Through the above processing steps, this system is able to deliver highly accurate advertisements and provide continuous feedback on their effectiveness based on the user's real-time emotional state.

[0974] (Application example 2)

[0975] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0976] Conventional ad delivery systems have had the challenge of being difficult to target accurately based on user interests and states. In particular, ad delivery utilizing emotional data has been rare, making it impossible to provide ads that match the user's emotional state in real time. This has led to problems such as a decline in user engagement and difficulty for advertisers in measuring effectiveness.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0978] In this invention, the server includes a transmission means for anonymizing a user's emotional data and transmitting it to the server while protecting their privacy, a feedback means for improving an ad delivery algorithm based on the emotional data and a history of past ad display, and a means for acquiring pulse data and facial expression data in real time from the emotional data, the in-camera, and the wearable device, thereby enabling highly targeted ad delivery based on the user's real-time emotional state.

[0979] The "sensor means" is a device for acquiring emotional data of the user.

[0980] The "analysis means" is a device that analyzes the acquired emotional data and identifies the emotional state of the user.

[0981] The "selection means" is a device that selects the most suitable advertisement based on the analyzed emotional state.

[0982] The "distribution means" is a device that distributes the selected advertisement to the user.

[0983] The "transmission means" is a device that anonymizes the user's emotional data and transmits it to the server while protecting privacy.

[0984] A "feedback means" is a device that improves the ad delivery algorithm based on emotional data and past ad display history.

[0985] A "wearable device" is a device worn by a user to acquire pulse data.

[0986] The "imaging device" is a camera device for capturing the user's facial expression.

[0987] The "emotional state" is the mental state of the user identified as a result of analyzing the emotion data.

[0988] The "advertising delivery algorithm" is an algorithm for controlling the display of advertisements based on emotional data and past advertisement display history.

[0989] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of the following components: sensor means, analysis means, selection means, delivery means, transmission means, and feedback means.

[0990] 1. Collecting Emotional Data

[0991] A user wears a wearable device and runs a specific application (e.g., an app for a smartphone or smart glasses). This application captures the user's facial expressions using an in-camera or an imaging device. Pulse data is also acquired from the wearable device in real time. This data is then transmitted to a terminal via Bluetooth.

[0992] 2. Emotion Data Analysis

[0993] The device sends the collected pulse data and facial expression data to the analysis means, which incorporates a machine learning algorithm that analyzes the emotional data to identify the user's emotional state (e.g., joy, sadness, surprise, etc.). The analysis results are updated in real time, and the emotional state is returned to the device.

[0994] 3. Sending Emotional Data

[0995] The device then anonymizes the analyzed emotional data and transmits it to a server, protecting privacy. The transmitted data includes the emotional state, device ID, and timestamp.

[0996] 4. Ad optimization

[0997] The server stores the received emotional data in a database. Based on the stored emotional data, it matches it with the advertisement database and selects advertisements that suit the user's emotional state. For example, if the user is relaxed, it will select relaxation-related advertisements, and if the user is excited, it will select exciting advertisements. It also takes into account the user's past ad viewing history and user action history.

[0998] 5. Delivery of advertisements

[0999] The server sends the optimized advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content. The device displays the advertisement to the user based on the received advertising data. While the advertisement is being displayed, the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) are monitored.

[1000] 6. Feedback and algorithm improvements

[1001] The device sends data on the user's reaction after the advertisement is displayed to the server, including the duration of gaze, click actions, scroll position, etc.

[1002] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback for corrections, and improves and updates the ad delivery algorithm based on new emotion and reaction data to continuously improve the accuracy and effectiveness of ad delivery.

[1003] Specific examples

[1004] When a user turns on their smart glasses to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an analysis means, which uses a machine learning algorithm to detect relaxation. The results are returned to the device, which then analyzes the data and sends the analyzed emotional data to the server. The server selects an advertisement suitable for the relaxation state (e.g., a hot spring trip or relaxation service) and sends it to the device. The device displays the advertisement to the user and monitors the user's reaction while viewing it. The reaction data is then sent back to the server and used to select future advertisements. In this way, using an emotion engine enables an advertising delivery system based on the user's real-time emotional state.

[1005] Prompt Sentence Examples

[1006] Collect real-time emotional data while the user is wearing the smart glasses, and if the user's current emotional state is analyzed as "relaxed," select and display appropriate relaxation-related advertisements, such as advertisements for hot spring trips or massage services.

[1007] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1008] Step 1:

[1009] The device uses a camera and a wearable device to collect user emotional data. Specifically, it captures the user's facial expressions using the in-camera and obtains pulse data from the wearable device via Bluetooth.

[1010] Input: User's facial expression data (image) and pulse data

[1011] Output: Captured facial expression data and acquired pulse data

[1012] Step 2:

[1013] The terminal transmits the collected facial expression data and pulse rate data to the analysis means, which uses a machine learning algorithm to analyze the emotion data and identify the user's emotional state.

[1014] Input: Captured facial expression data and acquired pulse data

[1015] Data processing / computation: Identifying emotional states using machine learning algorithms

[1016] Output: Identified user emotional state (happy, sad, surprised, etc.)

[1017] Step 3:

[1018] The device anonymizes the analyzed emotion data and transmits it to a server while protecting privacy.

[1019] Input: Identified user emotional state

[1020] Data processing / calculation: Data anonymization

[1021] Output: Anonymized emotional data (emotional state, device ID, timestamp)

[1022] Step 4:

[1023] The server stores the received emotional data in a database, matches the stored emotional data with the advertisement database, and selects advertisements that match the emotional state.

[1024] Input: Anonymized emotion data

[1025] Data processing / calculation: database search and matching processing

[1026] Output: Selected ad data (ad ID, display timing, ad content URL, etc.)

[1027] Step 5:

[1028] The server transmits the selected advertisement data to the terminal, and the terminal displays an advertisement to the user based on the received advertisement data.

[1029] Input: Selected advertising data

[1030] Data processing / calculation: Sending advertising data and displaying advertisements

[1031] Output: Ad content shown to the user

[1032] Step 6:

[1033] The device monitors the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) while the advertisement is displayed.

[1034] Input: User reaction data

[1035] Data processing / calculation: Reaction data capture and recording

[1036] Output: Captured reaction data

[1037] Step 7:

[1038] The device sends user reaction data after the ad is displayed to the server. The server analyzes the received reaction data, evaluates the effectiveness of the ad, and generates feedback to help select future ads.

[1039] Input: Captured reaction data

[1040] Data processing / calculation: Effect analysis and feedback generation

[1041] Output: Effect analysis results and feedback information

[1042] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1043] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1044] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1045] [Third embodiment]

[1046] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1047] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1048] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1049] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1050] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1051] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1052] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1053] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1054] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1055] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1056] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1057] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1058] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components, including emotion sensor means, analysis means, advertisement selection means, and advertisement delivery means. Below, we will explain the details of each component and how they work together.

[1059] 1. Collecting Emotional Data

[1060] User

[1061] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[1062] Terminal

[1063] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[1064] 2. Emotion Data Analysis

[1065] Terminal

[1066] The collected pulse data and facial expression data are analyzed by an AI module, which analyzes pulse fluctuations and facial expression characteristics to identify the user's current emotional state (e.g., relaxed, excited, happy, etc.).

[1067] 3. Sending Emotional Data

[1068] Terminal

[1069] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[1070] 4. Ad optimization

[1071] server

[1072] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[1073] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it will select ads related to relaxation, and if the user is excited, it will select ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[1074] 5. Delivery of advertisements

[1075] server

[1076] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[1077] Terminal

[1078] Based on the received advertising data, the advertisement is displayed to the user. The timing and format of the display depend on the content.

[1079] It monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, scrolling, etc.

[1080] 6. Feedback and algorithm improvements

[1081] Terminal

[1082] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze and click action.

[1083] server

[1084] The received reaction data is analyzed to evaluate the effectiveness of the advertisement.

[1085] The characteristics of ineffective ads are analyzed and reflected in future targeting, for example, determining that certain types of ads are inappropriate for certain emotional states.

[1086] Continually improve ad delivery algorithms based on new emotional and reaction data.

[1087] Specific examples

[1088] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for the relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is sent back to the server and used to select advertisements for future use.

[1089] In this way, an advertising delivery system based on a user's real-time emotional state can attract users' interest with greater accuracy than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[1090] The processing flow will be explained below.

[1091] Step 1:

[1092] The user puts on the wearable device and launches an application (e.g., a VR app) that collects emotion data.

[1093] Step 2:

[1094] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[1095] Step 3:

[1096] The device analyzes the pulse data received and detects fluctuations in the user's heart rate, as increases or decreases in heart rate can indicate changes in mood.

[1097] Step 4:

[1098] The AI ​​module analyzes facial images captured by the device and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.) from their facial features. Changes in facial expression are analyzed as movements of facial parts.

[1099] Step 5:

[1100] The device combines the analysis results of the pulse data and facial expression data to determine the user's final emotional state. For example, if the heart rate is stable and the user is smiling, the device will be classified as "Relaxed."

[1101] Step 6:

[1102] The device then anonymizes the analyzed emotional data and sends it to a server, including the emotional state, device ID, and timestamp.

[1103] Step 7:

[1104] The server stores the received emotion data in a database, and when storing it, it labels the data according to the user's emotional state.

[1105] Step 8:

[1106] The server analyzes the emotional data and compares it with an advertising database to select the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select an advertisement related to relaxation.

[1107] Step 9:

[1108] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[1109] Step 10:

[1110] The device displays advertisements to the user based on the received advertisement data. The timing of the display depends on the application status.

[1111] Step 11:

[1112] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[1113] Step 12:

[1114] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[1115] Step 13:

[1116] The server analyzes the reaction data received and evaluates the effectiveness of the ads, analyzes the characteristics of the ads that were less effective, and generates feedback to discontinue or modify them.

[1117] Step 14:

[1118] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[1119] Example 1

[1120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1121] Conventional ad delivery systems have had difficulty delivering optimal ads that reflect the user's emotional state in real time. Furthermore, there were insufficient methods for evaluating the effectiveness of ads and reflecting this in future targeting, making it difficult to simultaneously increase the satisfaction of both users and advertisers.

[1122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1123] In this invention, the server includes an advertisement selection means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the emotional state, a monitoring means for monitoring the user's reaction while the advertisement is being displayed, and an algorithm improvement means for evaluating the effectiveness of the advertisement based on the monitoring results and reflecting the results in subsequent targeting. This makes it possible to optimize advertisements that reflect the user's emotional state in real time and to continuously improve the advertisement distribution algorithm.

[1124] The "emotion sensor means" is a sensor for acquiring the user's emotional data, and specifically is a device including a wearable device for acquiring pulse data and a camera for capturing the user's facial expression.

[1125] The "analysis means" is a means for analyzing the emotional data acquired by the emotion sensor means and identifying the emotional state of the user. Specifically, it performs processing to analyze the emotional state using an AI module.

[1126] The "transmission means" is a means for anonymizing the analyzed emotion data and transmitting it to the server while protecting privacy.

[1127] The "advertisement selection means" is a means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the user's emotional state.

[1128] The "advertisement distribution means" is a means for distributing the selected advertisement to the user.

[1129] The "monitoring means" is a means for monitoring users' reactions while the advertisement is being displayed and collecting the data. Specifically, it is a means for recording gaze retention time, click actions, etc.

[1130] "Algorithm improvement measures" are measures for continuously improving the ad delivery algorithm by evaluating the effectiveness of advertising based on monitoring results and reflecting the evaluation results in future targeting.

[1131] This invention is a system that collects and analyzes real-time user emotional data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components: an emotion sensor means, an analysis means, a transmission means, an advertisement selection means, an advertisement delivery means, a monitoring means, and an algorithm improvement means. Below, we will explain in detail each component and how they work together.

[1132] Collecting Emotional Data

[1133] User

[1134] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[1135] Terminal

[1136] The device uses the in-camera to capture the user's facial expression data in real time, and simultaneously obtains pulse data from the wearable device via Bluetooth.

[1137] Emotional Data Analysis

[1138] Terminal

[1139] The device passes the acquired facial expression and pulse data to an AI module, which uses deep learning frameworks such as TensorFlow and PyTorch.

[1140] The device's AI module analyzes pulse fluctuations and facial features to identify the user's current emotional state, which can be relaxation, excitement, joy, etc.

[1141] Sending emotional data

[1142] Terminal

[1143] The device anonymizes the analyzed emotion data by removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[1144] The anonymized emotion data is sent to a server.

[1145] Ad optimization

[1146] server

[1147] The server stores the received emotion data in a database, which uses a relational database such as MySQL or PostgreSQL.

[1148] Based on the stored emotion data, the server matches it with an advertisement database: if the user is relaxed, it selects advertisements related to relaxation from the advertisement database.

[1149] The server takes into consideration the past ad display history and the user's action history to select the most suitable ad.

[1150] Ad serving

[1151] server

[1152] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[1153] Terminal

[1154] The device will then display the advertisement on the user's screen based on the received advertising data. The timing and format of the display will depend on the content. For example, the advertisements will be designed to appear naturally during the VR experience.

[1155] While the ad is being displayed, the device monitors the user's reactions, specifically recording actions such as gaze retention time, clicks on the ad, and scrolling.

[1156] Feedback and algorithm improvements

[1157] Terminal

[1158] After the ad is displayed, the user's reaction data, including gaze duration and click action, is sent to the server.

[1159] server

[1160] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It also analyzes the characteristics of ads that were less effective and reflects this in future targeting.

[1161] New emotional and reaction data will be used to continuously improve ad serving algorithms, for example by determining that certain types of ads are inappropriate for certain emotional states.

[1162] Specific examples

[1163] For example, if a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The device's AI module analyzes this data and determines that the user is relaxed. The device anonymizes this analysis result and sends it to a server. The server then selects advertisements appropriate for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions, such as gaze duration and clicks. This data is then sent back to the server and used to analyze the effectiveness of the advertisements and improve the algorithm.

[1164] The present invention enables optimal advertisement delivery based on the user's real-time emotional state, improving advertisement accuracy and user satisfaction.

[1165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1166] Step 1:

[1167] User provides emotion data

[1168] The user puts on the wearable device and launches the VR application. The front camera captures facial expressions. At the same time, the terminal acquires pulse data from the wearable device via Bluetooth.

[1169] Input: User's facial expression data, pulse data

[1170] Output: Raw data collected

[1171] Step 2:

[1172] The device analyzes the emotional data

[1173] The device passes the acquired facial expression and pulse data to an AI module, which then uses a deep learning framework (such as TensorFlow or PyTorch) to analyze the user's emotional state.

[1174] Input: facial expression data, pulse data

[1175] Data processing / calculation: Analysis and processing using facial expression data and pulse rate data

[1176] Output: User's emotional state (relaxed, excited, happy, etc.)

[1177] Step 3:

[1178] The device sends analysis data

[1179] The device anonymizes the analyzed emotion data and transmits it to a server in a privacy-preserving manner, which involves removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[1180] Input: Parsed emotion data

[1181] Data processing: Data anonymization

[1182] Output: Anonymized emotion data

[1183] Step 4:

[1184] The server receives and stores emotion data.

[1185] The server stores the received emotion data in a database using a relational database (e.g., MySQL or PostgreSQL).

[1186] Input: Anonymized emotion data

[1187] Data processing: data storage processing

[1188] Output: Data stored in the database

[1189] Step 5:

[1190] Server optimizes ads

[1191] The server matches the stored emotional data with the advertisement database. If the user is relaxed, it selects advertisements related to relaxation, taking into account the user's past ad viewing history and user action history.

[1192] Input: Stored emotion data

[1193] Data processing / calculation: Matching of emotion data with advertising database

[1194] Output: Optimized advertising data

[1195] Step 6:

[1196] The server sends the advertisement

[1197] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[1198] Input: Optimized Ad Data

[1199] Data processing: Generation of transmission data

[1200] Output: Data sent to the terminal

[1201] Step 7:

[1202] The device displays the ad and monitors the reaction

[1203] The device displays the advertisement on the user's screen based on the received advertisement data, and monitors the user's reactions (e.g., gaze retention time, clicks, scrolling, etc.) while the advertisement is displayed.

[1204] Input: Transmission data (advertising ID, display timing, content URL)

[1205] Data processing: Displaying advertisements, monitoring user reactions

[1206] Output: Reaction data (gaze duration, click actions, etc.)

[1207] Step 8:

[1208] The device sends reaction data

[1209] The terminal transmits reaction data after the advertisement is displayed to the server.

[1210] Input: Reaction data

[1211] Data processing: Preparing data for transmission

[1212] Output: Reaction data to the server

[1213] Step 9:

[1214] The server analyzes the feedback and improves the algorithm

[1215] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It reanalyzes the characteristics of ads that were ineffective and reflects them in future targeting. It also continuously improves the ad delivery algorithm based on new emotion and reaction data.

[1216] Input: Reaction data

[1217] Data processing / calculation: Analysis of reaction data, updating of algorithms

[1218] Output: Improved ad serving algorithm

[1219] (Application example 1)

[1220] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1221] Conventional ad delivery systems deliver ads without considering the user's emotional state, making it difficult to attract users' attention. They also lacked a means to understand the effectiveness of ads in real time and reflect this in future ad optimizations. Furthermore, there was no mechanism for continuously improving the ad delivery algorithm based on user reactions, making effective targeting impossible.

[1222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1223] In this invention, the server includes an emotion data analysis means, a transmission means for transmitting user reaction data after displaying an advertisement to the server and evaluating the effectiveness of the advertisement, and an algorithm improvement means for reselecting advertisements using an improved advertisement distribution algorithm, thereby enabling effective advertisement distribution based on the user's emotional state and continuous optimization of advertisements based on the reaction data.

[1224] An "emotion sensor means" is a device or apparatus used to acquire emotion data of a user.

[1225] The "analysis means" refers to a device or software that analyzes the acquired emotion data and identifies the user's emotional state.

[1226] An "advertising selector" is a device or software for selecting an optimized advertisement based on the identified emotional state.

[1227] The "monitoring means" refers to a device or software for monitoring and recording user reaction data to advertisements.

[1228] The "transmission means" refers to a device or software that transmits user reaction data after an advertisement is displayed to a server and evaluates the effectiveness of the advertisement.

[1229] "Algorithm improvement means" refers to devices or software that continuously improve ad delivery algorithms based on analyzed reaction data.

[1230] The "advertising distribution means" refers to a device or software for distributing selected advertisements to users.

[1231] A "wearable device" is a device that is worn on a user's body and is used to acquire biometric information such as pulse data.

[1232] A "camera" is a photographing device for capturing the user's facial expression.

[1233] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

[1234] 1. System Overview

[1235] This system collects and analyzes user emotional data and delivers optimal advertisements based on the results. The system consists of an emotion sensor, analysis means, advertisement selection means, monitoring means, transmission means, algorithm improvement means, and advertisement delivery means.

[1236] 2. Collecting Emotional Data

[1237] The user wears the wearable device and runs a specific application. The device captures the user's facial expressions using the front camera and receives pulse data via Bluetooth. These emotion data are collected in real time by the device.

[1238] 3. Emotion Data Analysis

[1239] The device analyzes the collected pulse data and facial expression data using an AI module. The software used includes an emotion recognition model using Keras and facial expression capture using OpenCV. The AI ​​module analyzes pulse fluctuations and facial expression characteristics to identify the user's emotional state (e.g., relaxed, excited, happy, etc.).

[1240] 4. Sending Emotional Data

[1241] The device sends anonymized analysis results to a server, including data such as device ID and timestamp to protect privacy.

[1242] 5. Ad optimization

[1243] The server stores the received emotional data in a database and matches it with the advertisement database to select an advertisement that best suits the emotional state. This process also takes into account the user's past ad viewing history and user action history.

[1244] 6. Delivery of advertisements

[1245] The server sends the optimized ad data to the device, which then displays the ad to the user and monitors the user's reaction during viewing. The software used includes a web view and an external browser for displaying the ad.

[1246] 7. Feedback and algorithm improvements

[1247] The device sends user reaction data (such as gaze duration, clicks, and scrolling) after the ad is displayed to the server. The server analyzes the received reaction data and evaluates the effectiveness of the ad. It analyzes the characteristics of ads that were less effective and reflects this in future targeting. It continuously improves the ad delivery algorithm based on new emotional and reaction data.

[1248] Adding specific examples

[1249] When a user launches a smartphone app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while viewing them. The reaction data is sent back to the server and used to select advertisements for future use.

[1250] Prompt Sentence Examples

[1251] "Generate an ad to show me when I'm relaxing. The dataset contains facial expression data and pulse rate data, and the information has been analyzed using a Keras model."

[1252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1253] Step 1:

[1254] The user puts on the wearable device and launches the smartphone application. The wearable device acquires pulse data and transmits it to the device via Bluetooth. The device's front camera also captures the user's facial expressions and obtains facial expression data. This data is collected in real time.

[1255] Input: User's pulse data, facial expression data

[1256] Output: Real-time pulse data, facial expression data

[1257] Specific operation: The user launches a smartphone application and acquires emotion data using the wearable device and the device's in-camera.

[1258] Step 2:

[1259] The device analyzes the acquired pulse data and facial expression data using an AI module. The pulse data is analyzed for fluctuation patterns, and the facial expression data is converted to grayscale, the facial area is cut out and resized, and then analyzed using an emotion recognition model. The AI ​​module identifies the user's emotional state (relaxed, excited, happy, etc.).

[1260] Input: Real-time pulse data, facial expression data

[1261] Output: Parsed emotional state data

[1262] Specific operation: The device's AI module analyzes pulse data and facial expression data to identify emotional states such as relaxation, excitement, and joy.

[1263] Step 3:

[1264] The device anonymizes the analyzed emotion data and sends it to the server, along with other data such as the device ID and timestamp. The emotion data is anonymized to protect privacy.

[1265] Input: Parsed emotional state data

[1266] Output: Anonymized emotion data sent to the server

[1267] Specific operation: The device anonymizes the analysis results and sends them to the server.

[1268] Step 4:

[1269] The server stores the received emotion data in a database and matches it with the advertisement database. Taking into account the user's past ad display history and action history, the server selects advertisements that best fit the user's emotional state.

[1270] Input: Anonymized emotion data sent to the server

[1271] Output: Optimized advertising data

[1272] Specific operation: The server stores the emotion data in a database and matches it with the advertisement database to select the most suitable advertisement.

[1273] Step 5:

[1274] The server then sends the optimized advertising data to the device, which includes the advertising ID, display timing, and URL of the advertising content.

[1275] Input: Optimized Ad Data

[1276] Output: Advertising data sent to the device

[1277] Specific operation: The server sends the selected advertising data to the terminal.

[1278] Step 6:

[1279] The device displays advertisements to the user based on the received advertising data. The timing and format of the advertisements depend on the content. The device also monitors the user's reactions (e.g., gaze retention time, clicks, scrolling) while the advertisements are displayed.

[1280] Input: Advertising data sent to the device

[1281] Output: Ads shown to users and their reaction data

[1282] Specific operation: The device displays advertisements and monitors the user's reactions.

[1283] Step 7:

[1284] The device sends user reaction data after the advertisement is displayed to the server, including gaze duration, click actions, etc.

[1285] Input: User reaction data

[1286] Output: Reaction data sent to the server

[1287] Specific operation: The device sends the user's reaction data to the server.

[1288] Step 8:

[1289] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It analyzes the characteristics of ineffective ads and reflects them in future targeting. It continuously improves the ad delivery algorithm based on new emotion and reaction data.

[1290] Input: Reaction data sent to the server

[1291] Output: Improved ad serving algorithm

[1292] What it does: The server analyzes the reaction data, evaluates the effectiveness of the ads, and improves the algorithm.

[1293] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1294] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes an emotion sensor means, an analysis means, an emotion engine, an advertisement selection means, and an advertisement delivery means. Below, we will explain the details of each component and how they work together.

[1295] 1. Collecting Emotional Data

[1296] User

[1297] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[1298] Terminal

[1299] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[1300] 2. Emotion Data Analysis

[1301] Terminal

[1302] The collected pulse data and facial expression data are sent to the emotion engine, which uses machine learning algorithms to analyze the data and identify the user's emotional state (e.g., joy, sadness, surprise, etc.), allowing the user's emotional state to be updated in real time.

[1303] Emotion Engine

[1304] Based on the received data, the user's emotional state is analyzed and the results are returned to the device. The emotion engine also accumulates past data and continues to learn, improving the accuracy of its analysis.

[1305] 3. Sending Emotional Data

[1306] Terminal

[1307] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[1308] 4. Ad optimization

[1309] server

[1310] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[1311] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it selects ads related to relaxation, and if the user is excited, it selects ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[1312] 5. Delivery of advertisements

[1313] server

[1314] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[1315] Terminal

[1316] Based on the received advertising data, advertisements are displayed to the user. The timing and format of the display depend on the content.

[1317] The system monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, and scrolling of the screen.

[1318] 6. Feedback and algorithm improvements

[1319] Terminal

[1320] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze, click actions, scroll position, etc.

[1321] server

[1322] Analyze the reaction data received to evaluate the effectiveness of the ads, analyze the characteristics of the ads that were less effective, and generate feedback to discontinue or modify them.

[1323] Improve and update ad delivery algorithms based on new emotional and reaction data, which continuously improves the accuracy and effectiveness of ad delivery.

[1324] Specific examples

[1325] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect that the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[1326] In this way, the use of an emotion engine makes it possible to realize an advertising delivery system based on the user's real-time emotional state, which can attract user interest with greater precision than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[1327] The processing flow will be explained below.

[1328] Step 1:

[1329] A user puts on a wearable device and launches an application (e.g., a VR app) that collects emotion data. The application captures facial expressions using the front camera.

[1330] Step 2:

[1331] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[1332] Step 3:

[1333] The device sends the received pulse data to the emotion engine, which analyzes the pulse fluctuations to detect changes in the user's heart rate. For example, an increase in heart rate indicates stress or excitement.

[1334] Step 4:

[1335] The device sends the captured facial image to the emotion engine, which analyzes the facial image, extracts facial features, and identifies the user's emotional state. For example, a smile indicates "joy."

[1336] Step 5:

[1337] The emotion engine integrates both the pulse data and facial expression data to determine the user's final emotional state (e.g., "relaxed," "excited," etc.), and the analysis results are returned to the device.

[1338] Step 6:

[1339] The device then anonymizes the analyzed emotional data, protecting privacy, and transmits it to a server, including the emotional state, device ID, and timestamp.

[1340] Step 7:

[1341] The server stores the received emotional data in a database, and labels it according to the emotional state.

[1342] Step 8:

[1343] The server analyzes the emotional data, compares it with an advertisement database, and selects the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select a relaxation-related advertisement.

[1344] Step 9:

[1345] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[1346] Step 10:

[1347] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the application's context.

[1348] Step 11:

[1349] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[1350] Step 12:

[1351] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[1352] Step 13:

[1353] The server analyzes the reaction data received and evaluates the effectiveness of the advertisement. The characteristics of advertisements that were less effective are analyzed and reflected in future advertisement selection.

[1354] Step 14:

[1355] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[1356] Example 2

[1357] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1358] Conventional ad delivery systems were unable to accurately reflect the user's real-time emotional state, making it difficult to deliver appropriate ads. Furthermore, they lacked a mechanism for evaluating the effectiveness of ads and providing feedback for future ad delivery, making it difficult to adequately optimize ads. As a result, ads failed to attract user interest and were less effective.

[1359] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1360] In this invention, the server includes sensor means for acquiring user emotional data, analysis means for analyzing the emotional data to identify the user's emotional state, transmission means for anonymizing the identified emotional state and transmitting it to the server, advertisement selection means for selecting an optimized advertisement based on the emotional state and past behavioral history, advertisement distribution means for delivering the selected advertisement to the user, means for monitoring user reaction data during advertisement delivery, and feedback means for transmitting the monitored data to the server and improving the advertisement delivery algorithm. This enables appropriate advertisement delivery based on the user's real-time emotional state, and enables continuous feedback and optimization to improve advertisement effectiveness.

[1361] The term "sensor means" refers to a device for acquiring emotional data of a user.

[1362] The term "analysis means" refers to a device that has the function of analyzing acquired emotion data and identifying the user's emotional state.

[1363] "Transmitting means" refers to a device that has the function of anonymizing the identified emotional state and transmitting it to a server.

[1364] "Advertisement selection means" refers to a device that has the function of selecting an optimized advertisement based on emotional state and past behavioral history.

[1365] "Advertisement distribution means" refers to a device that has the function of distributing selected advertisements to users.

[1366] The term "monitoring means" refers to a device that has the function of monitoring user reaction data during advertisement distribution.

[1367] "Feedback means" refers to a device that has the function of transmitting monitored data to a server and generating feedback to improve the ad delivery algorithm.

[1368] "Emotion data" refers to data that indicates the user's emotional state, and includes facial expressions, pulse rate, and other biological information.

[1369] "Emotional State" refers to the user's current psychological and physiological state as identified by analytical means.

[1370] "Reaction data" refers to data regarding the reactions and actions of users during advertisement delivery.

[1371] "Anonymization" refers to the act of processing data so that it cannot be used to identify individuals.

[1372] "Server" refers to a computer system that receives emotion data and reaction data via a network and performs analysis and advertisement selection.

[1373] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes emotion sensor means, analysis means, transmission means, advertisement selection means, advertisement delivery means, monitoring means, and feedback means. Below, we will provide details of each component and how they work together, along with specific examples.

[1374] Collecting Emotional Data

[1375] User

[1376] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[1377] Terminal

[1378] The device collects pulse and facial expression data in real time from the front camera and the wearable device. Pulse data is transmitted to the device via Bluetooth, and facial expression data is captured by the front camera.

[1379] Emotional Data Analysis

[1380] Terminal

[1381] The device temporarily stores the collected pulse data and facial expression data locally and sends it to the emotion engine.

[1382] Emotion Engine

[1383] The emotion engine uses machine learning algorithms (e.g., neural networks) to analyze the data and identify the user's emotional state, which may include happiness, sadness, surprise, relaxation, etc., and returns the analysis results to the device.

[1384] Sending emotional data

[1385] Terminal

[1386] After receiving the analysis results from the emotion engine, the device anonymizes the emotion data and sends it to the server along with the device ID and timestamp to protect privacy.

[1387] Ad optimization

[1388] server

[1389] The server stores the received emotion data in a database and runs an algorithm to select the most appropriate ad based on the emotion data, past ad display history, and user action history.

[1390] Ad serving

[1391] server

[1392] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[1393] Terminal

[1394] The device displays advertisements to the user based on the received advertising data. The timing and format of the display depend on the content. The device also monitors the user's reactions while the advertisement is being displayed (for example, how long the user's gaze remains on the advertisement, whether or not the user clicks, whether the user scrolls the screen, etc.).

[1395] Feedback and algorithm improvements

[1396] Terminal

[1397] After the ad is displayed, the device sends the user's reaction data to the server, including gaze duration, click actions, scroll position, etc.

[1398] server

[1399] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback to discontinue or modify them, and updates and improves the ad delivery algorithm based on the new emotion and reaction data.

[1400] Implementing specific examples

[1401] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect when the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[1402] Prompt Sentence Examples

[1403] An example prompt for a generative AI model might look something like this:

[1404] "Select the best ad based on the following emotional state data:

[1405] Emotional state: Relaxed

[1406] Device ID: 12345XYZ

[1407] Timestamp: 2023-10-09 10:00:00

[1408] Advertising history: Hot spring trips, yoga classes

[1409] When this prompt sentence is input into a generative AI model, the optimal advertisement is selected based on the emotional data.

[1410] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1411] Step 1: Collecting emotion data

[1412] User

[1413] The user puts on the wearable device and launches the VR application, which starts capturing the user's facial expressions through the front camera.

[1414] Terminal

[1415] The terminal receives pulse data from the wearable device via Bluetooth, and simultaneously captures the user's facial expression data in real time using the front camera.

[1416] input

[1417] Pulse data and facial expression data

[1418] output

[1419] Raw data temporarily stored in local data storage

[1420] Step 2: Analyze the emotion data

[1421] Terminal

[1422] The device sends the stored pulse data and facial expression data to the emotion engine.

[1423] Emotion Engine

[1424] The emotion engine uses machine learning algorithms to analyze data and identify the user's emotional state. For example, it uses a neural network to analyze facial expression data and pulse rate data to identify an emotional state such as "relaxed."

[1425] input

[1426] Collected pulse data and facial expression data

[1427] Data Processing

[1428] Data analysis using machine learning algorithms

[1429] output

[1430] The user's emotional state (e.g., "Relaxed")

[1431] Step 3: Sending emotion data

[1432] Terminal

[1433] The device anonymizes the emotional state obtained from the emotion engine, and sends it to the server with a device ID and a timestamp.

[1434] input

[1435] Analyzed emotional state, device ID, timestamp

[1436] Data Processing

[1437] Data anonymization

[1438] output

[1439] Anonymized emotion data (e.g., "Relaxed, Device ID: 12345XYZ, Timestamp: 2023-10-09 10:00:00")

[1440] Step 4: Optimize your ads

[1441] server

[1442] The server stores the received emotional data in a database, and then selects the most appropriate ad based on the emotional data, past ad display history, and user action history.

[1443] input

[1444] Anonymized emotional data, past ad viewing history, and user action history

[1445] Data Calculation

[1446] Implementing ad selection algorithms using generative AI models

[1447] output

[1448] Optimized advertising data (e.g., "hot spring trip advertisement")

[1449] Step 5: Serving Ads

[1450] server

[1451] The server then sends the selected advertising data to the device, including the advertising ID, display timing, and the URL of the advertising content.

[1452] Terminal

[1453] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the content.

[1454] input

[1455] Optimized Advertising Data

[1456] output

[1457] Ad content displayed to users

[1458] Step 6: Monitor while your ad is showing

[1459] Terminal

[1460] While the ad is displayed, the device monitors the user's reactions (such as gaze retention, ad clicks, and scrolling).

[1461] input

[1462] User gaze data, click data, scroll data

[1463] output

[1464] User reaction data stored as log files

[1465] Step 7: Feedback and algorithm refinement

[1466] Terminal

[1467] After the advertisement is displayed, the terminal transmits the user's reaction data to the server.

[1468] server

[1469] The server analyzes the received reaction data and evaluates the effectiveness of the advertisement. It analyzes the characteristics of ineffective advertisements and generates feedback for future advertisements. It also updates and improves the ad delivery algorithm based on the new emotional and reaction data.

[1470] input

[1471] User reaction data

[1472] Data Calculation

[1473] Evaluating advertising effectiveness using analytical algorithms and creating new advertising selection algorithms

[1474] output

[1475] Updated ad serving algorithm

[1476] Through the above processing steps, this system is able to deliver highly accurate advertisements and provide continuous feedback on their effectiveness based on the user's real-time emotional state.

[1477] (Application example 2)

[1478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1479] Conventional ad delivery systems have had the challenge of being difficult to target accurately based on user interests and states. In particular, ad delivery utilizing emotional data has been rare, making it impossible to provide ads that match the user's emotional state in real time. This has led to problems such as a decline in user engagement and difficulty for advertisers in measuring effectiveness.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1481] In this invention, the server includes a transmission means for anonymizing a user's emotional data and transmitting it to the server while protecting their privacy, a feedback means for improving an ad delivery algorithm based on the emotional data and a history of past ad display, and a means for acquiring pulse data and facial expression data in real time from the emotional data, the in-camera, and the wearable device, thereby enabling highly targeted ad delivery based on the user's real-time emotional state.

[1482] The "sensor means" is a device for acquiring emotional data of the user.

[1483] The "analysis means" is a device that analyzes the acquired emotional data and identifies the emotional state of the user.

[1484] The "selection means" is a device that selects the most suitable advertisement based on the analyzed emotional state.

[1485] The "distribution means" is a device that distributes the selected advertisement to the user.

[1486] The "transmission means" is a device that anonymizes the user's emotional data and transmits it to the server while protecting privacy.

[1487] A "feedback means" is a device that improves the ad delivery algorithm based on emotional data and past ad display history.

[1488] A "wearable device" is a device worn by a user to acquire pulse data.

[1489] The "imaging device" is a camera device for capturing the user's facial expression.

[1490] The "emotional state" is the mental state of the user identified as a result of analyzing the emotion data.

[1491] The "advertising delivery algorithm" is an algorithm for controlling the display of advertisements based on emotional data and past advertisement display history.

[1492] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of the following components: sensor means, analysis means, selection means, delivery means, transmission means, and feedback means.

[1493] 1. Collecting Emotional Data

[1494] A user wears a wearable device and runs a specific application (e.g., an app for a smartphone or smart glasses). This application captures the user's facial expressions using an in-camera or an imaging device. Pulse data is also acquired from the wearable device in real time. This data is then transmitted to a terminal via Bluetooth.

[1495] 2. Emotion Data Analysis

[1496] The device sends the collected pulse data and facial expression data to the analysis means, which incorporates a machine learning algorithm that analyzes the emotional data to identify the user's emotional state (e.g., joy, sadness, surprise, etc.). The analysis results are updated in real time, and the emotional state is returned to the device.

[1497] 3. Sending Emotional Data

[1498] The device then anonymizes the analyzed emotional data and transmits it to a server, protecting privacy. The transmitted data includes the emotional state, device ID, and timestamp.

[1499] 4. Ad optimization

[1500] The server stores the received emotional data in a database. Based on the stored emotional data, it matches it with the advertisement database and selects advertisements that suit the user's emotional state. For example, if the user is relaxed, it will select relaxation-related advertisements, and if the user is excited, it will select exciting advertisements. It also takes into account the user's past ad viewing history and user action history.

[1501] 5. Delivery of advertisements

[1502] The server sends the optimized advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content. The device displays the advertisement to the user based on the received advertising data. While the advertisement is being displayed, the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) are monitored.

[1503] 6. Feedback and algorithm improvements

[1504] The device sends data on the user's reaction after the advertisement is displayed to the server, including the duration of gaze, click actions, scroll position, etc.

[1505] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback for corrections, and improves and updates the ad delivery algorithm based on new emotion and reaction data to continuously improve the accuracy and effectiveness of ad delivery.

[1506] Specific examples

[1507] When a user turns on their smart glasses to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an analysis means, which uses a machine learning algorithm to detect relaxation. The results are returned to the device, which then analyzes the data and sends the analyzed emotional data to the server. The server selects an advertisement suitable for the relaxation state (e.g., a hot spring trip or relaxation service) and sends it to the device. The device displays the advertisement to the user and monitors the user's reaction while viewing it. The reaction data is then sent back to the server and used to select future advertisements. In this way, using an emotion engine enables an advertising delivery system based on the user's real-time emotional state.

[1508] Prompt Sentence Examples

[1509] Collect real-time emotional data while the user is wearing the smart glasses, and if the user's current emotional state is analyzed as "relaxed," select and display appropriate relaxation-related advertisements, such as advertisements for hot spring trips or massage services.

[1510] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1511] Step 1:

[1512] The device uses a camera and a wearable device to collect user emotional data. Specifically, it captures the user's facial expressions using the in-camera and obtains pulse data from the wearable device via Bluetooth.

[1513] Input: User's facial expression data (image) and pulse data

[1514] Output: Captured facial expression data and acquired pulse data

[1515] Step 2:

[1516] The terminal transmits the collected facial expression data and pulse rate data to the analysis means, which uses a machine learning algorithm to analyze the emotion data and identify the user's emotional state.

[1517] Input: Captured facial expression data and acquired pulse data

[1518] Data processing / computation: Identifying emotional states using machine learning algorithms

[1519] Output: Identified user emotional state (happy, sad, surprised, etc.)

[1520] Step 3:

[1521] The device anonymizes the analyzed emotion data and transmits it to a server while protecting privacy.

[1522] Input: Identified user emotional state

[1523] Data processing / calculation: Data anonymization

[1524] Output: Anonymized emotional data (emotional state, device ID, timestamp)

[1525] Step 4:

[1526] The server stores the received emotional data in a database, matches the stored emotional data with the advertisement database, and selects advertisements that match the emotional state.

[1527] Input: Anonymized emotion data

[1528] Data processing / calculation: database search and matching processing

[1529] Output: Selected ad data (ad ID, display timing, ad content URL, etc.)

[1530] Step 5:

[1531] The server transmits the selected advertisement data to the terminal, and the terminal displays an advertisement to the user based on the received advertisement data.

[1532] Input: Selected advertising data

[1533] Data processing / calculation: Sending advertising data and displaying advertisements

[1534] Output: Ad content shown to the user

[1535] Step 6:

[1536] The device monitors the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) while the advertisement is displayed.

[1537] Input: User reaction data

[1538] Data processing / calculation: Reaction data capture and recording

[1539] Output: Captured reaction data

[1540] Step 7:

[1541] The device sends user reaction data after the ad is displayed to the server. The server analyzes the received reaction data, evaluates the effectiveness of the ad, and generates feedback to help select future ads.

[1542] Input: Captured reaction data

[1543] Data processing / calculation: Effect analysis and feedback generation

[1544] Output: Effect analysis results and feedback information

[1545] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1546] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1547] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1548] [Fourth embodiment]

[1549] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1550] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1551] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1552] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1553] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1554] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1555] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1556] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1557] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1558] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1559] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1560] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1561] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1562] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components, including emotion sensor means, analysis means, advertisement selection means, and advertisement delivery means. Below, we will explain the details of each component and how they work together.

[1563] 1. Collecting Emotional Data

[1564] User

[1565] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[1566] Terminal

[1567] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[1568] 2. Emotion Data Analysis

[1569] Terminal

[1570] The collected pulse data and facial expression data are analyzed by an AI module, which analyzes pulse fluctuations and facial expression characteristics to identify the user's current emotional state (e.g., relaxed, excited, happy, etc.).

[1571] 3. Sending Emotional Data

[1572] Terminal

[1573] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[1574] 4. Ad optimization

[1575] server

[1576] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[1577] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it will select ads related to relaxation, and if the user is excited, it will select ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[1578] 5. Delivery of advertisements

[1579] server

[1580] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[1581] Terminal

[1582] Based on the received advertising data, the advertisement is displayed to the user. The timing and format of the display depend on the content.

[1583] It monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, scrolling, etc.

[1584] 6. Feedback and algorithm improvements

[1585] Terminal

[1586] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze and click action.

[1587] server

[1588] The received reaction data is analyzed to evaluate the effectiveness of the advertisement.

[1589] The characteristics of ineffective ads are analyzed and reflected in future targeting, for example, determining that certain types of ads are inappropriate for certain emotional states.

[1590] Continually improve ad delivery algorithms based on new emotional and reaction data.

[1591] Specific examples

[1592] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for the relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is sent back to the server and used to select advertisements for future use.

[1593] In this way, an advertising delivery system based on a user's real-time emotional state can attract users' interest with greater accuracy than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[1594] The processing flow will be explained below.

[1595] Step 1:

[1596] The user puts on the wearable device and launches an application (e.g., a VR app) that collects emotion data.

[1597] Step 2:

[1598] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[1599] Step 3:

[1600] The device analyzes the pulse data received and detects fluctuations in the user's heart rate, as increases or decreases in heart rate can indicate changes in mood.

[1601] Step 4:

[1602] The AI ​​module analyzes facial images captured by the device and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.) from their facial features. Changes in facial expression are analyzed as movements of facial parts.

[1603] Step 5:

[1604] The device combines the analysis results of the pulse data and facial expression data to determine the user's final emotional state. For example, if the heart rate is stable and the user is smiling, the device will be classified as "Relaxed."

[1605] Step 6:

[1606] The device then anonymizes the analyzed emotional data and sends it to a server, including the emotional state, device ID, and timestamp.

[1607] Step 7:

[1608] The server stores the received emotion data in a database, and when storing it, it labels the data according to the user's emotional state.

[1609] Step 8:

[1610] The server analyzes the emotional data and compares it with an advertising database to select the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select an advertisement related to relaxation.

[1611] Step 9:

[1612] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[1613] Step 10:

[1614] The device displays advertisements to the user based on the received advertisement data. The timing of the display depends on the application status.

[1615] Step 11:

[1616] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[1617] Step 12:

[1618] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[1619] Step 13:

[1620] The server analyzes the reaction data received and evaluates the effectiveness of the ads, analyzes the characteristics of the ads that were less effective, and generates feedback to discontinue or modify them.

[1621] Step 14:

[1622] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[1623] Example 1

[1624] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1625] Conventional ad delivery systems have had difficulty delivering optimal ads that reflect the user's emotional state in real time. Furthermore, there were insufficient methods for evaluating the effectiveness of ads and reflecting this in future targeting, making it difficult to simultaneously increase the satisfaction of both users and advertisers.

[1626] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1627] In this invention, the server includes an advertisement selection means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the emotional state, a monitoring means for monitoring the user's reaction while the advertisement is being displayed, and an algorithm improvement means for evaluating the effectiveness of the advertisement based on the monitoring results and reflecting the results in subsequent targeting. This makes it possible to optimize advertisements that reflect the user's emotional state in real time and to continuously improve the advertisement distribution algorithm.

[1628] The "emotion sensor means" is a sensor for acquiring the user's emotional data, and specifically is a device including a wearable device for acquiring pulse data and a camera for capturing the user's facial expression.

[1629] The "analysis means" is a means for analyzing the emotional data acquired by the emotion sensor means and identifying the emotional state of the user. Specifically, it performs processing to analyze the emotional state using an AI module.

[1630] The "transmission means" is a means for anonymizing the analyzed emotion data and transmitting it to the server while protecting privacy.

[1631] The "advertisement selection means" is a means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the user's emotional state.

[1632] The "advertisement distribution means" is a means for distributing the selected advertisement to the user.

[1633] The "monitoring means" is a means for monitoring users' reactions while the advertisement is being displayed and collecting the data. Specifically, it is a means for recording gaze retention time, click actions, etc.

[1634] "Algorithm improvement measures" are measures for continuously improving the ad delivery algorithm by evaluating the effectiveness of advertising based on monitoring results and reflecting the evaluation results in future targeting.

[1635] This invention is a system that collects and analyzes real-time user emotional data and delivers optimal advertisements based on the analysis results. This system is composed of multiple components: an emotion sensor means, an analysis means, a transmission means, an advertisement selection means, an advertisement delivery means, a monitoring means, and an algorithm improvement means. Below, we will explain in detail each component and how they work together.

[1636] Collecting Emotional Data

[1637] User

[1638] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[1639] Terminal

[1640] The device uses the in-camera to capture the user's facial expression data in real time, and simultaneously obtains pulse data from the wearable device via Bluetooth.

[1641] Emotional Data Analysis

[1642] Terminal

[1643] The device passes the acquired facial expression and pulse data to an AI module, which uses deep learning frameworks such as TensorFlow and PyTorch.

[1644] The device's AI module analyzes pulse fluctuations and facial features to identify the user's current emotional state, which can be relaxation, excitement, joy, etc.

[1645] Sending emotional data

[1646] Terminal

[1647] The device anonymizes the analyzed emotion data by removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[1648] The anonymized emotion data is sent to a server.

[1649] Ad optimization

[1650] server

[1651] The server stores the received emotion data in a database, which uses a relational database such as MySQL or PostgreSQL.

[1652] Based on the stored emotion data, the server matches it with an advertisement database: if the user is relaxed, it selects advertisements related to relaxation from the advertisement database.

[1653] The server takes into consideration the past ad display history and the user's action history to select the most suitable ad.

[1654] Ad serving

[1655] server

[1656] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[1657] Terminal

[1658] The device will then display the advertisement on the user's screen based on the received advertising data. The timing and format of the display will depend on the content. For example, the advertisements will be designed to appear naturally during the VR experience.

[1659] While the ad is being displayed, the device monitors the user's reactions, specifically recording actions such as gaze retention time, clicks on the ad, and scrolling.

[1660] Feedback and algorithm improvements

[1661] Terminal

[1662] After the ad is displayed, the user's reaction data, including gaze duration and click action, is sent to the server.

[1663] server

[1664] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It also analyzes the characteristics of ads that were less effective and reflects this in future targeting.

[1665] New emotional and reaction data will be used to continuously improve ad serving algorithms, for example by determining that certain types of ads are inappropriate for certain emotional states.

[1666] Specific examples

[1667] For example, if a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The device's AI module analyzes this data and determines that the user is relaxed. The device anonymizes this analysis result and sends it to a server. The server then selects advertisements appropriate for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions, such as gaze duration and clicks. This data is then sent back to the server and used to analyze the effectiveness of the advertisements and improve the algorithm.

[1668] The present invention enables optimal advertisement delivery based on the user's real-time emotional state, improving advertisement accuracy and user satisfaction.

[1669] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1670] Step 1:

[1671] User provides emotion data

[1672] The user puts on the wearable device and launches the VR application. The front camera captures facial expressions. At the same time, the terminal acquires pulse data from the wearable device via Bluetooth.

[1673] Input: User's facial expression data, pulse data

[1674] Output: Raw data collected

[1675] Step 2:

[1676] The device analyzes the emotional data

[1677] The device passes the acquired facial expression and pulse data to an AI module, which then uses a deep learning framework (such as TensorFlow or PyTorch) to analyze the user's emotional state.

[1678] Input: facial expression data, pulse data

[1679] Data processing / calculation: Analysis and processing using facial expression data and pulse rate data

[1680] Output: User's emotional state (relaxed, excited, happy, etc.)

[1681] Step 3:

[1682] The device sends analysis data

[1683] The device anonymizes the analyzed emotion data and transmits it to a server in a privacy-preserving manner, which involves removing the user's personal information and converting it into a format that contains only the device ID and a timestamp.

[1684] Input: Parsed emotion data

[1685] Data processing: Data anonymization

[1686] Output: Anonymized emotion data

[1687] Step 4:

[1688] The server receives and stores emotion data.

[1689] The server stores the received emotion data in a database using a relational database (e.g., MySQL or PostgreSQL).

[1690] Input: Anonymized emotion data

[1691] Data processing: data storage processing

[1692] Output: Data stored in the database

[1693] Step 5:

[1694] Server optimizes ads

[1695] The server matches the stored emotional data with the advertisement database. If the user is relaxed, it selects advertisements related to relaxation, taking into account the user's past ad viewing history and user action history.

[1696] Input: Stored emotion data

[1697] Data processing / calculation: Matching of emotion data with advertising database

[1698] Output: Optimized advertising data

[1699] Step 6:

[1700] The server sends the advertisement

[1701] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[1702] Input: Optimized Ad Data

[1703] Data processing: Generation of transmission data

[1704] Output: Data sent to the terminal

[1705] Step 7:

[1706] The device displays the ad and monitors the reaction

[1707] The device displays the advertisement on the user's screen based on the received advertisement data, and monitors the user's reactions (e.g., gaze retention time, clicks, scrolling, etc.) while the advertisement is displayed.

[1708] Input: Transmission data (advertising ID, display timing, content URL)

[1709] Data processing: Displaying advertisements, monitoring user reactions

[1710] Output: Reaction data (gaze duration, click actions, etc.)

[1711] Step 8:

[1712] The device sends reaction data

[1713] The terminal transmits reaction data after the advertisement is displayed to the server.

[1714] Input: Reaction data

[1715] Data processing: Preparing data for transmission

[1716] Output: Reaction data to the server

[1717] Step 9:

[1718] The server analyzes the feedback and improves the algorithm

[1719] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It reanalyzes the characteristics of ads that were ineffective and reflects them in future targeting. It also continuously improves the ad delivery algorithm based on new emotion and reaction data.

[1720] Input: Reaction data

[1721] Data processing / calculation: Analysis of reaction data, updating of algorithms

[1722] Output: Improved ad serving algorithm

[1723] (Application example 1)

[1724] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1725] Conventional ad delivery systems deliver ads without considering the user's emotional state, making it difficult to attract users' attention. They also lacked a means to understand the effectiveness of ads in real time and reflect this in future ad optimizations. Furthermore, there was no mechanism for continuously improving the ad delivery algorithm based on user reactions, making effective targeting impossible.

[1726] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1727] In this invention, the server includes an emotion data analysis means, a transmission means for transmitting user reaction data after displaying an advertisement to the server and evaluating the effectiveness of the advertisement, and an algorithm improvement means for reselecting advertisements using an improved advertisement distribution algorithm, thereby enabling effective advertisement distribution based on the user's emotional state and continuous optimization of advertisements based on the reaction data.

[1728] An "emotion sensor means" is a device or apparatus used to acquire emotion data of a user.

[1729] The "analysis means" refers to a device or software that analyzes the acquired emotion data and identifies the user's emotional state.

[1730] An "advertising selector" is a device or software for selecting an optimized advertisement based on the identified emotional state.

[1731] The "monitoring means" refers to a device or software for monitoring and recording user reaction data to advertisements.

[1732] The "transmission means" refers to a device or software that transmits user reaction data after an advertisement is displayed to a server and evaluates the effectiveness of the advertisement.

[1733] "Algorithm improvement means" refers to devices or software that continuously improve ad delivery algorithms based on analyzed reaction data.

[1734] The "advertising distribution means" refers to a device or software for distributing selected advertisements to users.

[1735] A "wearable device" is a device that is worn on a user's body and is used to acquire biometric information such as pulse data.

[1736] A "camera" is a photographing device for capturing the user's facial expression.

[1737] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

[1738] 1. System Overview

[1739] This system collects and analyzes user emotional data and delivers optimal advertisements based on the results. The system consists of an emotion sensor, analysis means, advertisement selection means, monitoring means, transmission means, algorithm improvement means, and advertisement delivery means.

[1740] 2. Collecting Emotional Data

[1741] The user wears the wearable device and runs a specific application. The device captures the user's facial expressions using the front camera and receives pulse data via Bluetooth. These emotion data are collected in real time by the device.

[1742] 3. Emotion Data Analysis

[1743] The device analyzes the collected pulse data and facial expression data using an AI module. The software used includes an emotion recognition model using Keras and facial expression capture using OpenCV. The AI ​​module analyzes pulse fluctuations and facial expression characteristics to identify the user's emotional state (e.g., relaxed, excited, happy, etc.).

[1744] 4. Sending Emotional Data

[1745] The device sends anonymized analysis results to a server, including data such as device ID and timestamp to protect privacy.

[1746] 5. Ad optimization

[1747] The server stores the received emotional data in a database and matches it with the advertisement database to select an advertisement that best suits the emotional state. This process also takes into account the user's past ad viewing history and user action history.

[1748] 6. Delivery of advertisements

[1749] The server sends the optimized ad data to the device, which then displays the ad to the user and monitors the user's reaction during viewing. The software used includes a web view and an external browser for displaying the ad.

[1750] 7. Feedback and algorithm improvements

[1751] The device sends user reaction data (such as gaze duration, clicks, and scrolling) after the ad is displayed to the server. The server analyzes the received reaction data and evaluates the effectiveness of the ad. It analyzes the characteristics of ads that were less effective and reflects this in future targeting. It continuously improves the ad delivery algorithm based on new emotional and reaction data.

[1752] Adding specific examples

[1753] When a user launches a smartphone app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. The analysis means determines that the user is relaxed and sends this data to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while viewing them. The reaction data is sent back to the server and used to select advertisements for future use.

[1754] Prompt Sentence Examples

[1755] "Generate an ad to show me when I'm relaxing. The dataset contains facial expression data and pulse rate data, and the information has been analyzed using a Keras model."

[1756] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1757] Step 1:

[1758] The user puts on the wearable device and launches the smartphone application. The wearable device acquires pulse data and transmits it to the device via Bluetooth. The device's front camera also captures the user's facial expressions and obtains facial expression data. This data is collected in real time.

[1759] Input: User's pulse data, facial expression data

[1760] Output: Real-time pulse data, facial expression data

[1761] Specific operation: The user launches a smartphone application and acquires emotion data using the wearable device and the device's in-camera.

[1762] Step 2:

[1763] The device analyzes the acquired pulse data and facial expression data using an AI module. The pulse data is analyzed for fluctuation patterns, and the facial expression data is converted to grayscale, the facial area is cut out and resized, and then analyzed using an emotion recognition model. The AI ​​module identifies the user's emotional state (relaxed, excited, happy, etc.).

[1764] Input: Real-time pulse data, facial expression data

[1765] Output: Parsed emotional state data

[1766] Specific operation: The device's AI module analyzes pulse data and facial expression data to identify emotional states such as relaxation, excitement, and joy.

[1767] Step 3:

[1768] The device anonymizes the analyzed emotion data and sends it to the server, along with other data such as the device ID and timestamp. The emotion data is anonymized to protect privacy.

[1769] Input: Parsed emotional state data

[1770] Output: Anonymized emotion data sent to the server

[1771] Specific operation: The device anonymizes the analysis results and sends them to the server.

[1772] Step 4:

[1773] The server stores the received emotion data in a database and matches it with the advertisement database. Taking into account the user's past ad display history and action history, the server selects advertisements that best fit the user's emotional state.

[1774] Input: Anonymized emotion data sent to the server

[1775] Output: Optimized advertising data

[1776] Specific operation: The server stores the emotion data in a database and matches it with the advertisement database to select the most suitable advertisement.

[1777] Step 5:

[1778] The server then sends the optimized advertising data to the device, which includes the advertising ID, display timing, and URL of the advertising content.

[1779] Input: Optimized Ad Data

[1780] Output: Advertising data sent to the device

[1781] Specific operation: The server sends the selected advertising data to the terminal.

[1782] Step 6:

[1783] The device displays advertisements to the user based on the received advertising data. The timing and format of the advertisements depend on the content. The device also monitors the user's reactions (e.g., gaze retention time, clicks, scrolling) while the advertisements are displayed.

[1784] Input: Advertising data sent to the device

[1785] Output: Ads shown to users and their reaction data

[1786] Specific operation: The device displays advertisements and monitors the user's reactions.

[1787] Step 7:

[1788] The device sends user reaction data after the advertisement is displayed to the server, including gaze duration, click actions, etc.

[1789] Input: User reaction data

[1790] Output: Reaction data sent to the server

[1791] Specific operation: The device sends the user's reaction data to the server.

[1792] Step 8:

[1793] The server analyzes the received reaction data and evaluates the effectiveness of the ads. It analyzes the characteristics of ineffective ads and reflects them in future targeting. It continuously improves the ad delivery algorithm based on new emotion and reaction data.

[1794] Input: Reaction data sent to the server

[1795] Output: Improved ad serving algorithm

[1796] What it does: The server analyzes the reaction data, evaluates the effectiveness of the ads, and improves the algorithm.

[1797] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1798] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes an emotion sensor means, an analysis means, an emotion engine, an advertisement selection means, and an advertisement delivery means. Below, we will explain the details of each component and how they work together.

[1799] 1. Collecting Emotional Data

[1800] User

[1801] A wearable device is worn and a specific application (e.g., a VR application) is executed, which captures facial expressions using the front camera.

[1802] Terminal

[1803] Pulse data and facial expression data are collected in real time from the front camera and the wearable device. Pulse data is sent to the device via Bluetooth, and facial expression data is captured by the front camera.

[1804] 2. Emotion Data Analysis

[1805] Terminal

[1806] The collected pulse data and facial expression data are sent to the emotion engine, which uses machine learning algorithms to analyze the data and identify the user's emotional state (e.g., joy, sadness, surprise, etc.), allowing the user's emotional state to be updated in real time.

[1807] Emotion Engine

[1808] Based on the received data, the user's emotional state is analyzed and the results are returned to the device. The emotion engine also accumulates past data and continues to learn, improving the accuracy of its analysis.

[1809] 3. Sending Emotional Data

[1810] Terminal

[1811] The analyzed emotional data is anonymized and sent to a server with privacy protection, including the emotional state, device ID, and timestamp.

[1812] 4. Ad optimization

[1813] server

[1814] The received emotional data is stored in a database, and the stored emotional data is matched with the advertising database.

[1815] The system selects ads that match the user's emotional state. For example, if the user is relaxed, it selects ads related to relaxation, and if the user is excited, it selects ads that are exciting. It also takes into account the user's past ad viewing history and user action history.

[1816] 5. Delivery of advertisements

[1817] server

[1818] Optimized advertising data is sent to the device, including the advertising ID, display timing, and URL of the advertising content.

[1819] Terminal

[1820] Based on the received advertising data, advertisements are displayed to the user. The timing and format of the display depend on the content.

[1821] The system monitors user reactions while the ad is displayed, specifically recording gaze duration, clicks, and scrolling of the screen.

[1822] 6. Feedback and algorithm improvements

[1823] Terminal

[1824] After the ad is displayed, the user's reaction data is sent to the server, including the duration of gaze, click actions, scroll position, etc.

[1825] server

[1826] Analyze the reaction data received to evaluate the effectiveness of the ads, analyze the characteristics of the ads that were less effective, and generate feedback to discontinue or modify them.

[1827] Improve and update ad delivery algorithms based on new emotional and reaction data, which continuously improves the accuracy and effectiveness of ad delivery.

[1828] Specific examples

[1829] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect that the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[1830] In this way, the use of an emotion engine makes it possible to realize an advertising delivery system based on the user's real-time emotional state, which can attract user interest with greater precision than conventional targeting methods, resulting in more satisfying results for both users and advertisers.

[1831] The processing flow will be explained below.

[1832] Step 1:

[1833] A user puts on a wearable device and launches an application (e.g., a VR app) that collects emotion data. The application captures facial expressions using the front camera.

[1834] Step 2:

[1835] The device begins receiving real-time pulse data from the wearable device via Bluetooth, while simultaneously capturing the user's facial expressions using the front camera.

[1836] Step 3:

[1837] The device sends the received pulse data to the emotion engine, which analyzes the pulse fluctuations to detect changes in the user's heart rate. For example, an increase in heart rate indicates stress or excitement.

[1838] Step 4:

[1839] The device sends the captured facial image to the emotion engine, which analyzes the facial image, extracts facial features, and identifies the user's emotional state. For example, a smile indicates "joy."

[1840] Step 5:

[1841] The emotion engine integrates both the pulse data and facial expression data to determine the user's final emotional state (e.g., "relaxed," "excited," etc.), and the analysis results are returned to the device.

[1842] Step 6:

[1843] The device then anonymizes the analyzed emotional data, protecting privacy, and transmits it to a server, including the emotional state, device ID, and timestamp.

[1844] Step 7:

[1845] The server stores the received emotional data in a database, and labels it according to the emotional state.

[1846] Step 8:

[1847] The server analyzes the emotional data, compares it with an advertisement database, and selects the most suitable advertisement for the user. For example, if the user is in a relaxed state, it will select a relaxation-related advertisement.

[1848] Step 9:

[1849] The server sends the selected advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content.

[1850] Step 10:

[1851] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the application's context.

[1852] Step 11:

[1853] The device monitors the user's reactions while the ad is displayed, specifically recording the length of time the user's gaze remains on the ad, whether or not they click, and whether or not they scroll the screen.

[1854] Step 12:

[1855] The device then sends the collected reaction data back to the server, including gaze duration, click actions, scroll position, and other information.

[1856] Step 13:

[1857] The server analyzes the reaction data received and evaluates the effectiveness of the advertisement. The characteristics of advertisements that were less effective are analyzed and reflected in future advertisement selection.

[1858] Step 14:

[1859] The server improves and updates the ad delivery algorithm based on new emotional and reaction data, thereby continuously improving the accuracy and effectiveness of ad delivery.

[1860] Example 2

[1861] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1862] Conventional ad delivery systems were unable to accurately reflect the user's real-time emotional state, making it difficult to deliver appropriate ads. Furthermore, they lacked a mechanism for evaluating the effectiveness of ads and providing feedback for future ad delivery, making it difficult to adequately optimize ads. As a result, ads failed to attract user interest and were less effective.

[1863] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1864] In this invention, the server includes sensor means for acquiring user emotional data, analysis means for analyzing the emotional data to identify the user's emotional state, transmission means for anonymizing the identified emotional state and transmitting it to the server, advertisement selection means for selecting an optimized advertisement based on the emotional state and past behavioral history, advertisement distribution means for delivering the selected advertisement to the user, means for monitoring user reaction data during advertisement delivery, and feedback means for transmitting the monitored data to the server and improving the advertisement delivery algorithm. This enables appropriate advertisement delivery based on the user's real-time emotional state, and enables continuous feedback and optimization to improve advertisement effectiveness.

[1865] The term "sensor means" refers to a device for acquiring emotional data of a user.

[1866] The term "analysis means" refers to a device that has the function of analyzing acquired emotion data and identifying the user's emotional state.

[1867] "Transmitting means" refers to a device that has the function of anonymizing the identified emotional state and transmitting it to a server.

[1868] "Advertisement selection means" refers to a device that has the function of selecting an optimized advertisement based on emotional state and past behavioral history.

[1869] "Advertisement distribution means" refers to a device that has the function of distributing selected advertisements to users.

[1870] The term "monitoring means" refers to a device that has the function of monitoring user reaction data during advertisement distribution.

[1871] "Feedback means" refers to a device that has the function of transmitting monitored data to a server and generating feedback to improve the ad delivery algorithm.

[1872] "Emotion data" refers to data that indicates the user's emotional state, and includes facial expressions, pulse rate, and other biological information.

[1873] "Emotional State" refers to the user's current psychological and physiological state as identified by analytical means.

[1874] "Reaction data" refers to data regarding the reactions and actions of users during advertisement delivery.

[1875] "Anonymization" refers to the act of processing data so that it cannot be used to identify individuals.

[1876] "Server" refers to a computer system that receives emotion data and reaction data via a network and performs analysis and advertisement selection.

[1877] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system includes emotion sensor means, analysis means, transmission means, advertisement selection means, advertisement delivery means, monitoring means, and feedback means. Below, we will provide details of each component and how they work together, along with specific examples.

[1878] Collecting Emotional Data

[1879] User

[1880] A user puts on a wearable device and runs a specific application (e.g., a VR application), which captures the user's facial expressions using the front camera.

[1881] Terminal

[1882] The device collects pulse and facial expression data in real time from the front camera and the wearable device. Pulse data is transmitted to the device via Bluetooth, and facial expression data is captured by the front camera.

[1883] Emotional Data Analysis

[1884] Terminal

[1885] The device temporarily stores the collected pulse data and facial expression data locally and sends it to the emotion engine.

[1886] Emotion Engine

[1887] The emotion engine uses machine learning algorithms (e.g., neural networks) to analyze the data and identify the user's emotional state, which may include happiness, sadness, surprise, relaxation, etc., and returns the analysis results to the device.

[1888] Sending emotional data

[1889] Terminal

[1890] After receiving the analysis results from the emotion engine, the device anonymizes the emotion data and sends it to the server along with the device ID and timestamp to protect privacy.

[1891] Ad optimization

[1892] server

[1893] The server stores the received emotion data in a database and runs an algorithm to select the most appropriate ad based on the emotion data, past ad display history, and user action history.

[1894] Ad serving

[1895] server

[1896] The server then sends the optimized ad data to the device, including the ad ID, display timing, and the URL of the ad content.

[1897] Terminal

[1898] The device displays advertisements to the user based on the received advertising data. The timing and format of the display depend on the content. The device also monitors the user's reactions while the advertisement is being displayed (for example, how long the user's gaze remains on the advertisement, whether or not the user clicks, whether the user scrolls the screen, etc.).

[1899] Feedback and algorithm improvements

[1900] Terminal

[1901] After the ad is displayed, the device sends the user's reaction data to the server, including gaze duration, click actions, scroll position, etc.

[1902] server

[1903] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback to discontinue or modify them, and updates and improves the ad delivery algorithm based on the new emotion and reaction data.

[1904] Implementing specific examples

[1905] When a user launches a VR app to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an emotion engine, which uses a machine learning algorithm to detect when the user is relaxed. The results are returned to the device, which then analyzes the emotion data and sends it to the server. The server selects advertisements suitable for a relaxed state (e.g., hot spring trips or relaxation services) and sends them to the device. The device displays these advertisements to the user and monitors the user's reactions while they are displayed. The reaction data is then sent back to the server and used to select advertisements for future use.

[1906] Prompt Sentence Examples

[1907] An example prompt for a generative AI model might look something like this:

[1908] "Select the best ad based on the following emotional state data:

[1909] Emotional state: Relaxed

[1910] Device ID: 12345XYZ

[1911] Timestamp: 2023-10-09 10:00:00

[1912] Advertising history: Hot spring trips, yoga classes

[1913] When this prompt sentence is input into a generative AI model, the optimal advertisement is selected based on the emotional data.

[1914] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1915] Step 1: Collecting emotion data

[1916] User

[1917] The user puts on the wearable device and launches the VR application, which starts capturing the user's facial expressions through the front camera.

[1918] Terminal

[1919] The terminal receives pulse data from the wearable device via Bluetooth, and simultaneously captures the user's facial expression data in real time using the front camera.

[1920] input

[1921] Pulse data and facial expression data

[1922] output

[1923] Raw data temporarily stored in local data storage

[1924] Step 2: Analyze the emotion data

[1925] Terminal

[1926] The device sends the stored pulse data and facial expression data to the emotion engine.

[1927] Emotion Engine

[1928] The emotion engine uses machine learning algorithms to analyze data and identify the user's emotional state. For example, it uses a neural network to analyze facial expression data and pulse rate data to identify an emotional state such as "relaxed."

[1929] input

[1930] Collected pulse data and facial expression data

[1931] Data Processing

[1932] Data analysis using machine learning algorithms

[1933] output

[1934] The user's emotional state (e.g., "Relaxed")

[1935] Step 3: Sending emotion data

[1936] Terminal

[1937] The device anonymizes the emotional state obtained from the emotion engine, and sends it to the server with a device ID and a timestamp.

[1938] input

[1939] Analyzed emotional state, device ID, timestamp

[1940] Data Processing

[1941] Data anonymization

[1942] output

[1943] Anonymized emotion data (e.g., "Relaxed, Device ID: 12345XYZ, Timestamp: 2023-10-09 10:00:00")

[1944] Step 4: Optimize your ads

[1945] server

[1946] The server stores the received emotional data in a database, and then selects the most appropriate ad based on the emotional data, past ad display history, and user action history.

[1947] input

[1948] Anonymized emotional data, past ad viewing history, and user action history

[1949] Data Calculation

[1950] Implementing ad selection algorithms using generative AI models

[1951] output

[1952] Optimized advertising data (e.g., "hot spring trip advertisement")

[1953] Step 5: Serving Ads

[1954] server

[1955] The server then sends the selected advertising data to the device, including the advertising ID, display timing, and the URL of the advertising content.

[1956] Terminal

[1957] The device displays advertisements to the user based on the received advertisement data. The timing and format of the display depend on the content.

[1958] input

[1959] Optimized Advertising Data

[1960] output

[1961] Ad content displayed to users

[1962] Step 6: Monitor while your ad is showing

[1963] Terminal

[1964] While the ad is displayed, the device monitors the user's reactions (such as gaze retention, ad clicks, and scrolling).

[1965] input

[1966] User gaze data, click data, scroll data

[1967] output

[1968] User reaction data stored as log files

[1969] Step 7: Feedback and algorithm refinement

[1970] Terminal

[1971] After the advertisement is displayed, the terminal transmits the user's reaction data to the server.

[1972] server

[1973] The server analyzes the received reaction data and evaluates the effectiveness of the advertisement. It analyzes the characteristics of ineffective advertisements and generates feedback for future advertisements. It also updates and improves the ad delivery algorithm based on the new emotional and reaction data.

[1974] input

[1975] User reaction data

[1976] Data Calculation

[1977] Evaluating advertising effectiveness using analytical algorithms and creating new advertising selection algorithms

[1978] output

[1979] Updated ad serving algorithm

[1980] Through the above processing steps, this system is able to deliver highly accurate advertisements and provide continuous feedback on their effectiveness based on the user's real-time emotional state.

[1981] (Application example 2)

[1982] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1983] Conventional ad delivery systems have had the challenge of being difficult to target accurately based on user interests and states. In particular, ad delivery utilizing emotional data has been rare, making it impossible to provide ads that match the user's emotional state in real time. This has led to problems such as a decline in user engagement and difficulty for advertisers in measuring effectiveness.

[1984] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1985] In this invention, the server includes a transmission means for anonymizing a user's emotional data and transmitting it to the server while protecting their privacy, a feedback means for improving an ad delivery algorithm based on the emotional data and a history of past ad display, and a means for acquiring pulse data and facial expression data in real time from the emotional data, the in-camera, and the wearable device, thereby enabling highly targeted ad delivery based on the user's real-time emotional state.

[1986] The "sensor means" is a device for acquiring emotional data of the user.

[1987] The "analysis means" is a device that analyzes the acquired emotional data and identifies the emotional state of the user.

[1988] The "selection means" is a device that selects the most suitable advertisement based on the analyzed emotional state.

[1989] The "distribution means" is a device that distributes the selected advertisement to the user.

[1990] The "transmission means" is a device that anonymizes the user's emotional data and transmits it to the server while protecting privacy.

[1991] A "feedback means" is a device that improves the ad delivery algorithm based on emotional data and past ad display history.

[1992] A "wearable device" is a device worn by a user to acquire pulse data.

[1993] The "imaging device" is a camera device for capturing the user's facial expression.

[1994] The "emotional state" is the mental state of the user identified as a result of analyzing the emotion data.

[1995] The "advertising delivery algorithm" is an algorithm for controlling the display of advertisements based on emotional data and past advertisement display history.

[1996] This invention is a system that collects and analyzes user emotion data and delivers optimal advertisements based on the analysis results. This system is composed of the following components: sensor means, analysis means, selection means, delivery means, transmission means, and feedback means.

[1997] 1. Collecting Emotional Data

[1998] A user wears a wearable device and runs a specific application (e.g., an app for a smartphone or smart glasses). This application captures the user's facial expressions using an in-camera or an imaging device. Pulse data is also acquired from the wearable device in real time. This data is then transmitted to a terminal via Bluetooth.

[1999] 2. Emotion Data Analysis

[2000] The device sends the collected pulse data and facial expression data to the analysis means, which incorporates a machine learning algorithm that analyzes the emotional data to identify the user's emotional state (e.g., joy, sadness, surprise, etc.). The analysis results are updated in real time, and the emotional state is returned to the device.

[2001] 3. Sending Emotional Data

[2002] The device then anonymizes the analyzed emotional data and transmits it to a server, protecting privacy. The transmitted data includes the emotional state, device ID, and timestamp.

[2003] 4. Ad optimization

[2004] The server stores the received emotional data in a database. Based on the stored emotional data, it matches it with the advertisement database and selects advertisements that suit the user's emotional state. For example, if the user is relaxed, it will select relaxation-related advertisements, and if the user is excited, it will select exciting advertisements. It also takes into account the user's past ad viewing history and user action history.

[2005] 5. Delivery of advertisements

[2006] The server sends the optimized advertising data to the device. The sent data includes the advertising ID, display timing, and URL of the advertising content. The device displays the advertisement to the user based on the received advertising data. While the advertisement is being displayed, the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) are monitored.

[2007] 6. Feedback and algorithm improvements

[2008] The device sends data on the user's reaction after the advertisement is displayed to the server, including the duration of gaze, click actions, scroll position, etc.

[2009] The server analyzes the received reaction data to evaluate the effectiveness of the ads, analyzes the characteristics of ineffective ads and generates feedback for corrections, and improves and updates the ad delivery algorithm based on new emotion and reaction data to continuously improve the accuracy and effectiveness of ad delivery.

[2010] Specific examples

[2011] When a user turns on their smart glasses to relax on a holiday morning, the device captures the user's smile through the front camera and simultaneously receives pulse data from the wearable device. This data is sent to an analysis means, which uses a machine learning algorithm to detect relaxation. The results are returned to the device, which then analyzes the data and sends the analyzed emotional data to the server. The server selects an advertisement suitable for the relaxation state (e.g., a hot spring trip or relaxation service) and sends it to the device. The device displays the advertisement to the user and monitors the user's reaction while viewing it. The reaction data is then sent back to the server and used to select future advertisements. In this way, using an emotion engine enables an advertising delivery system based on the user's real-time emotional state.

[2012] Prompt Sentence Examples

[2013] Collect real-time emotional data while the user is wearing the smart glasses, and if the user's current emotional state is analyzed as "relaxed," select and display appropriate relaxation-related advertisements, such as advertisements for hot spring trips or massage services.

[2014] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2015] Step 1:

[2016] The device uses a camera and a wearable device to collect user emotional data. Specifically, it captures the user's facial expressions using the in-camera and obtains pulse data from the wearable device via Bluetooth.

[2017] Input: User's facial expression data (image) and pulse data

[2018] Output: Captured facial expression data and acquired pulse data

[2019] Step 2:

[2020] The terminal transmits the collected facial expression data and pulse rate data to the analysis means, which uses a machine learning algorithm to analyze the emotion data and identify the user's emotional state.

[2021] Input: Captured facial expression data and acquired pulse data

[2022] Data processing / computation: Identifying emotional states using machine learning algorithms

[2023] Output: Identified user emotional state (happy, sad, surprised, etc.)

[2024] Step 3:

[2025] The device anonymizes the analyzed emotion data and transmits it to a server while protecting privacy.

[2026] Input: Identified user emotional state

[2027] Data processing / calculation: Data anonymization

[2028] Output: Anonymized emotional data (emotional state, device ID, timestamp)

[2029] Step 4:

[2030] The server stores the received emotional data in a database, matches the stored emotional data with the advertisement database, and selects advertisements that match the emotional state.

[2031] Input: Anonymized emotion data

[2032] Data processing / calculation: database search and matching processing

[2033] Output: Selected ad data (ad ID, display timing, ad content URL, etc.)

[2034] Step 5:

[2035] The server transmits the selected advertisement data to the terminal, and the terminal displays an advertisement to the user based on the received advertisement data.

[2036] Input: Selected advertising data

[2037] Data processing / calculation: Sending advertising data and displaying advertisements

[2038] Output: Ad content shown to the user

[2039] Step 6:

[2040] The device monitors the user's reactions (how long they maintain their gaze, whether they click, whether they scroll the screen, etc.) while the advertisement is displayed.

[2041] Input: User reaction data

[2042] Data processing / calculation: Reaction data capture and recording

[2043] Output: Captured reaction data

[2044] Step 7:

[2045] The device sends user reaction data after the ad is displayed to the server. The server analyzes the received reaction data, evaluates the effectiveness of the ad, and generates feedback to help select future ads.

[2046] Input: Captured reaction data

[2047] Data processing / calculation: Effect analysis and feedback generation

[2048] Output: Effect analysis results and feedback information

[2049] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2050] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2051] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2052] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2053] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2054] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2055] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2056] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2057] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2058] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2059] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2060] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2061] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2062] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2063] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2064] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2065] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2066] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2067] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2068] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2069] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2070] The following is further disclosed regarding the above embodiment.

[2071] (Claim 1)

[2072] emotion sensor means for acquiring emotion data of a user;

[2073] analysis means for analyzing the emotion data to identify the user's emotional state;

[2074] an advertisement selection means for selecting an advertisement optimized based on the emotional state;

[2075] advertisement distribution means for distributing the advertisement to users;

[2076] A system including:

[2077] (Claim 2)

[2078] 2. The system of claim 1, wherein the emotion sensor means includes a wearable device that acquires pulse data of the user.

[2079] (Claim 3)

[2080] 10. The system of claim 1, wherein the emotion sensor means includes a camera that captures the user's facial expressions.

[2081] (Claim 4)

[2082] 2. The system of claim 1, wherein the analyzing means analyzes both pulse data and facial expression data to identify an emotional state.

[2083] (Claim 5)

[2084] 2. The system according to claim 1, wherein the advertisement selection means optimizes advertisements based on the user's past advertisement display history and behavior history.

[2085] (Claim 6)

[2086] 2. The system according to claim 1, wherein the advertisement distribution means monitors user reactions and transmits reaction data to a server.

[2087] (Claim 7)

[2088] 7. The system according to claim 6, wherein the advertisement selection means continuously improves the advertisement distribution algorithm based on the reaction data.

[2089] "Example 1"

[2090] (Claim 1)

[2091] emotion sensor means for acquiring emotion data of a user;

[2092] analysis means for analyzing the emotion data to identify the user's emotional state;

[2093] a transmitting means for anonymizing the emotional state and transmitting it to a server while protecting privacy;

[2094] an advertisement selection means for matching the received emotion data with an advertisement database and selecting an advertisement optimized based on the emotion state;

[2095] advertisement distribution means for distributing the advertisement to users;

[2096] a monitoring means for monitoring a user's reaction while the advertisement is being displayed;

[2097] An algorithm improvement means for evaluating the effectiveness of advertisements based on the monitoring results and reflecting the results in subsequent targeting;

[2098] A system including:

[2099] (Claim 2)

[2100] 2. The system of claim 1, wherein the emotion sensor means includes a wearable device that acquires pulse data of the user.

[2101] (Claim 3)

[2102] 10. The system of claim 1, wherein the emotion sensor means includes a camera that captures the user's facial expressions.

[2103] "Application Example 1"

[2104] (Claim 1)

[2105] emotion sensor means for acquiring emotion data of a user;

[2106] analysis means for analyzing the emotion data to identify the user's emotional state;

[2107] an advertisement selection means for selecting an advertisement optimized based on the emotional state;

[2108] a monitoring means for monitoring user reaction data to the advertisement;

[2109] a transmitting means for transmitting user reaction data after the advertisement is displayed to a server and evaluating the effectiveness of the advertisement;

[2110] an algorithm improvement means for reselecting advertisements using an improved advertisement distribution algorithm;

[2111] advertisement distribution means for distributing the advertisement to users;

[2112] A system including:

[2113] (Claim 2)

[2114] 2. The system of claim 1, wherein the emotion sensor means includes a wearable device that acquires pulse data of the user.

[2115] (Claim 3)

[2116] 10. The system of claim 1, wherein the emotion sensor means includes a camera that captures the user's facial expressions.

[2117] "Example 2: Combining Emotion Engines"

[2118] (Claim 1)

[2119] a sensor means for acquiring emotion data of a user;

[2120] analysis means for analyzing the emotion data to identify the user's emotional state;

[2121] a transmitting means for anonymizing the identified emotional state and transmitting the same to a server;

[2122] an advertisement selection means for selecting an advertisement optimized based on the emotional state and the past behavioral history;

[2123] an advertisement distribution means for distributing the selected advertisement to a user;

[2124] means for monitoring user reaction data during the advertisement distribution;

[2125] a feedback means for transmitting the monitored data to a server and improving an advertisement delivery algorithm;

[2126] A system including:

[2127] (Claim 2)

[2128] 10. The system of claim 1, wherein the sensor means includes a wearable device that acquires pulse data of the user.

[2129] (Claim 3)

[2130] 10. The system of claim 1, wherein the sensor means includes a camera that captures the user's facial expressions.

[2131] "Application example 2 when combining emotion engines"

[2132] (Claim 1)

[2133] a sensor means for acquiring emotion data of a user;

[2134] analysis means for analyzing the emotion data to identify the user's emotional state;

[2135] a selection means for selecting an advertisement optimized based on the emotional state;

[2136] a distribution means for distributing the advertisement to a user;

[2137] a transmitting means for anonymizing the user's emotion data and transmitting the data to a server while protecting the user's privacy;

[2138] a feedback means for improving an advertisement delivery algorithm based on the emotion data and a history of past advertisement display;

[2139] A system including:

[2140] (Claim 2)

[2141] 2. The system according to claim 1, wherein the emotion sensor means includes a wearable device that acquires pulse data of the user.

[2142] (Claim 3)

[2143] 10. The system of claim 1, wherein the emotion sensor means includes an image capture device that captures the user's facial expressions. [Explanation of symbols]

[2144] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. emotion sensor means for acquiring emotion data of a user; analysis means for analyzing the emotion data to identify the user's emotional state; an advertisement selection means for selecting an advertisement optimized based on the emotional state; advertisement distribution means for distributing the advertisement to users; A system including:

2. 2. The system of claim 1, wherein the emotion sensor means includes a wearable device that acquires pulse data of the user.

3. 2. The system of claim 1, wherein said emotion sensor means includes a camera for capturing a user's facial expressions.

4. 2. The system of claim 1, wherein said analyzing means analyzes both pulse data and facial expression data to identify emotional states.

5. 2. The system according to claim 1, wherein said advertisement selection means optimizes advertisements based on the user's past advertisement display history and behavior history.

6. 2. The system according to claim 1, wherein the advertisement distribution means monitors user reactions and transmits reaction data to a server.

7. 7. The system according to claim 6, wherein the advertisement selection means continuously improves the advertisement distribution algorithm based on the reaction data.

Images