Electronic device, control method, and program
By using user eyeball information to adjust advertising content on wearable devices, the device enhances user engagement and advertising effectiveness by showing content aligned with user preferences, addressing the challenge of low-interest content on wearable devices.
Patent Information
- Application Number
- PCT/JP2024/014546
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-04-10
- Publication Date
- 2024-11-21
AI Technical Summary
Existing methods for displaying advertising content on wearable devices, such as head-mounted displays, fail to effectively increase user engagement as they often show content of low interest to users, leading to reduced advertising effectiveness.
An electronic device with a display unit, an acquisition unit for user eyeball information, and a control mechanism that adjusts the display of advertising content based on user preference levels, estimated through gaze detection and pupil response analysis, to show content that aligns with user interest.
Enhances advertising effectiveness by displaying preferred content to users, thereby improving user engagement and increasing the relevance of advertisements during video content playback on wearable devices.
Smart Images

Figure JP2024014546_21112024_PF_FP_ABST
Abstract
Description
Electronic device, control method, and program
[0001] The present invention relates to an electronic device that can be worn and used by a user.
[0002] Personal video content viewing environments using wearable devices such as head-mounted displays (HMDs) are becoming increasingly common. When viewing personal video content, advertising content may be displayed during playback of the video content, similar to viewing on a television receiver or a tablet computer (PC). In this case, if advertising content is unconditionally displayed, users may feel annoyed by uninterested advertising content, or the effectiveness of advertising may be reduced for users who are not interested in the advertising content.
[0003] Patent Document 1 describes a method for changing the frequency or duration of displaying advertising content depending on the user's level of interest.
[0004] Japanese Patent Application Laid-Open No. 2019-40565
[0005] However, in Patent Document 1, advertising content that is of little interest to users is displayed infrequently and for short periods of time, which is not sufficient to increase advertising effectiveness for users watching video content on wearable devices such as HMDs.
[0006] The present invention has been made in view of the above-mentioned problems, and realizes a technology for displaying content that a user prefers.
[0007] In order to solve the above problem, the electronic device of the present invention has a display means for displaying a first content, an acquisition means for acquiring eyeball information of a user viewing the first content, and a control means for setting a second content to be displayed while the first content is being played based on the eyeball information.
[0008] According to the present invention, it is possible to display content that a user prefers.
[0009] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.
[0010] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and are used, together with the description, to explain the principles of the present invention. FIG. 1 is an external view of a wearable device according to this embodiment. FIG. 2 is a block diagram illustrating the configuration of a wearable device according to this embodiment. FIG. 3A is a flowchart illustrating a control process of a wearable device according to embodiment 1. FIG. 3B is a flowchart illustrating a control process of a wearable device according to embodiment 1. FIG. 4A is an explanatory diagram of a preference estimation method according to this embodiment. FIG. 4B is an explanatory diagram of a preference estimation method according to this embodiment. FIG. 4C is an explanatory diagram of a preference estimation method according to this embodiment. FIG. 4D is an explanatory diagram of a preference estimation method according to this embodiment. FIG. 4E is an explanatory diagram of a preference estimation method according to this embodiment. FIG. 5A is an explanatory diagram of a preference update method according to this embodiment. FIG. 5B is an explanatory diagram of a preference update method according to this embodiment. FIG. 5C is an explanatory diagram of a preference update method according to this embodiment. FIG. 5D is an explanatory diagram of a preference update method according to this embodiment. FIG. 6A is an explanatory diagram of an advertising content setting method according to this embodiment. FIG. 6B is an explanatory diagram of an advertisement content setting method of this embodiment. FIG. 6C is an explanatory diagram of an advertisement content setting method of this embodiment. FIG. 7A is a flowchart illustrating a control process of a wearable device of embodiment 2. FIG. 7B is a flowchart illustrating a control process of a wearable device of embodiment 2. FIG. 8A is an explanatory diagram of a preference updating method of embodiment 2. FIG. 8B is an explanatory diagram of a preference updating method of embodiment 2. FIG. 9A is an explanatory diagram of a preference updating method of embodiment 2. FIG. 9B is an explanatory diagram of a preference updating method of embodiment 2. FIG. 10A is a flowchart illustrating a control process of a wearable device of embodiment 3. FIG. 10B is a flowchart illustrating a control process of a wearable device of embodiment 3.
[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.
[0012] Hereinafter, an embodiment in which the electronic device of the present invention is applied to a wearable device, a head mounted display (HMD) having a gaze detection function, will be described in detail with reference to the accompanying drawings. Note that the wearable device is not limited to an HMD, and may be a smart device such as a smartphone, tablet computer, eyeglass-type device, or wristwatch-type device having a gaze detection function.
[0013] <Device Configuration> The configuration and functions of the wearable device 10 of this embodiment will be described with reference to FIGS.
[0014] FIG. 1 is an external view of a wearable device 10 according to the present embodiment.
[0015] The wearable device 10 of this embodiment can be worn on the user's head with goggle-shaped displays covering both eyes. The wearable device 10 has a left-eye display device 120L and a right-eye display device 120R. The left-eye display device 120L displays image content to be viewed with the user's left eye, and the right-eye display device 120R displays image content to be viewed with the user's right eye. The left-eye display device 120L and the right-eye display device 120R can also display image content such as virtual reality (VR) images using the parallax between the left and right eyes. The image content may be still images or videos. The wearable device 10 of this embodiment is a binocular type equipped with a left-eye display device 120L and a right-eye display device 120R, but may also be a monocular type equipped with a left-eye display device 120L or a right-eye display device 120R. In this embodiment, the image content includes videos and advertisements, and the videos include audio.
[0016] FIG. 2 is a block diagram illustrating an example of the internal configuration of the wearable device of this embodiment.
[0017] The system control unit 101 includes an arithmetic processor such as a CPU or MPU that controls the entire wearable device 10, and executes programs stored in the nonvolatile memory 102 to realize the processing of each flowchart described below. The system memory 103 is a RAM or the like, and is also used as a work memory for expanding constants and variables for the operation of the system control unit 101, programs read from the nonvolatile memory 102, and the like. The system control unit 101 also controls the display of the left eye display device 120L and the right eye display device 120R. The system timer 104 is a timing unit that measures the time used for various controls and the time of the built-in clock.
[0018] The image processing unit 105 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the data from the memory control unit 106 .
[0019] The memory control unit 106 controls the exchange of data between the image processing unit 105 and the memory 107. Data output from the image processing unit 105 is written to the memory 107 via the memory control unit 106. The memory 107 stores display data for content such as videos and advertisements to be displayed on the display unit 122. The memory 107 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of video and audio. The memory 107 also serves as a memory for displaying images (video memory). The memory 107 also stores information (user information) related to the user's status, such as the user's preferences, determined by the status determination unit 135 (described later). The system control unit 101 transmits the user information stored in the memory 107 to the content setting unit 140 (described later) and to the database 30 (an external device) via the communication unit 113.
[0020] The operation unit 108 includes operation members such as various switches and buttons that receive various operations from the user and notify the system control unit 101. The operation unit 108 includes at least a power switch 109.
[0021] The power supply control unit 110 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 110 also controls the DC-DC converter based on the detection results and instructions from the system control unit 101, and supplies the required voltage for the required period to each unit, including the recording medium 200.
[0022] The power supply unit 111 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li-ion battery, an AC adapter, or the like.
[0023] A recording medium interface (I / F) 112 is an interface with a recording medium 200 such as a memory card.
[0024] The communication unit 113 is communicatively connected to an external device such as the database 30 via a wireless antenna or a wired cable, and transmits and receives images, audio, and other data. The communication unit 113 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 113 can transmit data recorded on the recording medium 200 to an external device, and can also receive necessary data from an external device. Note that the communication unit 113 is not limited to a wireless LAN; it may also use a wireless communication module such as infrared communication, Bluetooth (registered trademark), Bluetooth (registered trademark) Low Energy, or Wireless USB, or a wired connection means such as a USB cable, HDMI (registered trademark), or IEEE 1394.
[0025] The left-eye display device 120L and the right-eye display device 120R each include an eyepiece 121, a display 122, and an eyeball detection unit 130. The eyepiece 121 is an eyepiece of an eyepiece finder (a peer-type finder), and a user can view content such as videos and advertisements displayed on the display 122 through the eyepiece 121. The display 122 is an electronic viewfinder (EVF) that displays content such as videos and advertisements.
[0026] The eyepiece detection unit 114 detects whether an eye (object) is approaching (approaching) or moving away (away from) the eyepiece unit 121. The system control unit 101 determines whether the wearable device 10 is being worn by a user, based on the state detected by the eyepiece detection unit 114. If the system control unit 101 determines that the wearable device 10 is being worn by a user, it turns on the power of the wearable device 10 and switches the display unit 122 to a display state. If the system control unit 101 determines that the wearable device 10 is not being worn by a user, it turns off the power of the wearable device 10 and switches the display unit 122 to a non-display state.
[0027] When an object approaches, infrared light emitted from a light-emitting unit (not shown) of the eyepiece detection unit 114 is reflected and incident on a light-receiving unit (not shown) of the infrared proximity sensor. The amount of incident infrared light received by the infrared proximity sensor can be used to detect the approach of an object to the eyepiece 121 and determine the distance (eyepiece distance) from the eyepiece 121. When an object is detected approaching the eyepiece 121, the system control unit 101 can start displaying on the display unit 122. This allows the display unit 122 to be displayed with as little delay as possible when the user looks into the eyepiece 121.
[0028] Furthermore, when the eyepiece detection unit 114 detects an object approaching within a predetermined distance from the eyepiece unit 121 from a non-eyepiece state (non-approach state), it determines that eyepiece detection has been detected, and transmits an eyepiece detection notification to the system control unit 101. Furthermore, when the object that was detected as approaching from the eyepiece state (approach state) moves away by more than a predetermined distance, it determines that eyepiece separation has been detected, and transmits an eyepiece separation detection notification to the system control unit 101. The threshold for detecting eyepiece separation and the threshold for detecting eyepiece separation may be different, for example, by providing hysteresis. Furthermore, after eyepiece separation is detected, the eyepiece remains in the eyepiece state until eyepiece separation is detected. After eyepiece separation is detected, the non-eyepiece state remains until eyepiece separation is detected. As a result, the system control unit 101 controls the display of the display unit 122 according to the eyepiece state or eyepiece separation state detected by the eyepiece detection unit 114.
[0029] The eye proximity detection unit 114 is not limited to an infrared proximity sensor, and any other sensor may be used as long as it can detect the approach of an eye or an object that can be considered as an eye proximity.
[0030] The eyeball detection unit 130 includes an infrared light-emitting element 131, an imaging lens 132, and an event sensor 133, and can detect information about the user's eyes (eyes) 20L, 20R (hereinafter referred to as eyeball information) looking into the eyepiece unit 121. The eyeball detection unit 130 can also detect information about the user's line of sight toward the display screen of the display unit 122. The eyeball detection unit 130 detects line of sight information, such as the position and direction of the line of sight, from the positional relationship between the light that is generated by infrared light emitted from the infrared light-emitting element 131 and reflected by the eyeballs (eyes) 20L, 20R and the pupils of the eyeballs (eyes) 20L, 20R.
[0031] The infrared light emitting element 131 is a diode that emits infrared light, and irradiates the infrared light toward the vicinity of the centers of the eyeballs (eyes) 20L, 20R of the user looking through the eyepiece 121. The infrared light irradiated from the infrared light emitting element 131 is reflected by the eyeballs (eyes) 20L, 20R, and the reflected infrared light reaches the event sensor 133 via the imaging lens 132. The imaging lens 132 has the function of reflecting only infrared light and transmitting visible light, and the reflected infrared light whose optical path has been changed forms an image on the imaging surface of the event sensor 133 via the imaging lens 132.
[0032] When the gaze detection function of the eyeball detection unit 130 is enabled, the system control unit 101 can determine the gaze input state of the user looking at the display unit 122 through the eyepiece unit 121 based on the gaze detection result of the eyeball detection unit 130, and executes processing according to the operation performed by the gaze input.
[0033] The event sensor 133 is an event-based vision sensor that detects changes in the luminance of light incident on each pixel and outputs information (coordinates, polarity, and time) of the pixel where the luminance change was detected as event information asynchronously with other pixels. The event information output from the event sensor 133 includes, for example, the coordinates of the pixel where the luminance change (event) occurred, the polarity (positive or negative) of the luminance change, and time information indicating the time when the event occurred. Compared to existing frame-based synchronous sensors such as image sensors, the event sensor 133 eliminates redundancy in the output information and has characteristics such as high-speed operation, a high dynamic range, and low power consumption. Because the event information is output asynchronously, various calculations must be performed on the results of accumulating event information that occurs within a certain period of time in order to determine the correlation between the event information.
[0034] The event calculation unit 134 performs calculations to detect eye information from event information continuously and asynchronously output from the event sensor 133. For example, the event calculation unit 134 accumulates event information occurring within a predetermined time period and processes the accumulated information as a set of data to determine the presence or absence of eye information. Changing the accumulation time of the event information makes it possible to detect multiple pieces of eye information occurring at different speeds. The eye information includes eye movements such as gaze position information, saccade information including the direction and speed of rapid eye movements (saccades), and microsaccade information including the frequency and amplitude of microsaccades. Pupil information including pupil size and pupil changes, and blink information such as blink speed and frequency may also be added. Note that the above-described eye information is merely an example and is not limited to these. The event calculation unit 134 may be configured to map the event information for the accumulation time as one frame of image data based on the event occurrence coordinates and perform image processing. This allows multiple pieces of eye information to be acquired from the event information using techniques conventionally used in frame-based image processing.
[0035] The state determination unit 135 determines the user's state based on the eyeball information detected by the event calculation unit 134. For example, the gaze range or gaze intensity (overhead gaze intensity) can be determined from the frequency and amplitude of microsaccades. Here, the gaze range is synonymous with the attention range or focus range. The gaze intensity is higher when the gaze range is narrower and lower when the gaze range is wider. The overhead gaze intensity is defined as the antonym of the gaze intensity. Preference levels, such as facial preferences, are related to the speed of microsaccades and pupil diameter and can be determined from both parameters. The state determination unit 135 can be configured, for example, by a neural network that receives parameters related to microsaccades, blinks, pupils, and subject identification results as input and outputs indicators representing the user's state, such as the gaze range or gaze intensity (antonymously, overhead gaze intensity) and preference level (hereinafter, referred to as user state information). Note that the state determination unit 135 is not limited to the configuration described above. Furthermore, the eyeball information and the determination results used by the state determination unit 135 are not limited to those described above.
[0036] The gaze input setting unit 136, under the control of the system control unit 101, can enable or disable the gaze detection function of the eyeball detection unit 130, and set parameters and detection conditions used by the event calculation unit 134 and the state determination unit 135. The enable or disable of the gaze detection function and the parameters and detection conditions of the event calculation unit 134 and the state determination unit 135 can be set, for example, by operating a GUI (Graphical User Interface) using the operation unit 108.
[0037] When the gaze detection function of the eyeball detection unit 130 is disabled, the system control unit 101 does not perform preference estimation by the state determination unit 135 because the gaze detection result of the eyeball detection unit 130 cannot be obtained.
[0038] The system control unit 101 can detect information about the position and size of objects (people (faces), animals, vehicles, etc.) included in the video content being played back on the display screen of the display unit 122. Furthermore, the system control unit 101 can determine the object that the user gazing at the display unit 122 through the eyepiece unit 121 is gazing at, based on the user's line of sight information detected by the eyeball detection unit 130.
[0039] The content setting unit 140 changes the advertising content to be displayed on the display unit 122 based on the user's preference. The content setting unit 140 controls the change of advertising content based on, for example, whether the preference estimated by the state determination unit 135 is higher than a threshold value. Details of the control process will be described later with reference to FIGS. 3A to 10B.
[0040] First Embodiment Next, a control process according to a first embodiment will be described with reference to FIGS. 3A to 6C.
[0041] In the first embodiment, a user's preference for an object that the user is gazing at while video content is being played back is estimated, and advertising content is displayed according to the user's preference.
[0042] 3A and 3B are flowcharts illustrating the control process of the first embodiment.
[0043] The processing in Figures 3A-3B is realized by the system control unit 101 expanding a program stored in nonvolatile memory 102 into system memory 103, executing it, and controlling each component of wearable device 10. The processing in Figures 3A-3B starts when wearable device 10 is powered on. The same applies to Figures 7A-7B and 10A-10B, which will be described later.
[0044] In step S301, the system control unit 101 sets a classification or type (hereinafter, "class") of an object for which the state determination unit 135 estimates the user's preference. The object class may include, for example, at least one of people (faces), animals, vehicles, food, and fashion, and may be set by the user operating the GUI using the operation unit 108, or a class related to the video content to be played may be automatically set.
[0045] In step S302, the system control unit 101 starts playing back the video content.
[0046] In step S303, the system control unit 101 determines whether the state determination unit 135 has detected the user's gaze in a range narrower than the predetermined gaze range. If the system control unit 101 determines that the user's gaze has been detected in a range narrower than the predetermined gaze range, the process proceeds to step S303. If the system control unit 101 does not determine that the user's gaze has been detected in a range narrower than the predetermined gaze range, the process proceeds to step S308. A method for detecting the user's gaze range will be described later. Note that in step S303, the range in which the user is gazing is limited to a range narrower than the predetermined gaze range as a condition for performing preference estimation in step S305, but this limitation does not have to be imposed.
[0047] In step S304, the system control unit 101 determines whether one or more target objects related to the class set in step S301 are present in the user's gaze range detected by the state determination unit 135. If the system control unit 101 determines that a target object is present in the gaze range detected by the state determination unit 135, the process proceeds to step S305. If the system control unit 101 determines that a target object is not present in the gaze range detected by the state determination unit 135, the process proceeds to step S308.
[0048] In step S305, the system control unit 101 estimates the user's preference for one or more target objects determined in step S304, and stores the estimated preference in the memory 107. The user's preference for a target object is estimated, for example, from the amount of change in pupil diameter, as will be described in detail later. Note that the method for estimating the preference is not limited to using the amount of change in pupil diameter, and the speed of microsaccades, the degree of gaze, the gaze duration, or other parameters may also be used.
[0049] In step S306, the system control unit 101 determines whether the user's preference for the target object estimated in step S305 (hereinafter referred to as the current preference) is higher than the user's preference for the same target object (hereinafter referred to as the previous preference) stored in the memory 107. If the system control unit 101 determines that the current preference for the same target object is higher than the previous preference, the process proceeds to step S307. If the system control unit 101 determines that the current preference for the same target object is equal to or lower than the previous preference, the process proceeds to step S308.
[0050] In step S307, the system control unit 101 updates the previous preference level for the same target object stored in the memory 107 to the current preference level, and then proceeds to step S308.
[0051] In step S308, the system control unit 101 determines whether or not the playback of the video content for the predetermined section has ended. If the system control unit 101 determines that the playback of the video content for the predetermined section has ended, the process proceeds to step S309. If the system control unit 101 determines that the playback of the video content for the predetermined section has not ended, the process returns to step S303.
[0052] In step S309, the system control unit 101 determines whether or not there is any object with a preference level higher than a predetermined threshold 601 among the user's preference levels for all target objects determined in step S304. If the system control unit 101 determines that there is an object with a preference level higher than the predetermined threshold 601, the process proceeds to step S310, and if the system control unit 101 determines that there is no object with a preference level higher than the predetermined threshold 601, the process proceeds to step S313. The predetermined threshold 601 will be described later.
[0053] In step S310, the system control unit 101 sets the advertising content related to the object for which the content setting unit 140 has a preference level higher than the predetermined threshold 601 as the display target.
[0054] In steps S309 and S310, the predetermined threshold value 601 is used for the determination, but it is also possible to set the advertising content related to the object with the highest preference as the display target without setting a threshold value.
[0055] In step S311, the system control unit 101 plays back the advertising content set in step S310 or S313.
[0056] In step S312, the system control unit 101 determines whether or not to end the playback of the video content. If the system control unit 101 determines that the playback of the video content should be ended, the process ends. If the system control unit 101 determines that the playback of the video content should not be ended, the process proceeds to step S314.
[0057] In step S313, the system control unit 101 causes the content setting unit 171 to set the default advertising content as the display target.
[0058] In step S314, the system control unit 101 initializes the user's preference degrees for all target objects stored in the memory 107 by the memory control unit 106, and returns the process to step S303. Note that in step S314, the preference degrees stored in the memory 107 are initialized when the playback of the video content ends, but initialization is not necessary.
[0059] 4A to 4E are diagrams for explaining the method of estimating the preference level in step S305 of FIG. 3A.
[0060] 4A and 4B show microsaccade information, which is one type of eyeball information acquired by the event calculation unit 134. FIG.
[0061] 4C-4E are diagrams illustrating examples of the user's gaze state on the display screen of the display unit 122. In Fig. 4C, for example, the class of the target object for which preference level is to be estimated is a person (face), and there is a predetermined gaze range 405, a first object 406, and a second object 407.
[0062] 4A illustrates a microsaccade waveform when the gaze range is relatively wide. The vertical axis represents the position (angle) of the pupil center in the eyeball-centered coordinate system, and the horizontal axis represents time. Area 401 shows the change in eyeball position when a microsaccade occurs. The wider the gaze range, the larger the amplitude of eye movement and the higher the oscillatory rate (lower the attenuation rate). Furthermore, the frequency of microsaccade occurrence over a given period of time tends to be higher.
[0063] 4B illustrates an example of a microsaccade waveform when the gaze range is relatively narrow. 403 indicates the change in eye position when a microsaccade occurs. The narrower the gaze range, the smaller the amplitude of the eye movement and the lower the oscillatory nature (higher the attenuation rate). Furthermore, the frequency of microsaccade occurrence during period 404, which is the same as period 402 in FIG. 4A, tends to be low. As described above, the size of the gaze range can be estimated using the feature amount of eye movement as a parameter.
[0064] 4C illustrates an example of a user's gaze state when the gaze range of FIG. 4A is relatively wide. Region 408 is the user's gaze range. In the example of FIG. 4C , for example, the user is looking down on the first object 406 and the scenery, and the user's gaze range 408 is larger than the predetermined gaze range 405. In this case, preference estimation is not performed.
[0065] 4D illustrates an example of a user's gaze state when the gaze range of FIG. 4B is relatively narrow. Region 409 is the user's gaze range. In the example of FIG. 4D, the user is gazing at a first object 406, and the user's gaze range 409 is narrower than the predetermined gaze range 405. In this case, a preference level is estimated.
[0066] 4E illustrates an example of a user's gaze state when the gaze range of FIG. 4B is relatively narrow. Region 410 is the user's gaze range. In the example of FIG. 4E, the user is gazing at a portion of the landscape. The user's gaze range 410 is narrower than the predetermined gaze range 405. However, since the user is not gazing at a person (face), which is the target object for which preference level estimation is to be performed, preference level estimation is not performed.
[0067] 5A and 5B are diagrams for explaining a method for determining whether to update the preference level in step S306 of FIG. 3A.
[0068] 5A and 5B show examples of changes in pupil diameter, which is one piece of eyeball information acquired by the event calculation unit 134. FIG.
[0069] 5C and 5D show examples of the current preference level and the previous preference level estimated by the state determination unit 135. The preference levels 503 and 504 are the preference levels of the first object 406, for example.
[0070] 5A illustrates an example of the change in pupil diameter when the preference level is relatively high. The vertical axis represents pupil diameter, and the horizontal axis represents time. Waveform 501 illustrates an example of the change in pupil diameter when the user views a favorite object. The higher the user's preference, i.e., the preference level, the greater the change in pupil diameter tends to be.
[0071] FIG. 5B illustrates an example of the change in pupil diameter over time when the preference level is relatively low. The vertical axis represents pupil diameter, and the horizontal axis represents time. Waveform 502 illustrates an example of the change in pupil diameter when the user views an object that is not to their liking. The less the user likes an object, i.e., the lower the preference level, the smaller the change in pupil diameter tends to be. In this way, the preference level can be estimated by using the amount of change in pupil diameter as a parameter.
[0072] 5C illustrates a case where the preference level in FIG. 5A is relatively high and the current preference level estimated by the state determination unit 135 is higher than the previous preference level. When the current preference level 504 is higher than the previous preference level 503, the previous preference level 503 stored in the memory 107 is updated to the current preference level 504.
[0073] 5D illustrates a case where the preference level in FIG. 5B is relatively low and the current preference level estimated by the state determination unit 135 is equal to or lower than the previous preference level. If the current preference level 505 is lower than the previous preference level 503, the previous preference level 503 stored in the memory 107 is not updated to the current preference level 505.
[0074] 6A to 6C are diagrams for explaining the method of setting the advertising content in steps S310 and S313 of FIG. 3B.
[0075] For example, the class of the target object for which preference is to be estimated is assumed to be a person (face).
[0076] A predetermined threshold 601 is set for the user's preference for each target object in order to set advertising content to be displayed. If there is an object for which the user's preference is higher than the predetermined threshold 601, advertising content for that object is set as the display target. Preference levels 602 to 606 are the preference levels for objects A to E. In the example of Figures 6A to 6C, for ease of explanation, there are five types of target objects, but this is not limited to this and there may be more than five or less than five.
[0077] 6A illustrates an example of preference levels when advertising content related to one object is set as a display target. In the example of FIG. 6A, the preference level 606 of object E is higher than a predetermined threshold value 601. In this case, advertising content related to object E is set as a display target. For example, advertising content in which object E appears is set as a display target.
[0078] 6B illustrates an example of preference levels when advertising content related to multiple objects is set as a display target. The preference level 602 for object A and the preference level 606 for object E are higher than a predetermined threshold 601. In this case, advertising content related to object A and object E is set as a display target. For example, advertising content featuring object A and object E, or advertising content featuring object A or object E, is set as a display target.
[0079] 6C illustrates an example of preference levels when default advertising content is set as the display target. Preference levels 602 to 606 are all lower than a predetermined threshold value 601. In this case, the default advertising content is set as the display target.
[0080] Although an example has been given in which an object with a user preference level higher than a predetermined threshold 601 appears as the advertising content to be displayed, other advertising content relevant to the object may also be set, or the advertising content may be changed based on the setting of the object's class.
[0081] As described above, according to the first embodiment, while video content is being played, the user's preference for an object that the user is gazing at is estimated, and advertising content according to the user's preference is displayed, thereby improving the effectiveness of advertising while video content is being played.
[0082] Second Embodiment Next, a second embodiment will be described with reference to FIGS. 7A to 9B.
[0083] In embodiment 2, while video content is being played, the user's preference for an object that the user is gazing at is estimated, and advertising content is displayed according to the user's preference, taking into account the user's past preference estimated when the user previously viewed the same video content.
[0084] In the second embodiment, the configuration and functions of the wearable device 10 are similar to those of the first embodiment shown in FIGS.
[0085] 7A and 7B are flowcharts illustrating control processing in embodiment 2. In Figures 7A and 7B, the same processes as those in Figures 3A and 3B are assigned the same step numbers, and descriptions thereof will be omitted.
[0086] In step S701, the system control unit 101 determines whether the video content whose playback started in step S302 has been viewed in the past. If the system control unit 101 determines that the video content has been viewed in the past, the process proceeds to step S702. If the system control unit 101 determines that the video content has not been viewed in the past, the process proceeds to step S306.
[0087] In step S702, the system control unit 101 determines whether the user's preference for the target object estimated in step S305 (current preference) is lower than the user's preference for the target object estimated when the video content was viewed in the past (hereinafter referred to as past preference). The past preference is stored in the non-volatile memory 102 when the past video content is played back, and is loaded into the memory 107 when the same content is played back this time. If the system control unit 101 determines that the current preference is lower than the past preference, the process proceeds to step S703. If the system control unit 101 determines that the current preference is higher than the past preference, the process proceeds to step S704.
[0088] In step S703, the system control unit 101 updates the current preference level stored in the memory 107 to the past preference level, and proceeds to step S306. A method for determining whether to update the preference level will be described later.
[0089] In step S704, the system control unit 101 determines whether the current preference level is higher than the update parameter of the previous preference level. The update parameter of the previous preference level is stored in the non-volatile memory 102 when the previous video content is played back, and is loaded into the memory 107 when the current content is played back. If the system control unit 101 determines that the current preference level is higher than the update parameter of the previous preference level, the process proceeds to step S705. If the system control unit 101 determines that the current preference level is equal to or lower than the update parameter of the previous preference level, the process proceeds to step S306.
[0090] In step S705, the system control unit 101 updates the update parameters for the past preference levels stored in the memory 107 to the current preference levels, and the process proceeds to step S306.
[0091] In step S706, the system control unit 101 updates the past preference degrees stored in the memory 107 to the update parameters of the past preference degrees updated in step S705, and stores the updated parameters in the non-volatile memory 102. The method for updating the update parameters of the past preference degrees will be described later.
[0092] In step S707, the system control unit 101 initializes the update parameters of the past preference levels stored in the memory 107 using the memory control unit 106, and then ends the process.
[0093] 8A and 8B are diagrams for explaining a method for determining whether to update the preference level in step S702 of FIG. 7A.
[0094] 8A and 8B illustrate examples of current preference degrees and past preference degrees estimated by the state determination unit 135. Past preference degree 801 is, for example, the past preference degree of the first object 406 in FIGS. 4C and 4E. Past preference degree 801 is a preference degree estimated when a video content is played in the past, and has no value when the same video content is played for the first time.
[0095] 8A illustrates a case where the current preference level estimated by the state determination unit 135 is lower than the past preference level. When the current preference level 802 is lower than the past preference level 801, the current preference level 802 stored in the memory 107 is updated to the past preference level 801.
[0096] 8B illustrates an example in which the current preference level estimated by the state determination unit 135 is higher than the past preference level. The current preference level 803 is the current preference level estimated by the state determination unit 135 while the current video content is being played. If the current preference level 803 is higher than the past preference level 801, the past preference level 801 stored in the memory 107 is not updated.
[0097] 9A and 9B are diagrams illustrating a method for determining whether to update the preference update parameters in step S705 of FIG. 7A.
[0098] 9A and 9B show examples of update parameters for the current preference level and past preference levels estimated by the state determination unit 135. Update parameter 901 is, for example, an update parameter for the past preference level of the first object 406 in FIGS. 4C to 4E. Update parameter 901 for the past preference level is a parameter used to update the past preference level when playback of past video content ends and the content is next played.
[0099] 9A illustrates a case where the current preference level estimated by the state determination unit 135 is higher than the update parameter of the past preference level. When the current preference level 902 is higher than the update parameter 901 of the past preference level, the update parameter 901 of the past preference level stored in the memory 107 is updated to the current preference level 902.
[0100] 9B illustrates a case where the current preference level estimated by the state determination unit 135 is equal to or lower than the update parameter 901 of the past preference level. If the current preference level 903 is lower than the update parameter 901 of the past preference level, the update parameter 901 of the past preference level stored in the memory 107 is not updated.
[0101] The method for correcting the variation in the preference degree estimation results in steps S702 and S703 is not limited to the method of updating the current preference degree to a past preference degree, but may be another correction method. For example, a correction method may be used in which a plurality of past preference degrees for the same target object are stored, and a moving average or median value with respect to the current preference degree is calculated.
[0102] As described above, according to the second embodiment, it is possible to estimate a user's preference for an object that the user is gazing at while video content is being played, and to display advertising content according to the user's preference by taking into consideration past preferences estimated when the user previously viewed the same video content. In this way, by taking into consideration past preferences when playing back past video content, it is possible to correct variations in the estimation results of preference for the same video content, and further improve the advertising effectiveness when playing back video content.
[0103] Third Embodiment Next, a third embodiment will be described with reference to FIGS. 10A and 10B.
[0104] In embodiment 3, while video content is being played, the user's preference for an object that the user is gazing at is estimated, advertising content according to the user's preference is displayed, and the user's viewing information is sent to an external database 30.
[0105] In the third embodiment, the configuration and functions of the wearable device 10 are similar to those of the first embodiment shown in FIGS.
[0106] 10A and 10B are flowcharts illustrating control processing in embodiment 3. In Figures 10A and 10B, the same processes as those in Figures 3A and 3B are assigned the same step numbers, and descriptions thereof will be omitted.
[0107] In step S1001, the system control unit 101 transmits the user's viewing information to the database 30 via the communication unit 113, and then ends the process. The user's viewing information includes, for example, information such as the name of the video content, the user's preference for each object, and the frequency with which advertising content is set. Note that the timing for transmitting the user's viewing information to the external device is not limited to after the video content playback has ended, and may be other timing.
[0108] As described above, according to the third embodiment, during playback of video content, a user's preference for an object that the user is gazing at is estimated, and advertising content according to the user's preference is displayed, and the user's viewing information is transmitted to the external database 30. This makes it possible to improve advertising effectiveness during playback of video content. Furthermore, since the user's viewing information can be fed back to an external device that creates and provides advertising content, it is possible to encourage the creation and provision of advertising content with higher advertising effectiveness.
[0109] In each of the above-described embodiments, the advertising content to be displayed may be set by the user operating the GUI using the operation unit 108. In addition, the object class may be set so that multiple classes can be set, such as people (faces) and animals.
[0110] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.
[0111] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.
[0112] This application claims priority based on Japanese Patent Application No. 2023-81831, filed on May 17, 2023, the entire contents of which are incorporated herein by reference.
[0113] 10...wearable device, 101...controller, 120L...display device for left eye, 120R...display device for right eye, 130...eyeball detector, 135...state determination unit, 140...content setting unit
Claims
1. An electronic device comprising: a display means for displaying a first content; an acquisition means for acquiring eye information of a user viewing the first content; and a control means for setting a second content to be displayed during playback of the first content based on the eye information.
2. The electronic device described in claim 1, further comprising an estimation means for estimating the preference of a user viewing the first content based on the eyeball information, and wherein the control means sets the second content to be displayed during playback of the first content based on the user's preference.
3. The electronic device according to claim 2, wherein the estimation means estimates the user's preference level based on an object that the user is gazing at in relation to the first content.
4. The electronic device according to claim 3, wherein the estimation means estimates the user's preference level based on one or more objects on which the user is gazing within a range narrower than a predetermined range.
5. The electronic device described in claim 3 or 4, characterized in that the estimation means estimates the user's preference during playback of a specified section of the first content, stores the user's preference estimated based on a specified object in a memory means, and if the newly estimated current preference based on the specified object is higher than the previous preference stored in the memory means, updates the previous preference stored in the memory means to the current preference.
6. The electronic device described in any one of claims 3 to 5, characterized in that the control means compares the user's preference with a threshold value, and if there is no object whose user's preference is higher than the threshold value, sets a predetermined content as the second content, and if there is an object whose user's preference is higher than the threshold value, sets content related to that object as the second content.
7. The electronic device according to claim 6, characterized in that, when there are a plurality of objects whose preference level of the user is higher than the threshold value, the control means sets content relating to the plurality of objects as the second content.
8. The electronic device described in any one of claims 3 to 7, characterized in that the control means, when the first content has been viewed in the past, determines whether the current preference level estimated based on a specified object is higher than a past preference level estimated based on the specified object when the first content was viewed in the past, and if the current preference level is lower than the past preference level, updates the current preference level to the past preference level, and if the current preference level is higher than the past preference level and the past preference level is higher than the predetermined preference level, updates the past preference level to the predetermined preference level.
9. The electronic device according to any one of claims 1 to 8, characterized in that the control means transmits user viewing information regarding the first content to an external device when playback of the first content is terminated.
10. The electronic device described in any one of claims 3 to 8, characterized in that when the control means terminates playback of the first content, it transmits user viewing information regarding the first content to an external device, and the user viewing information includes information about the first content, the user's preference for each object, and the setting frequency of the second content.
11. An electronic device as described in any one of claims 2 to 8, characterized in that it has a setting means for setting a class of object for which the user's preference is to be estimated, the class including at least one of people (faces), animals, vehicles, food, and fashion.
12. The electronic device according to any one of claims 2 to 8, characterized in that the estimation of the user's preference degree by the estimation means can be set to be valid or invalid.
13. An electronic device as claimed in any one of claims 1 to 12, characterized in that the eye information includes at least one of the frequency of occurrence of microsaccades, the amplitude of microsaccades, gaze position information, saccade direction, saccade speed, pupil size, and change in pupil diameter.
14. The electronic device according to any one of claims 1 to 13, wherein the first content is video content and the second content is advertising content.
15. The electronic device according to any one of claims 1 to 14, characterized in that the electronic device is a head-mounted display.
16. A control method for an electronic device having a display means for displaying a first content, comprising: a step of acquiring eye information of a user viewing the first content; and a step of setting a second content to be displayed during playback of the first content based on the eye information.
17. A program for causing a computer to function as the electronic device according to any one of claims 1 to 15.
Citation Information
Patent Citations
Method and system for providing advertisement based on gaze of user
JP2015158946A
Processing device, processing method, and processing program
JP2022187859A
Information processing device, information processing method, and computer-readable recording medium
WO2020189254A1
