Electronic apparatus, control method, and program

Abstract

To display contents preferred by a user.SOLUTION: An electronic apparatus has: display means that displays first contents; acquisition means that acquires information on the eyeballs of a user who views the first contents; and control means that sets second contents to be displayed during playback of the first contents on the basis of the eyeball information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an electronic device that can be worn and used by a user. [Background technology]

[0002] Wearable devices such as head mounted displays (HMDs) are becoming more and more popular for personal viewing of video content. When viewing video content at home, advertising content may be displayed during playback of the video content, as in the case of viewing on a television receiver or tablet computer (PC). In this case, if advertising content is displayed unconditionally, users may feel uncomfortable viewing advertising content that they are not interested in, and the advertising effect may be reduced for users who are not interested in the advertising content.

[0003] Patent Document 1 describes a method for changing the frequency or time for displaying advertising content depending on the user's level of interest. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-40565 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, advertising content that is of low interest to the user is displayed infrequently and for a short period of time, which is not sufficient to increase advertising effectiveness for users who are watching video content on a wearable device such as an HMD.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to realize a technology for displaying content that a user prefers. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the electronic device of the present invention has a display means for displaying a first content, an acquisition means for acquiring eye information of a user viewing the first content, and a control means for setting a second content to be displayed during playback of the first content based on the eye information. Effect of the Invention

[0008] According to the present invention, it is possible to display content that is preferred by a user. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is an external view of the wearable device according to the embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of a wearable device according to an embodiment of the present invention. [Diagram 3] 5 is a flowchart illustrating a control process of the wearable device of the first embodiment. [Figure 4] 3 is an explanatory diagram of a preference degree estimation method according to the present embodiment. [Diagram 5] FIG. 4 is an explanatory diagram of a preference updating method according to the present embodiment. [Figure 6] FIG. 4 is an explanatory diagram of an advertisement content setting method according to the embodiment. [Figure 7] 10 is a flowchart illustrating a control process of the wearable device of the second embodiment. [Figure 8] FIG. 11 is an explanatory diagram of a preference updating method according to the second embodiment. [Figure 9] FIG. 11 is an explanatory diagram of a preference updating method according to the second embodiment. [Figure 10] 11 is a flowchart illustrating a control process of the wearable device of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] Hereinafter, an embodiment in which the electronic device of the present invention is applied to a head mounted display having a gaze detection function as a wearable device will be described in detail with reference to the attached drawings. Note that the wearable device is not limited to an HMD, and may be a smart device such as a smartphone, tablet computer, glasses-type device, or wristwatch-type device having a gaze detection function.

[0012] <Device Configuration> The configuration and functions of a wearable device 10 of this embodiment will be described with reference to Figs.

[0013] FIG. 1 is an external view of a wearable device 10 according to the present embodiment.

[0014] The wearable device 10 of this embodiment can be used by wearing it on the user's head so that a display shaped like goggles is placed over both eyes. The wearable device 10 has a display device 120L for the left eye and a display device 120R for the right eye. The display device 120L for the left eye displays image content to be viewed with the left eye of the user, and the display device 120R for the right eye displays image content to be viewed with the right eye of the user. The display device 120L for the left eye and the display device 120R for the right eye can also display image content such as VR (virtual reality) by a stereoscopic image using the parallax between the left and right eyes. The image content is a still image or a video. The wearable device 10 of this embodiment is a binocular type having a display device 120L for the left eye and a display device 120R for the right eye, but may be a monocular type having a display device 120L for the left eye or a display device 120R for the right eye. In this embodiment, the image content includes a video and an advertisement, and the video includes audio.

[0015] FIG. 2 is a block diagram illustrating an internal configuration of the wearable device of this embodiment.

[0016] The system control unit 101 includes an arithmetic processor such as a CPU or MPU that controls the entire wearable device 10, and executes programs stored in the nonvolatile memory 102 to realize the processing of each flowchart described below. The system memory 103 is a RAM or the like, and is also used as a work memory for expanding constants and variables for the operation of the system control unit 101, programs read from the nonvolatile memory 102, and the like. The system control unit 101 also controls the display of the left eye display device 120L and the right eye display device 120R. The system timer 104 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0017] The image processing unit 105 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the data from the memory control unit 106 .

[0018] The memory control unit 106 controls data exchange between the image processing unit 105 and the memory 107. Data output from the image processing unit 105 is written to the memory 107 via the memory control unit 106. The memory 107 stores display data of content such as videos and advertisements to be displayed on the display unit 122. The memory 107 has a storage capacity sufficient to store a predetermined number of still images and videos and audio for a predetermined period of time. The memory 107 also serves as a memory for displaying images (video memory). The memory 107 also stores information (user information) related to a user state such as the user's preferences determined by a state determination unit 135 described later. The system control unit 101 transmits the user information stored in the memory 107 to a content setting unit 140 described later, and transmits the information to a database 30 as an external device via a communication unit 113.

[0019] The operation unit 108 includes operation members such as various switches and buttons that receive various operations from a user and notify the system control unit 101. The operation unit 108 includes at least a power switch 109.

[0020] The power supply control unit 110 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, and detects whether a battery is attached, the type of battery, and the remaining battery power. The power supply control unit 110 also controls the DC-DC converter based on the detection results and instructions from the system control unit 101, and supplies the necessary voltage to each unit including the recording medium 200 for the necessary period.

[0021] The power supply unit 111 is composed of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li-ion battery, an AC adapter, or the like.

[0022] A recording medium interface (I / F) 112 is an interface with a recording medium 200 such as a memory card.

[0023] The communication unit 113 is communicably connected to an external device such as the database 30 via a wireless antenna or a wired cable, and transmits and receives images, audio, and other data. The communication unit 113 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 113 can transmit data recorded in the recording medium 200 to the external device, and can also receive necessary data from the external device. Note that the communication unit 113 is not limited to a wireless LAN, and may use a wireless communication module such as infrared communication, Bluetooth (registered trademark), Bluetooth (registered trademark) Low Energy, or Wireless USB, or a wired connection means such as a USB cable, HDMI (registered trademark), or IEEE1394.

[0024] The display device 120L for the left eye and the display device 120R for the right eye each include an eyepiece unit 121, a display unit 122, and an eyeball detection unit 130. The eyepiece unit 121 is an eyepiece unit of an eyepiece finder (a peer-in type finder), and a user can view content such as videos and advertisements displayed on the display unit 122 through the eyepiece unit 121. The display unit 122 is an electronic viewfinder (EVF) that displays content such as videos and advertisements.

[0025] The eyepiece detection unit 114 detects the approach (eye approach) and departure (eye movement) of the eye (object) to the eyepiece unit 121. The system control unit 101 determines whether or not the wearable device 10 is worn by the user, depending on the state detected by the eyepiece detection unit 114. If the system control unit 101 determines that the wearable device 10 is worn by the user, it turns on the power of the wearable device 10 and switches the display unit 122 to a display state. If the system control unit 101 determines that the wearable device 10 is not worn by the user, it turns off the power of the wearable device 10 and switches the display unit 122 to a non-display state.

[0026] When an object approaches, infrared light irradiated from a light-projecting section (not shown) of eyepiece detection section 114 is reflected and enters a light-receiving section (not shown) of the infrared proximity sensor. Depending on the amount of incident infrared light received by the infrared proximity sensor, it is possible to detect the approach of an object to eyepiece section 121 and determine how close the object is to eyepiece section 121 (eyepiece distance). When it detects the approach of an object to eyepiece section 121, it becomes possible for system control section 101 to start displaying on display section 122. This makes it possible to display display section 122 with as little delay as possible when the user looks into eyepiece section 121.

[0027] Furthermore, when the eye contact detection unit 114 detects an object approaching within a predetermined distance from the eye contact unit 121 from a non-eye contact state (non-approaching state), it determines that the eye contact has been detected, and transmits an eye contact detection notification to the system control unit 101. Furthermore, when the object that was detected as approaching from the eye contact state (approaching state) moves away from the eye contact state by a predetermined distance or more, it determines that the eye contact has been detected, and transmits an eye contact detection notification to the system control unit 101. The threshold value for detecting the eye contact and the threshold value for detecting the eye contact may be different, for example, by providing a hysteresis. Furthermore, after the eye contact is detected, the eye contact state is assumed to remain until the eye contact is detected. After the eye contact is detected, the non-eye contact state is assumed to remain until the eye contact is detected. As a result, the system control unit 101 performs display control of the display unit 122 according to the eye contact state or the eye contact state detected by the eye contact detection unit 114.

[0028] The eye proximity detection unit 114 is not limited to an infrared proximity sensor, and other sensors may be used as long as they can detect the approach of an eye or an object that can be regarded as an eye.

[0029] Eyeball detection unit 130 includes an infrared light-emitting element 131, an imaging lens 132, and an event sensor 133, and can detect information about the user's eyes (eyes) 20L, 20R looking into eyepiece unit 121 (hereinafter, eyeball information). Eyeball detection unit 130 can also detect the user's line of sight information with respect to the display screen of display unit 122. Eyeball detection unit 130 detects line of sight information such as the position and direction of the line of sight from the positional relationship between the light reflected by eyeballs (eyes) 20L, 20R of infrared light emitted from infrared light-emitting element 131 and the pupils of eyeballs (eyes) 20L, 20R.

[0030] The infrared light emitting element 131 is a diode that emits infrared light, and irradiates the infrared light toward the vicinity of the centers of the eyeballs (eyes) 20L, 20R of the user looking through the eyepiece unit 121. The infrared light irradiated from the infrared light emitting element 131 is reflected by the eyeballs (eyes) 20L, 20R, and the reflected infrared light reaches the event sensor 133 via the imaging lens 132. The imaging lens 132 has a function of reflecting only infrared light and transmitting visible light, and the reflected infrared light whose optical path has been changed forms an image on the imaging surface of the event sensor 133 via the imaging lens 132.

[0031] When the gaze detection function of the eyeball detection unit 130 is set to enabled, the system control unit 101 can determine the gaze input state of the user looking at the display unit 122 through the eyepiece unit 121 based on the gaze detection result of the eyeball detection unit 130, and executes processing according to the operation performed by the gaze input.

[0032] The event sensor 133 is an event-based vision sensor that detects a change in luminance of light incident on each pixel, and outputs information (coordinates, polarity, time) of the pixel where the luminance change is detected as event information asynchronously with other pixels. The event information output from the event sensor 133 includes, for example, the coordinates of the pixel where the luminance change (event) occurred, the polarity (positive or negative) of the luminance change, and time information that is the time when the event occurred. Compared to a frame-based synchronous sensor such as an existing image sensor, the event sensor 133 has eliminated the redundancy of the output information, and has characteristics of high-speed operation, high dynamic range, and low power consumption. Since the event information is outputted as pixel information asynchronously, in order to determine the relevance between the event information, various calculation processes must be performed on the results of accumulating event information generated within a predetermined time.

[0033] The event calculation unit 134 performs calculation processing to detect eyeball information from event information that is continuously and asynchronously output from the event sensor 133. For example, the event calculation unit 134 accumulates event information that occurs within a predetermined time, and processes the accumulated information as a set of data to determine the presence or absence of eyeball information. By changing the accumulation time of the event information, it is possible to detect a plurality of pieces of eyeball information that occur at different speeds. The eyeball information includes eyeball movements such as gaze position information, saccade information including the direction and speed of rapid eyeball movement (saccade), and microsaccade information including the occurrence frequency and amplitude of microsaccades. In addition, pupil information including the size of the pupil and changes in the pupil, and blink information such as the blink speed and number of times may be added. Note that the above-mentioned eyeball information is an example, and is not limited to these. The event calculation unit 134 may be configured to map the event information for the accumulation time as one frame of image data based on the event occurrence coordinates, and perform image processing. This makes it possible to obtain a plurality of pieces of eyeball information from the event information using a method that is conventionally used in frame-based image processing.

[0034] The state determination unit 135 determines the state of the user based on the eyeball information detected by the event calculation unit 134. For example, the gaze range or gaze degree (overhead degree) can be determined from the frequency and amplitude of microsaccades. Here, the gaze range is synonymous with the attention range or the focus range. The gaze degree is an index that is higher as the gaze range is narrower and lower as the gaze range is wider. The overhead degree is defined as the antonym of the gaze degree. The preference degree, such as face preference, is related to the speed of the microsaccades and the pupil diameter, and can be determined from both parameters. The state determination unit 135 can be configured by a neural network that receives parameters related to the microsaccades, blinks, pupils, and subject identification results as input, and outputs an index (hereinafter, user state information) that represents the state of the user, such as the gaze range or gaze degree (overhead degree as an antonym), preference degree, etc. Note that the state determination unit 135 is not limited to the above-mentioned configuration. Moreover, the eyeball information and the determination results used by state determination unit 135 are not limited to those described above.

[0035] The gaze input setting unit 136, under the control of the system control unit 101, can enable or disable the gaze detection function of the eyeball detection unit 130, and set parameters and detection conditions used by the event calculation unit 134 and the state determination unit 135. The enable or disable of the gaze detection function and the parameters and detection conditions of the event calculation unit 134 and the state determination unit 135 can be set, for example, by operating a GUI (Graphical User Interface) by the operation unit 108.

[0036] When the gaze detection function of the eyeball detection unit 130 is disabled, the system control unit 101 does not execute preference estimation by the state determination unit 135 because the gaze detection result of the eyeball detection unit 130 cannot be obtained.

[0037] The system control unit 101 can detect information about the position and size of objects (people (faces), animals, vehicles, etc.) included in the video content being played back on the display screen of the display unit 122. Furthermore, the system control unit 101 can determine an object that a user gazing at the display unit 122 through the eyepiece unit 121 is gazing at, based on information of the user's line of sight detected by the eyeball detection unit 130.

[0038] The content setting unit 140 changes the advertisement content to be displayed on the display unit 122 based on the user's preference. For example, the content setting unit 140 performs control to change the advertisement content based on whether the preference estimated by the state determination unit 135 is higher than a threshold value. Details of the control process will be described later with reference to Figs. 3 to 10.

[0039] [Embodiment 1] Next, the control process of the first embodiment will be described with reference to FIG. 3 to FIG.

[0040] In the first embodiment, a user's preference for an object that the user is gazing at while video content is being played back is estimated, and advertising content is displayed according to the user's preference.

[0041] FIG. 3 is a flowchart illustrating a control process according to the first embodiment.

[0042] The processing in Fig. 3 is realized by the system control unit 101 expanding a program stored in the non-volatile memory 102 into the system memory 103, executing the program, and controlling each component of the wearable device 10. The processing in Fig. 3 is started when the power supply of the wearable device 10 is turned on. The same applies to Figs. 7 and 10 described later.

[0043] In step S301, the system control unit 101 sets a classification or type (hereinafter, class) of an object for which the state determination unit 135 estimates the user's preference. The object class includes, for example, at least one of people (faces), animals, vehicles, foods, and fashion, and may be set by the user operating the GUI using the operation unit 108, or a class related to the video content to be played may be automatically set.

[0044] In step S302, system control unit 101 starts playing back video content.

[0045] In step S303, the system control unit 101 determines whether the state determination unit 135 has detected the user's gaze in a range narrower than the predetermined gaze range. If the system control unit 101 determines that the user's gaze has been detected in a range narrower than the predetermined gaze range, the process proceeds to step S303. If the system control unit 101 does not determine that the user's gaze has been detected in a range narrower than the predetermined gaze range, the process proceeds to step S308. A method for detecting the user's gaze range will be described later. Note that in step S303, the range in which the user gazes is limited to a range narrower than the predetermined gaze range as a condition for performing preference estimation in step S305, but this restriction need not be imposed.

[0046] In step S304, the system control unit 101 determines whether or not one or more target objects related to the class set in step S301 are present in the gaze range of the user detected by the state determination unit 135. If the system control unit 101 determines that a target object is present in the gaze range detected by the state determination unit 135, the process proceeds to step S305. If the system control unit 101 determines that a target object is not present in the gaze range detected by the state determination unit 135, the process proceeds to step S308.

[0047] In step S305, the system control unit 101 estimates the user's preference for the one or target objects determined in step S304, and stores the estimated preference in the memory 107. The user's preference for the target object is estimated, for example, from the amount of change in pupil diameter, which will be described in detail later. Note that the estimation of the preference is not limited to the method using the amount of change in pupil diameter, and the speed of microsaccades, the degree of gaze, the gaze duration, or other parameters may be used.

[0048] In step S306, the system control unit 101 determines whether the user's preference for the target object estimated in step S305 (hereinafter, the current preference) is higher than the user's preference for the same target object stored in the memory 107 (hereinafter, the previous preference). If the system control unit 101 determines that the current preference for the same target object is higher than the previous preference, the process proceeds to step S307. If the system control unit 101 determines that the current preference for the same target object is equal to or lower than the previous preference, the process proceeds to step S308.

[0049] In step S307, the system control unit 101 updates the previous preference level for the same target object stored in the memory 107 to the current preference level, and the process proceeds to step S308.

[0050] In step S308, system control unit 101 determines whether or not the playback of the video content of the predetermined section has ended. If system control unit 101 determines that the playback of the video content of the predetermined section has ended, the process proceeds to step S309, and if system control unit 101 determines that the playback of the video content of the predetermined section has not ended, the process returns to step S303.

[0051] In step S309, the system control unit 101 determines whether or not there is an object whose preference level is higher than a predetermined threshold 601 among the user's preference levels for all the target objects determined in step S304. If the system control unit 101 determines that there is an object whose preference level is higher than the predetermined threshold 601, the process proceeds to step S310, and if the system control unit 101 determines that there is no object whose preference level is higher than the predetermined threshold 601, the process proceeds to step S313. The predetermined threshold 601 will be described later.

[0052] In step S310, the system control unit 101 sets, as the display target, the advertising content related to the object for which the content setting unit 140 has a preference degree higher than the predetermined threshold value 601.

[0053] In steps S309 and S310, the predetermined threshold value 601 is used for the determination, but it is also possible to set the advertisement content related to the object with the highest preference as the display target without setting a threshold value.

[0054] In step S311, the system control unit 101 plays back the advertisement content set in step S310 or S313.

[0055] In step S312, system control unit 101 determines whether or not to end the playback of the video content. If system control unit 101 determines that the playback of the video content is to be ended, it ends the process, and if system control unit 101 determines that the playback of the video content is not to be ended, it proceeds to step S314.

[0056] In step S313, the system control unit 101 causes the content setting unit 171 to set the default advertising content as the display target.

[0057] In step S314, the system control unit 101 initializes the user's preference degrees for all target objects stored in the memory 107 by the memory control unit 106, and returns the process to step S303. Note that in step S314, the preference degrees stored in the memory 107 are initialized when the playback of the video content ends, but initialization is not necessary.

[0058] FIG. 4 is a diagram for explaining a method of estimating the preference level in step S305 of FIG.

[0059] 4(a) and 4(b) show microsaccade information, which is one type of eyeball information acquired by the event calculation unit 134. FIG.

[0060] 4(c) to 4(e) are diagrams illustrating examples of the user's gaze state on the display screen of display unit 122. In Fig. 4(c), for example, the class of the target object for which the preference level is to be estimated is a person (face), and there are a predetermined gaze range 405, a first object 406, and a second object 407.

[0061] FIG. 4(a) illustrates an example of a microsaccade waveform when the gaze range is relatively wide. The vertical axis represents the position (angle) of the pupil center in the eyeball-centered coordinate system, and the horizontal axis represents time. Area 401 shows the change in eyeball position when a microsaccade occurs. The wider the gaze range, the larger the amplitude of eye movement tends to be and the higher the oscillatory nature (lower the attenuation rate). In addition, there is a tendency for microsaccades to occur more frequently in a given period of time.

[0062] FIG. 4(b) illustrates an example of a microsaccade waveform when the gaze range is relatively narrow. 403 indicates the change in eye position when a microsaccade occurs. The narrower the gaze range, the smaller the amplitude of eye movement and the lower the oscillatory tendency (the higher the attenuation rate). In addition, there is a tendency for the frequency of microsaccade occurrence to be low in period 404, which is the same as period 402 in FIG. 4(a). As described above, the size of the gaze range can be estimated using the features of eye movement as parameters.

[0063] Fig. 4(c) illustrates an example of a user's gaze state when the gaze range of Fig. 4(a) is relatively wide. Region 408 is the user's gaze range. In the example of Fig. 4(c), for example, the user is looking down on the first object 406 and the scenery, and the user's gaze range 408 is larger than the predetermined gaze range 405. In this case, preference estimation is not performed.

[0064] Fig. 4(d) illustrates an example of a user's gaze state when the gaze range in Fig. 4(b) is relatively narrow. Region 409 is the user's gaze range. In the example of Fig. 4(d), the user is gazing at a first object 406, and the user's gaze range 409 is narrower than the predetermined gaze range 405. In this case, a preference degree is estimated.

[0065] Fig. 4(e) illustrates a state of gaze of a user when the gaze range of Fig. 4(b) is relatively narrow. Region 410 is the gaze range of the user. In the example of Fig. 4(e), the user is gazing at a part of the landscape, and the gaze range 410 of the user is narrower than the predetermined gaze range 405, but the user is not gazing at a person (face), which is the target object for estimating the preference level, so the preference level is not estimated.

[0066] FIG. 5 is a diagram for explaining a method for determining whether or not to update the preference level in step S306 in FIG.

[0067] 5(a) and 5(b) show an example of a change in pupil diameter, which is one piece of eyeball information acquired by the event calculation section 134. FIG.

[0068] 5(c)-(d) show examples of the current preference degree and the previous preference degree estimated by the state determination unit 135. The preference degrees 503 and 504 are the preference degrees of the first object 406, for example.

[0069] FIG. 5(a) illustrates an example of the change in pupil diameter when the preference is relatively high. The vertical axis represents pupil diameter, and the horizontal axis represents time. Waveform 501 illustrates an example of the change in pupil diameter when the user views a favorite object. The higher the user's preference, i.e., the preference level, the greater the change in pupil diameter tends to be.

[0070] FIG. 5(b) illustrates an example of the change in pupil diameter over time when the level of preference is relatively low. The vertical axis represents the size of the pupil diameter, and the horizontal axis represents time. Waveform 502 illustrates an example of the change in pupil diameter when the user views an object that is not to their liking. The less the user likes an object, i.e., the lower the level of preference, the smaller the change in pupil diameter tends to be. In this way, the level of preference can be estimated by using the amount of change in pupil diameter as a parameter.

[0071] 5(c) illustrates a case where the preference level in FIG. 5(a) is relatively high and the current preference level estimated by the state determination unit 135 is higher than the previous preference level. When the current preference level 504 is higher than the previous preference level 503, the previous preference level 503 stored in the memory 107 is updated to the current preference level 504.

[0072] Fig. 5(d) illustrates a case where the preference level in Fig. 5(b) is relatively low and the current preference level estimated by the state determination unit 135 is equal to or lower than the previous preference level. When the current preference level 505 is lower than the previous preference level 503, the previous preference level 503 stored in the memory 107 is not updated to the current preference level 505.

[0073] FIG. 6 is a diagram for explaining a method for setting advertisement content in steps S310 and S313 in FIG.

[0074] For example, the class of the target object for which the preference level is to be estimated is a person (face).

[0075] A predetermined threshold 601 is set for the user's preference for each target object in order to set advertisement content to be displayed. If there is an object for which the user's preference is higher than the predetermined threshold 601, advertisement content for that object is set as a display target. Preference levels 602 to 606 are the preference levels for objects A to E. In the example of FIG. 6, for ease of explanation, there are five types of target objects, but the present invention is not limited to this and may be more than five or less than five.

[0076] Fig. 6(a) illustrates an example of preference when advertising content related to one object is set as a display target. In the example of Fig. 6(a), the preference 606 of object E is higher than a predetermined threshold 601. In this case, advertising content related to object E is set as a display target. For example, advertising content in which object E appears is set as a display target.

[0077] 6(b) illustrates an example of preference levels when advertising content related to multiple objects is set as the display target. The preference level 602 of object A and the preference level 606 of object E are higher than a predetermined threshold value 601. In this case, advertising content related to objects A and E is set as the display target. For example, advertising content in which objects A and E appear, or in which object A or object E appears, is set as the display target.

[0078] 6(c) illustrates an example of preference levels when default advertising content is set as the display target. All of preference levels 602 to 606 are lower than a predetermined threshold value 601. In this case, the default advertising content is set as the display target.

[0079] Although an example has been given in which an object with a user preference level higher than a predetermined threshold 601 appears as the advertising content to be displayed, other advertising content related to the object may also be set, or the advertising content may be changed based on the setting of the object's class.

[0080] As described above, according to the first embodiment, the user's preference for an object that the user is gazing at during playback of video content is estimated, and advertising content according to the user's preference is displayed, thereby improving the advertising effectiveness during playback of video content.

[0081] [Embodiment 2] Next, a second embodiment will be described with reference to FIGS.

[0082] In the second embodiment, while video content is being played, the user's preference for an object that the user is gazing at is estimated, and advertising content is displayed according to the user's preference, taking into consideration the user's past preference estimated when the user viewed the same video content in the past.

[0083] In the second embodiment, the configuration and functions of the wearable device 10 are similar to those of the first embodiment shown in FIGS.

[0084] Fig. 7 is a flowchart illustrating a control process of embodiment 2. In Fig. 7, the same processes as those in Fig. 3 are given the same step numbers and descriptions thereof will be omitted.

[0085] In step S701, the system control unit 101 determines whether the video content that has started to be played in step S302 has been viewed in the past. If the system control unit 101 determines that the video content has been viewed in the past, the process proceeds to step S702. If the system control unit 101 determines that the video content has not been viewed in the past, the process proceeds to step S306.

[0086] In step S702, the system control unit 101 determines whether the user's preference for the target object estimated in step S305 (current preference) is lower than the user's preference for the target object estimated when the video content was viewed in the past (hereinafter, past preference). The past preference is stored in the non-volatile memory 102 when the past video content is played back, and is loaded into the memory 107 when the same content is played back this time. If the system control unit 101 determines that the current preference is lower than the past preference, the process proceeds to step S703, and if the system control unit 101 determines that the current preference is higher than the past preference, the process proceeds to step S704.

[0087] In step S703, the system control unit 101 updates the current preference level stored in the memory 107 to the past preference level, and the process proceeds to step S306. A method for determining whether or not to update the preference level will be described later.

[0088] In step S704, the system control unit 101 determines whether the current preference level is higher than the update parameter of the past preference level. The update parameter of the past preference level is stored in the non-volatile memory 102 when the past video content is played back, and is loaded into the memory 107 when the same content is played back this time. If the system control unit 101 determines that the current preference level is higher than the update parameter of the past preference level, the process proceeds to step S705, and if the system control unit 101 determines that the current preference level is equal to or lower than the update parameter of the past preference level, the process proceeds to step S306.

[0089] In step S705, the system control unit 101 updates the update parameters of the past preference level stored in the memory 107 to the current preference level, and the process proceeds to step S306.

[0090] In step S706, the system control unit 101 updates the past preference levels stored in the memory 107 to the update parameters of the past preference levels updated in step S705, and stores the update parameters in the non-volatile memory 102. The method of updating the update parameters of the past preference levels will be described later.

[0091] In step S707, the system control unit 101 causes the memory control unit 106 to initialize update parameters for the past preference levels stored in the memory 107, and ends the process.

[0092] FIG. 8 is a diagram for explaining a method for determining whether or not to update the preference level in step S702 in FIG.

[0093] 8(a)-(b) show examples of the current preference degree and past preference degrees estimated by state determination unit 135. Past preference degree 801 is, for example, the past preference degree of first object 406 in Fig. 4. Past preference degree 801 is a preference degree estimated when a video content was played in the past, and has no value when the same video content is played for the first time.

[0094] 8(a) illustrates a case where the current preference level estimated by the state determination unit 135 is lower than the past preference level. When the current preference level 802 is lower than the past preference level 801, the current preference level 802 stored in the memory 107 is updated to the past preference level 801.

[0095] 8(b) illustrates a case where the current preference level estimated by the state determination unit 135 is higher than the past preference level. The current preference level 803 is the current preference level estimated by the state determination unit 135 during playback of the current video content. If the current preference level 803 is higher than the past preference level 801, the past preference level 801 stored in the memory 107 is not updated.

[0096] FIG. 9 is a diagram for explaining a method of determining whether to update the update parameter of the preference level in step S705 of FIG.

[0097] 9(a)-(b) show examples of update parameters of the current preference degree and the past preference degree estimated by the state determination unit 135. Update parameter 901 is, for example, an update parameter of the past preference degree of the first object 406 in Fig. 4. Update parameter 901 of the past preference degree is a parameter used for updating the past preference degree when the playback of the past video content ends and the content is played back next time.

[0098] 9(a) illustrates a case where the current preference level estimated by the state determination unit 135 is higher than the update parameter of the past preference level. When the current preference level 902 is higher than the update parameter 901 of the past preference level, the update parameter 901 of the past preference level stored in the memory 107 is updated to the current preference level 902.

[0099] 9(b) illustrates a case where the current preference level estimated by the state determination unit 135 is equal to or lower than the update parameter of the past preference level. When the current preference level 903 is lower than the update parameter 901 of the past preference level, the update parameter 901 of the past preference level stored in the memory 107 is not updated.

[0100] The method of correcting the variation in the estimation result of the preference level in steps S702 and S703 is not limited to the method of updating the current preference level to the past preference level, and other correction methods may be used. For example, a correction method may be used in which a plurality of past preference levels for the same target object are stored, and a moving average value or a median value with respect to the current preference level is calculated.

[0101] As described above, according to the second embodiment, during playback of video content, the user's preference for an object that the user is gazing at is estimated, and advertising content according to the user's preference is displayed in consideration of the past preference estimated when the user viewed the same video content in the past. In this way, by taking into consideration the past preference when playing back the past video content, the variation in the estimation result of the preference for the same video content can be corrected, and the advertising effect during playback of the video content can be further improved.

[0102] [Embodiment 3] Next, a third embodiment will be described with reference to FIG.

[0103] In the third embodiment, during playback of video content, a user's preference for an object that the user is gazing at is estimated, and advertising content according to the user's preference is displayed, while the user's viewing information is transmitted to an external database 30.

[0104] In the third embodiment, the configuration and functions of the wearable device 10 are similar to those of the first embodiment shown in FIGS.

[0105] Fig. 10 is a flowchart illustrating a control process of embodiment 3. In Fig. 10, the same processes as those in Fig. 3 are given the same step numbers and descriptions thereof will be omitted.

[0106] In step S1001, system control unit 101 transmits the user viewing information to database 30 via communication unit 113, and ends the process. The user viewing information includes, for example, information such as the name of the video content, the user's preference for each object, and the frequency of setting advertisement content. Note that the timing for transmitting the user viewing information to the external device is not limited to after the end of playback of the video content, and may be other timing.

[0107] As described above, according to the third embodiment, during playback of video content, the user's preference for an object that the user is gazing at is estimated, and advertising content according to the user's preference is displayed, and the user's viewing information is transmitted to the external database 30. This makes it possible to improve the advertising effectiveness when playing back video content. In addition, since the user's viewing information can be fed back to an external device that creates and supplies advertising content, it is possible to encourage the creation and supply of advertising content with higher advertising effectiveness.

[0108] In each of the above-described embodiments, the advertising content to be displayed may be set by the user operating the GUI using the operation unit 108. In addition, the object class may be set so that multiple classes, such as a person (face) and an animal, can be set.

[0109] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0110] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0111] The disclosure of this specification includes the following electronic device, control method, and program. [Configuration 1] An electronic device, A display means for displaying the first content; An acquisition means for acquiring eyeball information of a user viewing the first content; and a control means for setting a second content to be displayed during playback of the first content based on the eyeball information. [Configuration 2] an estimation means for estimating a preference of a user viewing the first content based on the eyeball information; 2. The electronic device according to configuration 1, wherein the control means sets the second content to be displayed during playback of the first content based on the user's preference. [Configuration 3] 3. The electronic device according to configuration 2, wherein the estimation means estimates the user's preference level based on an object that the user is gazing at in the first content. [Configuration 4] The electronic device according to configuration 3, wherein the estimation means estimates the user's preference level based on one or more objects that the user is gazing at within a range narrower than a predetermined range. [Configuration 5] the estimation means estimates a preference degree of the user during playback of a predetermined section of the first content; storing the user's preference estimated based on the predetermined object in a storage means; The electronic device described in configuration 3 or 4, characterized in that if a new preference level estimated based on the specified object is higher than a previous preference level stored in the storage means, the previous preference level stored in the storage means is updated to the current preference level. [Configuration 6] the control means compares the user's preference with a threshold value, and if there is no object having a preference of the user higher than the threshold value, sets a predetermined content as the second content; 6. The electronic device according to any one of configurations 3 to 5, wherein if there is an object whose preference level of the user is higher than the threshold, content related to that object is set as the second content. [Configuration 7] The electronic device according to configuration 6, wherein the control means, when there are multiple objects whose preference level of the user is higher than the threshold value, sets content related to the multiple objects as the second content. [Configuration 8] the control means, when the first content has been viewed in the past, determines whether or not a current preference degree estimated based on a predetermined object is higher than a past preference degree estimated based on the predetermined object when the first content was viewed in the past; If the current preference level is lower than the past preference level, the current preference level is updated to the past preference level; The electronic device described in any one of configurations 3 to 7, characterized in that if the current preference level is higher than the past preference level, and if the past preference level is higher than a predetermined preference level, the past preference level is updated to the predetermined preference level. [Configuration 9] The electronic device according to any one of configurations 1 to 8, wherein the control means transmits user viewing information regarding the first content to an external device when playback of the first content is terminated. [Configuration 10] When terminating the reproduction of the first content, the control means transmits user viewing information regarding the first content to an external device; 9. The electronic device of any one of configurations 3 to 8, wherein the user viewing information includes information of the first content, the user's preference for each of the objects, and a setting frequency of the second content. [Configuration 11] a setting means for setting a class of an object for which a preference of the user is to be estimated; 9. The electronic device according to any one of configurations 2 to 8, wherein the classes include at least one of people (faces), animals, vehicles, food, and fashion. [Configuration 12] 9. The electronic device according to any one of configurations 2 to 8, wherein the estimation of the user's preference degree by the estimation means can be set to be valid or invalid. [Configuration 13] 13. The electronic device of any one of configurations 1 to 12, wherein the eye information includes at least one of the frequency of occurrence of microsaccades, the amplitude of microsaccades, gaze position information, saccade direction, saccade speed, pupil size, and change in pupil diameter. [Configuration 14] 14. The electronic device according to any one of configurations 1 to 13, wherein the first content is video content, and the second content is advertising content. [Configuration 15] 15. The electronic device according to any one of configurations 1 to 14, wherein the electronic device is a head mounted display. [Configuration 16] A method for controlling an electronic device having a display means for displaying a first content, comprising: acquiring eyeball information of a user viewing the first content; and setting a second content to be displayed during playback of the first content based on the eyeball information. [Configuration 17] A program for causing a computer to function as an electronic device according to any one of configurations 1 to 15. [Explanation of symbols]

[0112] 10: wearable device, 101: control unit, 120L: left eye display device, 120R: right eye display device, 130: eyeball detection unit, 135: state determination unit, 140: content setting unit

Claims

1. It is an electronic device, A display means for displaying a first type of content different from advertising content, An acquisition means for acquiring eyeball information of a user who is visually viewing the first content, An electronic device characterized by having control means for setting a second content to be displayed during playback of the first content based on the eyeball information.

2. The system includes estimation means for estimating the degree of preference of a user who is visually viewing the first content based on the aforementioned eyeball information, The electronic device according to claim 1, characterized in that the control means sets the second content to be displayed during playback of the first content based on the user's preference.

3. The electronic device according to claim 2, characterized in that the estimation means estimates the user's preference based on the object the user is focusing on with respect to the first content.

4. The electronic device according to claim 3, characterized in that the estimation means estimates the user's preference based on one or more objects that the user is looking at within a range narrower than a predetermined range.

5. The estimation means estimates the user's preference level during playback of the first content for a predetermined section. The user's preference level, estimated based on a predetermined object, is stored in a memory device. The electronic device according to claim 3, characterized in that if the newly estimated preference level based on the predetermined object is higher than the previous preference level stored in the storage means, the previous preference level stored in the storage means is updated to the current preference level.

6. The control means compares the user's preference level with a threshold, and if there are no objects whose user preference level is higher than the threshold, it sets a predetermined content as the second content. The electronic device according to claim 3, characterized in that if there is an object whose user preference level is higher than the threshold, the content relating to that object is set as the second content.

7. The electronic device according to claim 6, characterized in that, if there are multiple objects whose user preference level is higher than the threshold, the control means sets the content relating to the multiple objects as the second content.

8. The control means, if the first content has been viewed in the past, determines whether the current preference level estimated based on a predetermined object is higher than the past preference level estimated based on the predetermined object when the first content was viewed in the past. If the current preference level is lower than the past preference level, the current preference level is updated to match the past preference level. The electronic device according to claim 3, characterized in that if the current preference level is higher than the past preference level, or if the past preference level is higher than a predetermined preference level, the past preference level is updated to the predetermined preference level.

9. The electronic device according to claim 1, characterized in that when the control means terminates playback of the first content, it transmits user viewing information regarding the first content to an external device.

10. When the control means terminates playback of the first content, it transmits the user's viewing information regarding the first content to an external device. The electronic device according to claim 3, characterized in that the user's viewing information includes information on the first content, the user's preference for each object, and the frequency of setting the second content.

11. The system has setting means for setting the class of the object to be used to estimate the user's preference, The electronic device according to claim 2, characterized in that the class includes at least one of a person (face), an animal, a vehicle, food, and fashion.

12. The electronic device according to claim 2, characterized in that the estimation of the user's preference by the estimation means can be set to be enabled or disabled.

13. The electronic device according to claim 1, characterized in that the eyeball information includes at least one of the frequency of microsaccade occurrence, the amplitude of microsaccades, gaze position information, the direction of saccades, the speed of saccades, the size of the pupil, and the change in pupil diameter.

14. The electronic device according to claim 1, characterized in that the first content is video content and the second content is advertising content.

15. The electronic device according to claim 1, characterized in that the electronic device is a head-mounted display.

16. A control method for an electronic device having a display means for displaying a first content different from advertising content, The steps include obtaining eye information of a user who is visually viewing the first content, A control method characterized by comprising the step of setting a second content to be displayed during playback of the first content based on the eyeball information.

17. A program for causing a computer to function as an electronic device according to any one of claims 1 to 15.