Information processing device

Abstract

To provide a technology capable of detecting a subject whom a user is interested in for a short period of time with high accuracy.SOLUTION: According to the present invention, an information processing device includes acquisition means for acquiring a line of sight information about a line of sight of a user, count means for counting the number of times when the line of sight enters an area of the subject on the basis of the line of sight information, and determination means for determining that the user is interested in the subject in the case that the number of times is more than the number of threshold times.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In recent years, devices that capture and display images in real time to expand the information that users can visually obtain have become widespread. Some head-mounted displays (HMDs) for experiencing virtual reality (VR) and augmented reality (AR) acquire and process the user's gaze information.

[0003] There is also a method for detecting a subject of interest (subject of interest) based on gaze information. There are two main methods for detecting a subject of interest. The first is a method for detecting a subject at which the user's gaze is directed as the subject of interest. The second is a method for detecting a subject of interest based on the correlation between the user's gaze position and the position of the subject. As the second method, for example, Patent Document 1 discloses a method for detecting a subject of interest based on the correlation between the user's gaze position and an area with high visibility. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2012 / 105196 Summary of the Invention [Problem to be solved by the invention]

[0005] The first method detects all subjects that the user's gaze is directed at, regardless of whether the subject is of interest to the user, and therefore cannot detect subjects of interest with high accuracy. The second method detects a subject as a subject of interest if the gaze position and the subject position match for a period longer than a threshold time. In order to detect a subject of interest with high accuracy, the threshold time must be increased, and it is not possible to detect a subject of interest with high accuracy in a short period of time. Furthermore, the technology disclosed in Patent Document 1 cannot detect a subject of interest in an area with low visibility.

[0006] An object of the present invention is to provide a technique that enables a subject of interest to a user to be detected with high accuracy in a short time. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having an acquisition means for acquiring gaze information regarding a user's gaze, a counting means for counting the number of times the gaze enters the area of ​​a subject based on the gaze information, and a determination means for determining that the user is interested in the subject if the number of times is greater than a threshold number. [Effects of the Invention]

[0008] According to the present invention, a subject in which a user is interested can be detected with high accuracy in a short time. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing the configuration of an HMD. [Figure 2] FIG. 2 is a schematic diagram showing a state in which an HMD is worn by a user. [Figure 3] 5 is a flowchart showing the operation of the HMD according to the first embodiment. [Figure 4] FIG. 10 is an explanatory diagram of a plurality of specific regions. [Figure 5] 10 is a flowchart showing the operation of an HMD according to a second embodiment. [Figure 6] FIG. 10 is an image diagram of the line of sight. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] (First embodiment) <Configuration> 1 is a block diagram showing an example of the configuration of a head-mounted display (HMD) 100. The HMD 100 is an example of an information processing device. The HMD 100 includes an imaging and display unit 250R for the right eye and an imaging and display unit 250L for the left eye. First, the imaging and display units 250R and 250L will be described.

[0012] The imaging display unit 250R includes an image imaging unit 200R for the right eye, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, the eyepiece 116, and a line-of-sight detection unit 240. Similarly, the imaging display unit 250L includes an image imaging unit 200L for the left eye, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, the eyepiece 116, and a line-of-sight detection unit 240. Each of the image imaging units 200R and 200L includes an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a shutter 210, and an imaging unit 211.

[0013] The lens 202 is made up of multiple lenses, but for simplicity's sake, only one lens is shown in Figure 1. The system control unit 218 communicates with the lens system control circuit 205 and controls the aperture 201 via the aperture drive circuit 203. The system control unit 218 also focuses on the target by displacing the focus lens included in the lens 202 via the AF drive circuit 204.

[0014] The shutter 210 is a focal plane shutter that can freely control the exposure time of the imaging unit 211 under the control of the system control unit 218. The imaging unit 211 is an imaging element (imaging sensor, image sensor) formed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 211 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 218.

[0015] The A / D converter 212 converts the analog signal output from the imaging unit 211 into a digital signal. The image processing unit 214 performs predetermined processing (such as pixel interpolation, resizing such as reduction, and color conversion) on data from the A / D converter 212 or data from a memory control unit 213 (described later). The image processing unit 214 also performs predetermined arithmetic processing using the captured image data, and the system control unit 218 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 214. This allows TTL (through-the-lens) type AF (autofocus) processing and AE (autoexposure) processing to be performed. The image processing unit 214 also performs predetermined arithmetic processing using the captured image data, and TTL type AWB (auto white balance) processing based on the arithmetic results obtained.

[0016] The memory control unit 213 controls the transmission and reception of data between the A / D converter 212, image processing unit 214, memory 215, and D / A converter 216. Image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and memory control unit 213, or via the memory control unit 213 without going through the image processing unit 214. The memory 215 stores image data obtained by the imaging unit 211 and converted into digital data by the A / D converter 212, and image data to be displayed on the EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 215 also serves as a memory for displaying images (video memory).

[0017] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the EVF 217. The display image data written in the memory 215 is displayed on the EVF 217 via the D / A converter 216.

[0018] The EVF 217 displays an image on a display such as an LCD or organic EL display in response to a signal from the memory control unit 213. Image data stored in the memory 215 is sequentially transferred to the EVF 217 for display, thereby enabling a through display of the captured image. The user can view the image displayed on the EVF 217 via the eyepiece unit 116. Through display is the same as what is known as a live view display in a typical digital camera, and in through display, the captured image is displayed with almost no delay. By viewing the through display image, the user can indirectly view the real space.

[0019] The gaze detection unit 240 detects the user's gaze (gaze position, gaze direction) and outputs the detection result to the system control unit 218. The system control unit 218 acquires gaze information regarding the user's gaze based on the detection result input from the gaze detection unit 240, and can execute various controls according to the gaze information.

[0020] The HMD 100 also includes a system control unit 218 , a system memory 219 , a nonvolatile memory 220 , a system timer 221 , a communication unit 222 , an orientation detection unit 223 , and an eyepiece detection unit 118 .

[0021] The system control unit 218 is a control unit made up of at least one processor or circuit, and controls the entire HMD 100. The system control unit 218 realizes each process described below by executing programs recorded in the nonvolatile memory 220. The system memory 219 is, for example, a RAM, and the system control unit 218 loads constants and variables for operation of the system control unit 218, programs read from the nonvolatile memory 220, and the like into the system memory 219. The system control unit 218 also controls the memory 215, the EVF 217, and the like to perform display control.

[0022] The system timer 221 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0023] The communication unit 222 transmits and receives video signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 222 can also transmit and receive data to and from an external database server. The communication unit 222 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 222 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy.

[0024] The orientation detection unit 223 detects the orientation of the HMD 100 with respect to the direction of gravity. The system control unit 218 can add orientation information corresponding to the orientation detected by the orientation detection unit 223 to the image file of the image captured by the imaging unit 211, or rotate and record the image. The orientation detection unit 223 can use, for example, an acceleration sensor or a gyro sensor. The orientation detection unit 223 can be used to detect the movement of the HMD 100 (pan, tilt, roll, whether or not it is stationary, etc.).

[0025] The eye proximity detection unit 118 is a sensor for detecting whether or not the user is wearing the HMD 100. The system control unit 218 controls the HMD 100 in accordance with the state detected by the eye proximity detection unit 118. The D100 can be switched on (powered on) and off (powered off). The eyepiece detection unit 118 may be configured to detect the approach of an object to the eyepiece unit 116, for example, using an infrared proximity sensor. When an object approaches, infrared light is emitted from a light-emitting unit (not shown) of the eyepiece detection unit 118 and reflected and received by a light-receiving unit (not shown) of the infrared proximity sensor. The amount of received infrared light can also be used to determine the distance the object has approached from the eyepiece unit 116. Note that the infrared proximity sensor is just one example, and other sensors, such as a capacitance sensor, may also be used for the eyepiece detection unit 118.

[0026] The HMD 100 also includes an outside-finder display drive circuit 224 , an outside-finder display 107 , a power supply control unit 225 , a power supply unit 226 , a recording medium I / F 227 , and an operation unit 229 .

[0027] Various setting values ​​such as shutter speed and aperture are displayed on the viewfinder display unit 107 via the viewfinder display unit drive circuit 224 .

[0028] The power supply control unit 225 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 225 also controls the DC-DC converter based on the detection results and instructions from the system control unit 218, and supplies the required voltage for the required period to each unit of the HMD 100. The power supply unit 226 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or lithium-ion batteries, an AC adapter, etc.

[0029] The recording medium I / F 227 is an interface with a recording medium 228 such as a memory card or a hard disk. The recording medium 228 is a recording medium such as a memory card for recording captured images, and is configured from a semiconductor memory, a magnetic disk, etc. The recording medium 228 may be removable or may be built-in.

[0030] The operation unit 229 is a variety of operation members that serve as an input unit for accepting operations from the user (user operations). The operation unit 229 includes the shutter button 101, the power switch 102, the mode selector switch 103, and other operation members 230. The other operation members 230 include an electronic dial, a direction key, and a menu button.

[0031] The shutter button 101 includes a first shutter switch 121 and a second shutter switch 122. The first shutter switch 121 is turned on when the shutter button 101 is pressed halfway (a shooting preparation command) and generates a first shutter switch signal SW1. The system control unit 218 starts shooting preparation processes such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing in response to the first shutter switch signal SW1. The second shutter switch 122 is turned on when the shutter button 101 is pressed fully (a shooting command) and generates a second shutter switch signal SW2. The system control unit 218 starts a series of shooting processes in response to the second shutter switch signal SW2, from reading out a signal from the imaging unit 211 to writing the captured image to the recording medium 228 as an image file.

[0032] The mode changeover switch 103 is an operating member for switching the operation mode of the system control unit 218 to one of a shooting / display mode, a playback mode, an AR display mode, etc. The user can directly switch to one of the above-mentioned shooting modes using the mode changeover switch 103. Alternatively, the user can use the mode changeover switch 103 to once switch to a list screen of operation modes, and then use the operation unit 229 to selectively switch to one of the displayed modes.

[0033] Fig. 2 is a schematic diagram showing an example of a state in which a user wears the HMD 100. Note that, among the HMD 100 shown in Fig. 2, the same components as those described in Fig. 1 are given the same reference numerals and description thereof will be omitted.

[0034] 2 illustrates the image capturing unit 200R, lens 202R, image capturing unit 211R, EVF 217R, rotation adjuster 231R, and gaze detection unit 240R for the right eye, which are components of the HMD 100. Also illustrated in FIG. 2 are the image capturing unit 200L, lens 202L, image capturing unit 211L, EVF 217L, rotation adjuster 231L, and gaze detection unit 240L for the left eye, which are components of the HMD 100. The rotation adjuster 231R adjusts the rotation angle of the image capturing unit 200R around the Z axis (Yaw direction; parallax direction). The rotation adjuster 231L adjusts the rotation angle of the image capturing unit 200L around the Z axis (Yaw direction; parallax direction).

[0035] <Method for detecting subjects of interest using gaze information> In fields such as virtual reality (VR) and augmented reality (AR), there is a demand for technology that uses a user's gaze information to detect subjects of interest (subjects of interest). If the information users want can be provided appropriately based on the results of subject of interest detection, users will be able to enjoy VR and AR spaces even more. In VR and AR spaces, it is expected that users will move around and look at an unspecified number of subjects. Therefore, technology that can not only detect subjects of interest with high accuracy but also detect them in a short time is required.

[0036] There are two main methods for detecting a subject of interest using gaze information. The first method detects the subject that the gaze is directed at as the subject of interest. The second method detects the subject of interest based on the correlation between the user's gaze position and the position of the subject.

[0037] In a method of detecting a subject that is gazed at as a subject of interest, all subjects that the gaze is directed at are detected, regardless of whether the user is interested in them. As a result, the accuracy of detecting subjects of interest is low. In order to detect subjects of interest with high accuracy, processing is required to detect specific relationships from a large amount of noise data, making it difficult to detect subjects of interest with high accuracy in a short period of time.

[0038] In a method for detecting a subject of interest based on the correlation between the user's gaze position and the subject's position, if the gaze position and the subject's position match for a period longer than a threshold time, the subject is detected as a subject of interest. This method assumes that if a user is interested in a subject, they will follow it with their eyes for a certain period of time. In order to detect a subject of interest with high accuracy using this method, a long threshold time must be set, resulting in a trade-off between the detection time and the detection accuracy. In other words, a subject of interest can be detected with high accuracy if it takes a long time, but it is difficult to detect a subject of interest with high accuracy in a short time.

[0039] In addition, there are cases where a subject happens to be in the direction of the user's gaze. For example, when the user is looking at something unintentionally, or when the subject is placed in the user's direction of travel, etc. In this case, the subject in the direction of the user's gaze may be erroneously detected as a subject of interest, even if the user is not interested in it.

[0040] When people are interested in a subject, they tend to look at multiple parts of the same subject, and they often look at it repeatedly while comparing it with other subjects. Therefore, by extracting these gaze transitions, it is possible to detect subjects that interest the user in a short amount of time with high accuracy.

[0041] 3 is a flowchart showing an example of the operation of the HMD 100 according to the first embodiment. This operation is realized by the system control unit 218 expanding a program recorded in the nonvolatile memory 220 into the system memory 219 and executing it. For example, when the power of the HMD 100 is turned on, the operation of FIG. 3 starts.

[0042] 3 illustrates an example in which the present invention is applied to a video see-through display (video see-through HMD). A video see-through display displays a virtual space obtained by capturing an image of real space (the outside world) in approximately real time. A user wearing a video see-through display cannot directly see the real space, but can indirectly see the real space by looking at the displayed virtual space.

[0043] In S301, the system control unit 218 detects the line of sight of the user looking at the image displayed on the EVF 217 (obtains line of sight information related to the line of sight). For example, the system control unit 218 detects the line of sight from the positional relationship between the pupil and the reflection image (Purkinje image) of light reflected from the cornea. Any method can be used as long as it can detect the user's line of sight.

[0044] In S302, the system control unit 218 detects the subject at which the user's gaze is directed, based on the gaze information acquired in S301. When detecting the subject at which the user's gaze is directed, the system control unit 218 also acquires information on the type, orientation, and size of the subject. The system control unit 218 also determines whether the detected subject is different from the subject detected in the previous frame (the previous processing of S302). If a subject at which the user's gaze is directed is detected, the process proceeds to S303; if not, the operation flow ends.

[0045] The subject at which the user's gaze is directed is the subject displayed at the position closest to the user's gaze position. For example, the gaze position is the position on the EVF 217 (display surface) of the detected gaze. The system control unit 218 may compare the distance from the detected gaze position on the EVF 217 to each subject on a two-dimensional plane, and detect the subject displayed at the position closest to that position. Alternatively, for example, the gaze position is the intersection of the gaze of the right eye and the gaze of the left eye. The system control unit 218 may compare the distance from the intersection of the gaze of the right eye and the gaze of the left eye to each subject in three-dimensional space, and detect the subject displayed at the position closest to that intersection.

[0046] In S303, the system control unit 218 sets multiple specific regions for the subject detected in S302. While subject recognition accuracy has improved in recent years through deep learning and other techniques, it is difficult to set multiple sub-regions for each subject with high accuracy in a short period of time. Therefore, templates are recorded in the non-volatile memory 220. The system control unit 218 sets multiple specific regions using a template corresponding to the subject toward which the user's gaze is directed. This makes it possible to detect the overall image of the subject and then set multiple specific regions that match the orientation and size of the subject with high accuracy in a short period of time. Note that if there is no template corresponding to the subject, this operation flow may be terminated, or multiple specific regions may be set in a predetermined region (e.g., the center or top of the subject).

[0047] FIG. 4 is an explanatory diagram of multiple specific regions. A template corresponding to a shirt 400 includes information such as coordinates and regions for setting a logo portion 401 in the center of the shirt 400, a collar portion 402 at the top, and a sleeve portion 403 at the right (or left) end. The system control unit 218 uses the template to set specific regions in the logo portion 401, the collar portion 402, and the sleeve portion 403. While each specific region is shown as an oval in FIG. 4, there are no limitations on the shape of the specific region. The shape of the specific region may be different depending on the subject, or specific regions of different shapes may be set within the same subject. The number of specific regions is not limited to three, as long as there is more than one, and the number of specific regions may be different depending on the subject.

[0048] The multiple specific regions may include a region presenting information about the subject. For example, the system control unit 218 may set a specific region in a region presenting information about the subject that is within a predetermined range from the subject detected in S302. The region presenting information about the subject is, for example, a region presenting a product description and price. Generally, the product description and price are superimposed on or displayed adjacent to the product. By setting a specific region in the region presenting information about the subject, information about the subject of interest can be extracted from multiple angles.

[0049] Returning to the explanation of Fig. 3, in S304, the system control unit 218 determines (visual determination) whether the user is looking at any of the multiple specific areas set in S303 (whether the user's gaze is on any of the multiple specific areas) based on the gaze information acquired in S301. If the user is looking at any of the multiple specific areas, the process proceeds to S305; if not, the operation flow ends.

[0050] In S305, the system control unit 218 determines the current visual state relative to the visual history (past state). If the user is looking at a specific area different from that of the previous frame, the system control unit 218 determines the current visual state as visual start, and if the user is looking at the same specific area as that of the previous frame, the system control unit 218 determines the current visual state as continuous visual state.

[0051] In S306, the system control unit 218 counts the number of times the user looks at the specific region (the number of times the user's line of sight enters the specific region, the number of visual contacts of the specific region). First, when the user looks at a subject different from that in the previous frame (when the user's line of sight enters the region of a first subject and then enters the region of a second subject different from the first subject), the system control unit 218 performs a process of resetting the number of visual contacts. In the process of resetting the number of visual contacts, all of the visual contacts of multiple specific regions related to the subject that the user looked at in the previous frame are reset to their initial values. When the user looks at the same subject as in the previous frame (when the user's line of sight continues to enter the region of the first subject), the system control unit 218 does not perform the reset process. Then, when it is determined in S305 that visual contact has started, the system control unit 218 counts up the number of visual contacts of the specific region from which visual contact has started, and when it is determined that visual contact has continued, the system control unit 218 does not count up the number of visual contacts. When the line of sight enters the specific area, the count is incremented, and when it remains in the area, the count is not incremented, so that the number of visual glances is not incremented when the user continues to look at the specific area unintentionally.

[0052] In S307, the system control unit 218 determines whether the total number of visual inspections of the subject detected in S302 (the total number of visual inspections of each specific area related to the subject) is equal to or greater than a threshold number. If the total number of visual inspections is equal to or greater than the threshold number, the system control unit 218 proceeds to S308; if not, the system control unit 218 ends this operation flow.

[0053] A specific example of the total number of visual glances will be described with reference to Figure 4. For example, if the user looks at all of the specific areas of shirt 400 (logo area 401, collar area 402, and sleeve area 403) once each, the total number of visual glances is 3. Also, if the user looks at logo area 401, collar area 402, and logo area 401 in that order, the total number of visual glances is also 3. If the user looks at at least two specific areas, the total number of visual glances in SS307 can be equal to or greater than the threshold number of times.

[0054] In S308, the system control unit 218 determines that the user is interested in the subject detected in S302, and records interest information about the subject in the memory 215. After recording the interest information in the memory 215, the system control unit 218 ends this operation flow. This operation flow is repeatedly executed until the HMD 100 is powered off.

[0055] The interest information may include, for example, an image of a subject, an image of a specific region, the number of times each specific region is viewed, and the total number of times each region is viewed. , or a gaze transition. The gaze transition may be a transition of a specific area where the user's gaze is directed (a general gaze transition), or a gaze position transition (a detailed gaze transition). The interest information may include GPS information that identifies the location where the subject of interest was viewed, and biological information such as the user's heart rate or brain waves when the subject of interest was viewed.

[0056] According to the processes of S304 to S308, if the total number of visual glances is equal to or greater than the threshold number, it is determined that the user is interested in the subject. Generally, when people look at a subject of interest, they often look at multiple regions rather than just one point. Therefore, the total number of visual glances is used to evaluate whether the user has made a gaze transition (trajectory) that looks at multiple regions. This makes it possible to distinguish between cases where the user intentionally looks at the subject and cases where the user unintentionally looks at the subject, thereby detecting the subject of interest with high accuracy. In addition, by setting multiple specific regions related to the subject and detecting whether the user looks at only the specific regions, it is possible to detect the subject of interest while reducing the processing load.

[0057] Note that it is not necessary to set a specific region. For example, the system control unit 218 may count the number of times the user's line of sight enters the area of ​​the subject, and if that number is greater than a threshold number, determine that the user is interested in the subject. By omitting the process of setting a specific region, the subject of interest can be detected with simpler processing.

[0058] Although an example in which the present invention is applied to a video see-through display (video see-through HMD) has been described, display devices to which the present invention can be applied are not limited to video see-through displays. For example, the present invention can also be applied to an optical see-through display (optical see-through HMD). An optical see-through display has, for example, lenses similar to lenses in ordinary eyeglasses, and projects an image of a virtual object onto the lenses. A user wearing the optical see-through display can directly view real space through the lenses. Furthermore, the user can also view a virtual object projected onto the lenses. The virtual object is positioned in real space with parallax between the projection position relative to the right eye lens and the projection position relative to the left eye lens.

[0059] The present invention can also be applied to an HMD that displays images in a VR space that is unrelated to real space. In an HMD that displays images in a VR space, the system control unit 218 may detect the object that the user is looking at based on information about the user's line of sight and information about the display position of the object in the VR space.

[0060] In this way, in the first embodiment, if the number of times the user enters the area of ​​a subject is greater than the threshold number, it is determined that the user is interested in that subject, which allows the subject of interest to be detected with high accuracy in a short time.

[0061] (Second embodiment) In the first embodiment, when the user's line of sight enters the area of ​​a first subject and then enters the area of ​​a second subject different from the first subject, the system control unit 218 resets the number of times the user's line of sight enters the area of ​​the first subject to an initial value. In the second embodiment, the system control unit 218 retains this number of times without resetting it. Furthermore, the system control unit 218 counts the number of times the user's line of sight enters the area of ​​a subject, taking into account the amount of change in the user's line of sight.

[0062] 5 is a flowchart showing an example of the operation of the HMD 100 according to the second embodiment. The processes of S501, S505, S507, and S510 are the same as the processes of S301, S303, S305, and S308 in FIG.

[0063] In S502, the system control unit 218 detects the amount of change in the line of sight, which indicates the amount of movement or vector direction of the line of sight, using the line of sight information acquired over a predetermined time period (acquires information on the amount of change). The information on the amount of change may include other parameters such as variance (variation).

[0064] In S503, similar to the process of S302 in FIG. 3, the system control unit 218 detects a subject toward which the user's gaze is directed and determines whether the detected subject is different from the subject detected in the previous frame. Furthermore, the system control unit 218 records the detection and determination results in the system memory 219, including the subject history information. The subject history information indicates a history of subjects viewed by the user, including, for example, the type of subject viewed by the user, the time the user viewed the subject, the number of glances, and update history information. However, the number of glances is not recorded in S503, but is recorded in S508, which will be described later. If the system control unit 218 does not detect a subject identical to a previously detected subject again within a predetermined time, it deletes information about the subject from the subject history. This prevents the system memory 219 from becoming overloaded. If a subject toward which the user's gaze is directed is detected, the system control unit 218 proceeds to S504; otherwise, it terminates this operation flow.

[0065] In S504, the system control unit 218 refers to the object history. The system control unit 218 determines whether the object detected in S503 is the same object as the object detected in the previous frame, whether the object is a different object and the object history does not record the number of visual observations for that object, or whether the object is a different object and the object history records the number of visual observations for that object. The system control unit 218 records the determination result in the system memory 219.

[0066] In S506, the system control unit 218 determines (visual determination) whether the user is looking at any of the multiple specific areas set in S505 (whether the user's gaze is on any of the multiple specific areas) based on the gaze information acquired in S501. If the user is looking at any of the multiple specific areas, the process proceeds to S507; if not, the operation flow ends.

[0067] S506 and S304 in Figure 3 differ in the gaze information used for visual determination. First, the raw data of the detected gaze (so-called RAW data) not only has large variations but also contains noise due to physiological phenomena such as saccades. Therefore, the raw data is subjected to low-pass filtering to reduce the influence of noise, such as moving average processing with a long tap count or statistical processing. However, this low-pass filtering process reduces responsiveness to changes in the user's gaze.

[0068] In order to improve responsiveness to gaze changes, it is necessary to reduce the number of samples used when performing moving average processing and statistical processing, but reducing the number of samples reduces accuracy. Therefore, the system control unit 218 determines whether to improve responsiveness based on the amount of gaze change acquired in S502. When the amount of gaze change is greater than a threshold amount, the system control unit 218 prioritizes responsiveness and uses gaze information acquired over a relatively short first period of time for visual judgment. When the amount of gaze change is less than the threshold amount, the system control unit 218 prioritizes accuracy and uses gaze information acquired over a second period of time longer than the first period of time for visual judgment. This ensures both responsiveness and accuracy.

[0069] Furthermore, the user's line of sight may enter an area that the user is not looking at intentionally. For example, this may be the case when a specific area exists on the path of the user's line of sight. If the count is increased in S508 in such a case, the accuracy of detecting the subject of interest will decrease. Here, if the amount of change in the user's line of sight (change in the amount of movement of the line of sight) is large, it is highly likely that the user is not looking at the subject intentionally. Therefore, if the amount of change in the line of sight acquired in S502 is greater than a threshold amount, the system control unit 218 prevents the count from being increased even if the line of sight is in the specific area. When the user's line of sight is in a specific area, if the amount of change in the line of sight is greater than a threshold amount, the system control unit 218 determines that the user's line of sight is not in the specific area (passing determination). If the amount of change in the line of sight is less than the threshold amount, the system control unit 218 determines that the user's line of sight is in the specific area. This makes it possible to detect the subject of interest with higher accuracy.

[0070] The threshold amount may be varied according to a change in the direction of gaze movement (vector change). When a user intentionally moves their gaze from a currently viewed subject to the next subject, the vector change is large immediately after the gaze movement because the gaze changes direction toward the next subject. After that, the gaze moves to the next subject over the shortest distance, so the vector change is small. A specific area present on a (linear) gaze path with a small vector change is likely to be an area that the user is not intentionally looking at. Therefore, the threshold amount when the vector change is large may be larger than the threshold amount when the vector change is small. This prevents the user from being judged as having passed if they suddenly (intentionally) go to look at the next subject.

[0071] In S508, the system control unit 218 counts the number of times the user looks at the specific region (the number of times the user's line of sight enters the specific region, the number of times the specific region is viewed) according to the determination result of S504.

[0072] If the subject detected in S503 is the same as the subject detected in the previous frame, and if the determination result in S507 is visual inspection start, the system control unit 218 counts up the number of visual inspections of the specific region that has started visual inspection. If the determination result in S507 is visual inspection continued, the system control unit 218 does not count up the number of visual inspections of the specific region that is being visually inspected continued.

[0073] If the subject detected in S503 is different from the subject detected in the previous frame and the subject history does not contain a record of the number of visual observations for that subject, the system control unit 218 sets all of the visual observations for the multiple specific regions for the subject detected in S503 to an initial value of 0. Furthermore, the system control unit 218 counts up the number of visual observations for the specific regions viewed by the user.

[0074] If the subject detected in S503 is different from the subject detected in the previous frame and the subject history has a record of the number of visual observations for that subject, the system control unit 218 obtains the number of previous visual observations of the specific area currently being viewed by the user from the subject history. The system control unit 218 counts up the obtained number of visual observations.

[0075] Furthermore, system control unit 218 includes the counted number of visual observations of the specific area in information on the subject history of the subject related to the specific area and records this in system memory 219. If the recorded number of visual observations is not updated for a predetermined period of time, system control unit 218 deletes the number of visual observations from system memory 219. This prevents the capacity of system memory 219 from becoming overwhelmed.

[0076] In S509, the system control unit 218 determines whether the total number of visual observations of the subject detected in S502 is equal to or greater than a threshold number. If the total number of visual observations is equal to or greater than the threshold number, the process proceeds to S310; if not, the operation flow ends. The flow in FIG. 5 differs from the flow in FIG. 3 in that the total number of visual observations is counted using subject history, taking into account gaze transitions such as looking at another subject and then returning. Under certain conditions, a subject for which the user has looked at one specific area multiple times is also detected as a subject of interest. Details will be described later with reference to FIG. 6.

[0077] In S511, the system control unit 218 presents related information related to the subject of interest. For example, the system control unit 218 connects to the Internet to acquire related information, and Display at 217.

[0078] If the subject detected in S503 is the same as the subject detected in the previous frame, the presentation of related information may be stopped after a predetermined time without recording interest information. Also, if the subject detected in S503 is the same as the subject detected in the previous frame, less information about the subject than the initially (or previously) recorded interest information may be recorded as new interest information, and the presented information may be reduced over time.

[0079] FIG. 6 is an image diagram of the path of the user's gaze when comparing multiple subjects. Shirts 601, 602, 603, and 604 each have a different shape. Starting point 605 indicates the starting point of the user's gaze path, and ending point 606 indicates the ending point of the user's gaze path. Although not shown in FIG. 6, specific areas are set in the logo, collar, and sleeve of each shirt. In FIG. 6, areas that pass through the specific areas but are not marked with a dot (for example, the right sleeve of shirt 601 or the right sleeve of shirt 603) are areas that are determined to have passed through because the amount of change in the gaze is greater than the threshold amount. The threshold number of times used in the determination in S509 is 3.

[0080] First, because starting point 605 is on shirt 601, shirt 601 is detected as the subject, and the number of sightings of the logo portion of shirt 601 is counted as 1. Thereafter, the user moves his / her gaze, in order, to the logo portion of shirt 602, the collar, and the logo portion of shirt 603. At this stage, the total number of sightings of shirt 602 is 2, so shirt 602 is not detected as a subject of interest. Thereafter, the user moves his / her gaze, in order, to the collar and sleeves of shirt 603. At this stage, the total number of sightings of shirt 603 (3 sightings of the logo portion, 2 sightings of the collar, and 1 sighting of the sleeve) is 3 or more, so shirt 603 is detected as a subject of interest.

[0081] The user then moves his or her gaze to the logo portion of shirt 604 and then to the logo portion of shirt 602. At this stage, the total number of glances at shirt 602 is three (two glances at the logo portion and one glance at the collar portion), so shirt 602 is detected as a subject of interest.

[0082] The user then moves his or her gaze from the logo portion of shirt 604 to the logo portion of shirt 603 and back to the logo portion of shirt 604 in that order. At this stage, the total number of visual inspections of shirt 604 (number of visual inspections of the logo portion) is three, so shirt 604 is detected as a subject of interest. Note that because the user also looked at other shirts before and after looking at the logo portion of shirt 604, shirt 604 is detected as a subject of interest even when only the logo portion is looked at, but it is not detected as a subject of interest even when the user looks at the logo portion of shirt 604 three or more times in a row.

[0083] The threshold number used for the determination in S509 may be varied depending on the number of specific regions into which the user's gaze fell (the user looked at). If the user looked at a large number of specific regions, it is likely that the user was looking at the subject from multiple angles and was interested in the subject. Therefore, the threshold number when the user looked at a large number of specific regions may be smaller than the threshold number when the user looked at a small number of specific regions.

[0084] A specific description will be given with reference to FIG. 6. For example, if the user looks at two or more specific regions, the threshold number of times is set to 3, and if the user looks at one specific region, the threshold number of times is set to 4. Shirts 602 and 603 are detected as subjects of interest in the same way as in the previous example, because the user looked at two or more specific regions. Shirt 604 is not detected as a subject of interest, unlike the previous example, because the user looked at one specific region. By differentiating the conditions for detecting a subject of interest when the subject is viewed from multiple angles and when the subject is viewed locally, it is possible to detect a subject of interest with higher accuracy.

[0085] As described above, in the first embodiment, the system control unit 218 detects changes in the subject viewed by the user. In the second embodiment, the number of visual glances is reset when the user's gaze changes, but in the third embodiment, the number of visual glances is maintained even if the user changes the subject they are looking at. By maintaining the number of visual glances, the subject of interest can be detected even when multiple specific areas of one subject are viewed consecutively, or when multiple subjects are viewed in comparison. For example, even when initially comparing multiple products (subjects), such as window shopping, and then narrowing down the viewing points for a product of interest and comparing multiple subjects, the subject of interest can be detected with high accuracy. Furthermore, in the second embodiment, the number of visual glances is counted taking into account the amount of change in the line of sight, so that the subject of interest to the user can be detected with high accuracy in a short period of time.

[0086] The various controls described above as being performed by the system control unit 218 may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0087] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0088] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0089] In the above-described embodiment, the present invention has been described as being applied to an HMD, but the present invention is not limited to this example and can be applied to any information processing device that can acquire line-of-sight information. For example, the present invention can be applied to an imaging device.

[0090] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0091] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An acquisition means for acquiring gaze information relating to a user's gaze; a counting means for counting the number of times the line of sight enters an area of ​​a subject based on the line of sight information; determining means for determining that the user is interested in the subject if the number of times is greater than a threshold number of times; have 1. An information processing device comprising: (Configuration 2) further comprising a setting means for setting a plurality of specific regions relating to the subject; The counting means counts the number of times the line of sight enters any one of the plurality of specific regions. 2. The information processing device according to configuration 1, (Configuration 3) The setting means sets the plurality of specific regions using a template corresponding to the subject. 3. The information processing device according to configuration 2. (Configuration 4) The specific area includes an area that presents information about the subject. 4. The information processing device according to configuration 2 or 3. (Configuration 5) The counting means When the line of sight enters an area of ​​a first subject and then enters an area of ​​a second subject different from the first subject, The number of times the line of sight has entered the area of ​​the first object is reset to an initial value. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The counting means When the line of sight enters an area of ​​a first subject and then enters an area of ​​a second subject different from the first subject, The number of times the line of sight enters the area of ​​the first object is stored. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 7) The counting means If the amount of change in the line of sight is greater than a threshold amount, counting up the number of times based on the line of sight information acquired at a first time; When the amount of change in the line of sight is smaller than the threshold amount, the number of times is counted up based on the line of sight information acquired during a second time period that is longer than the first time period. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The counting means If the amount of change in the line of sight is greater than a threshold amount, the number of times is not counted up, If the amount of change in the line of sight is smaller than the threshold amount, the number of times is counted up. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 9) The threshold amount when the change in the movement direction of the line of sight is large is larger than the threshold amount when the change in the movement direction of the line of sight is small. 9. The information processing device according to configuration 7 or 8. (Configuration 10) The threshold number of times when the number of specific regions into which the gaze falls is large is smaller than the threshold number of times when the number of specific regions into which the gaze falls is small. 10. The information processing device according to any one of configurations 2 to 9. (Configuration 11) The apparatus further includes a recording unit for recording interest information relating to the subject determined to be of interest by the determining unit. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) The interest information includes at least one of the transition of the line of sight in the area of ​​the subject, the number of times, and an image of the subject. 12. The information processing device according to configuration 11. (method) an acquiring step of acquiring gaze information regarding a user's gaze; a counting step of counting the number of times the line of sight enters an area of ​​the subject based on the line of sight information; determining that the user is interested in the subject if the number of times is greater than a threshold number of times; have 2. A method for controlling an information processing apparatus comprising: (program) 13. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 12. (medium) 13. A computer-readable storage medium storing a program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0092] 100: Information processing device 218: System control unit

Claims

1. An acquisition means for acquiring gaze information relating to a user's gaze; a counting means for counting the number of times the line of sight enters an area of ​​a subject based on the line of sight information; determining means for determining that the user is interested in the subject if the number of times is greater than a threshold number of times; have 1. An information processing device comprising:

2. further comprising a setting means for setting a plurality of specific regions relating to the subject; The counting means counts the number of times the line of sight enters any one of the plurality of specific regions.

2. The information processing apparatus according to claim 1, wherein:

3. The setting means sets the plurality of specific regions using a template corresponding to the subject.

3. The information processing apparatus according to claim 2, wherein:

4. The specific area includes an area that presents information about the subject.

3. The information processing apparatus according to claim 2, wherein:

5. The counting means When the line of sight enters an area of ​​a first subject and then enters an area of ​​a second subject different from the first subject, The number of times the line of sight has entered the area of ​​the first object is reset to an initial value.

2. The information processing apparatus according to claim 1, wherein:

6. The counting means When the line of sight enters an area of ​​a first subject and then enters an area of ​​a second subject different from the first subject, The number of times the line of sight enters the area of ​​the first object is stored.

2. The information processing apparatus according to claim 1, wherein:

7. The counting means If the amount of change in the line of sight is greater than a threshold amount, counting up the number of times based on the line of sight information acquired during a first period of time; If the amount of change in the line of sight is smaller than the threshold amount, the number of times is counted up based on the line of sight information acquired during a second time period that is longer than the first time period.

2. The information processing apparatus according to claim 1, wherein:

8. The counting means If the amount of change in the line of sight is greater than a threshold amount, the number of times is not counted up, If the amount of change in the line of sight is smaller than the threshold amount, the number of times is counted up.

2. The information processing apparatus according to claim 1, wherein:

9. The threshold amount when the change in the movement direction of the line of sight is large is larger than the threshold amount when the change in the movement direction of the line of sight is small.

8. The information processing apparatus according to claim 7,

10. The threshold number of times when the number of specific regions into which the gaze falls is large is smaller than the threshold number of times when the number of specific regions into which the gaze falls is small.

3. The information processing apparatus according to claim 2, wherein:

11. The apparatus further includes a recording unit for recording interest information relating to the subject determined to be of interest by the determining unit.

2. The information processing apparatus according to claim 1, wherein:

12. The interest information includes at least one of the transition of the line of sight in the area of ​​the subject, the number of times, and an image of the subject.

12. The information processing apparatus according to claim 11,

13. an acquiring step of acquiring gaze information regarding a user's gaze; a counting step of counting the number of times the line of sight enters an area of ​​the subject based on the line of sight information; determining that the user is interested in the subject if the number of times is greater than a threshold number of times; have 2. A method for controlling an information processing apparatus comprising:

14. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 12.

Images