Information processing apparatus

Abstract

To provide a technique that can accurately determine a person's degree of interest in an object.SOLUTION: An information processing apparatus of the present invention has: first acquisition means that acquires information on a user's line of sight; detection means that detects an object that the user looks at based on the line-of-sight information; second acquisition means that acquires, from a storage unit, reference information that is predetermined information corresponding to the object for determining the user's degree of interest in the object; and determination means that determines the degree of interest based on the line-of-sight information and the reference information.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an information processing device, and more particularly to a technique for determining a person's level of interest in an object. [Background technology]

[0002] A technology has been proposed that determines a person's level of interest in an object based on the person's gaze information. Patent Document 1 discloses a technology that measures the number of people viewing advertising content displayed on an advertising display device based on the gaze direction of people in an area where the advertising display device is installed. Patent Document 2 discloses a technology that determines a passenger's (passenger of a moving body) level of interest in a gazed-at object based on the duration of gaze of the passenger on the gazed-at object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-112401 A [Patent Document 2] JP 2007-172378 A Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technologies (e.g., technologies disclosed in Patent Documents 1 and 2), the act of a person looking at an object is regarded as the person's act of being interested in the object, and the person's level of interest in the object is judged. However, just because a person looks at an object does not necessarily mean that the person is interested in the object. Therefore, the conventional technologies cannot judge the level of interest with a high degree of accuracy.

[0005] The present invention aims to provide a technique capable of determining a person's level of interest in an object with high accuracy. [Means for solving the problem]

[0006] A first aspect of the present invention is an information processing device characterized by having a first acquisition means for acquiring a user's gaze information, a detection means for detecting an object the user is looking at based on the gaze information, a second acquisition means for acquiring from a memory unit reference information for judging the user's level of interest in the object, which is predetermined information corresponding to the object, and a judgment means for judging the level of interest based on the gaze information and the reference information.

[0007] A second aspect of the present invention is an information processing device characterized by having a first acquisition means for acquiring a user's gaze information, a detection means for detecting an object the user is looking at based on the gaze information, a second acquisition means for acquiring reference information for judging the user's level of interest in the object by analyzing an image of the object, and a judgment means for judging the level of interest based on the gaze information and the reference information.

[0008] A third aspect of the present invention is a control method for an information processing device, comprising the steps of acquiring a user's gaze information, detecting an object that the user is looking at based on the gaze information, acquiring from a memory unit reference information for judging the user's level of interest in the object, which is predetermined information corresponding to the object, and judging the level of interest based on the gaze information and the reference information.

[0009] A fourth aspect of the present invention includes a step of acquiring line-of-sight information of a user, and The control method for an information processing device comprises the steps of: detecting an object which the user is looking at based on gaze information; acquiring reference information for judging the user's level of interest in the object by analyzing an image of the object; and judging the level of interest based on the gaze information and the reference information.

[0010] A fifth aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device described above.A sixth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device described above. Effect of the Invention

[0011] According to the present invention, it is possible to determine a person's degree of interest in an object with high accuracy. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an external view of a display system. [Diagram 2] FIG. 2 is a block diagram showing an electrical configuration of the display system. [Diagram 3] 1 is a diagram for explaining the principle of a gaze detection method. [Figure 4] FIG. 13 is a diagram showing an eye image and the like. [Diagram 5] 13 is a flowchart of a gaze detection process. [Figure 6] 13 is a flowchart of the overall process. [Figure 7] FIG. 2 is a block diagram showing a functional configuration of the display system. [Figure 8] FIG. 13 is a diagram showing an object of interest and interest level reference information. [Figure 9] 13 is a flowchart of an interest level determination process. [Figure 10] 13 is a flowchart of a reference information acquisition process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] <<Example 1>> A first embodiment of the present invention will be described.

[0014] The present invention is applicable to various electronic devices capable of acquiring a person's line of sight information (information related to the line of sight). For example, the present invention is applicable to both video see-through display devices and optical see-through display devices. A video see-through display device displays an image (virtual space) captured from a real space on a display surface (a display surface that does not transmit light from the real space (outside world)) by synthesizing a graphic (e.g., a virtual object) as necessary. In this case, a user cannot directly see the real space, but can indirectly see the real space or see a graphic synthesized with the image of the real space by looking at the displayed image. A display device of the optical see-through type displays a graphic on a display surface (a display surface that transmits light from the real space), for example. In this case, a user can directly see the real space through the display surface or see a graphic displayed on the display surface.

[0015] The present invention is applicable to head-mounted display devices and other display devices. For example, the present invention is applicable to handheld display devices and stationary display devices. The head-mounted display device is, for example, smart glasses (AR (Augmented Reality) glasses) or HMD (head-mounted display). The handheld display device is, for example, a smartphone or a tablet terminal. A display device that a user holds in his / her hand and wears (places) on his / her head is a type of handheld display device and also a type of head-mounted display device. A smartphone mounted on a head-mounted adapter (for example, VR (Virtual Reality) goggles) is a type of head-mounted display device. The present invention is applicable to head-mounted display devices in which a user views an image with both eyes, and to head-mounted display devices in which a user views an image with one eye.

[0016] The present invention can also be applied to a method in which the user is allowed to view only a virtual space without viewing a real space. In other words, the present invention can be applied to various XR (Cross Reality) such as AR (Augmented Reality), MR (Mixed Reality), and VR (Virtual Reality).

[0017] The present invention is also applicable to electronic devices other than display devices. An information processing device to which the present invention is applied may be provided in a display device, or may be provided in an electronic device separate from the display device. For example, the present invention is also applicable to a controller or a PC (personal computer) connected to a display device. The present invention is also applicable to a surveillance camera.

[0018] In the first embodiment, gaze information of a person is acquired, and the person's interest level in an object is determined based on the gaze information. In the first embodiment, the object is assumed to be a real object, but the object may be a virtual object. In the conventional technology, the act of a person looking at an object is regarded as the person's act of being interested in the object, and the person's interest level in the object is determined. However, just because a person looks at an object does not necessarily mean that the person is interested in the object. Therefore, the conventional technology cannot determine the interest level with high accuracy.

[0019] For example, advertising media (e.g., posters, billboards, and digital signage) may have a large area for catching the eye (eye-catching area). An eye-catching area is, for example, an area in which a person's (advertising model's) photo or a catchphrase is placed prominently. Here, the eye-catching area may have low relevance to the advertising target (e.g., a product or service). Therefore, even if a person sees the eye-catching area, the person may not associate the advertising target with the advertising target and may not be interested in the advertising target and the advertising media. Note that areas with low relevance to the advertising target are not limited to eye-catching areas.

[0020] Therefore, in the first embodiment, the interest level of a person in an object is determined with high accuracy using a suitable criterion according to the object. For example, the interest level is determined with high accuracy using a criterion that takes into account the contents of the object (the design intent of the object).

[0021] <Configuration Description> 1(A) and 1(B) show the appearance of a display system 100 according to a first embodiment. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a rear perspective view. As shown in FIG. 1(A), the display system 100 includes an HMD 110, a controller 120, and a server 130. In the first embodiment, the HMD 110 is used as a video see-through type HMD. The controller 120 is connected to the HMD 110 by wire and is connected to the server 130 by wireless. The controller 120 may be connected to the HMD 110 by wireless. FIG. 2 is a block diagram showing an electrical configuration of the display system 100.

[0022] 1(A), 1(B), and 2, the HMD 110 has a photographing lens 111, a display unit 112, light sources 113a and 113b, a light receiving lens 114, an eye imaging element 115, and a knob 140. Furthermore, as shown in Fig. 2, the HMD 110 has an imaging element 116, a photometry circuit 117, a light source driving circuit 118, and a display unit driving circuit 119. As shown in Fig. 1(B), the display unit 112, the light sources 113a and 113b, the light receiving lens 114, and the eye imaging element 115 are provided for each of the user's right eye and left eye.

[0023] The photographing lens 111 is a lens for capturing an image of the outside world, and the image sensor 116 is an image sensor for capturing an image of the outside world. The image sensor 116 is disposed on a planned image forming plane of the photographing lens 111.

[0024] The display unit 112 displays various images (information). For example, the display unit 112 displays an image of the outside world captured by the imaging element 116, or displays information about an object that the user is looking at. The display unit drive circuit 119 is controlled by the controller 120 (CPU 121, described later) and drives the display unit 112.

[0025] Each of light source 113a and light source 113b is a light source that illuminates the user's eye, and is, for example, an infrared light emitting diode that emits infrared light that is insensitive to the user. Light source driving circuit 118 is controlled by controller 120 (CPU 121) and drives light sources 113a and 113b. A portion of the light emitted from light sources 113a and 113b and reflected by the user's eye is collected by light receiving lens 114 onto eye imaging element 115. Light receiving lens 114 is a lens for capturing an image of the user's eye, and eye imaging element 115 is an imaging element for capturing an image of the user's eye.

[0026] Knob 140 is a knob for adjusting the distance between right eye display unit 112 and left eye display unit 112 so as to match the interpupillary distance of the user.

[0027] The photometry circuit 117 amplifies, logarithmically compresses, and A / D converts the signal obtained from the image sensor 116, which also functions as a photometry sensor, specifically, the luminance signal corresponding to the brightness of the field, and sends the result to the controller 120 (CPU 121) as field luminance information.

[0028] The controller 120 includes a CPU 121, a memory unit 122, a gaze detection circuit 123, an LPF (low pass filter) 124, a display unit drive circuit 125, operation members 126 to 128, and a wireless communication circuit 129.

[0029] The CPU 121 is a central processing unit of a microcomputer built into the controller 120 , and controls the entire display system 100 .

[0030] The memory unit 122 has a function of storing the image signal from the eye imaging element 115 and a function of storing a gaze correction parameter. The gaze correction parameter is a parameter for correcting individual differences in the gaze. Furthermore, the memory unit 122 has a function of storing interest level reference information. The interest level reference information is predetermined information corresponding to an object, and is reference information for judging the user's interest level in the object. The memory unit 122 stores a plurality of interest level reference information corresponding to a plurality of objects, respectively.

[0031] Gaze detection circuit 123 A / D converts the output of eye imaging element 115 (eye image obtained by capturing an image of the eye) when an optical image of the eye is formed on eye imaging element 115, and transmits the result to CPU 121 via LPF 124. CPU 121 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm described later, and detects the user's gaze from the positions of the feature points.

[0032] The operation members 126 to 128 receive an operation from the user and output an operation signal (a signal corresponding to an operation performed by the user) to the CPU 121. For example, the operation member 126 is a touch panel capable of receiving a touch operation, the operation member 127 is an operation lever that can be pushed down in each direction, and the operation member 128 is a four-way key that can be pushed in each of four directions. The operation member 126 (touch panel) has a function of displaying an image (information). In this manner, the operation member 126 has a function as a touch panel and a function as a display unit. The display unit driving circuit 125 is controlled by the CPU 121 and drives the operation member 126 (display unit). The user can perform various operations (instructions) using the operation members 126 to 128. For example, the user can finely adjust the position of a UI (user interface, for example, an indicator) displayed on the display unit 112 using the operation members 126 to 128.

[0033] The wireless communication circuit 129 communicates with external devices under the control of the CPU 121. For example, the wireless communication circuit 129 transmits the interest level (the user's interest level in an object) determined by the CPU 121 to the server 130 via the Internet and records it.

[0034] <Explanation of gaze detection process> The gaze detection process (gaze detection method) will be described with reference to Figs. 3, 4(A), 4(B), and 5. The gaze of the right eye and the gaze of the left eye are detected by the following gaze detection method. Fig. 3 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for detecting the gaze. As shown in Fig. 3, light sources 113a and 113b are arranged approximately symmetrically with respect to the optical axis of light receiving lens 114, and illuminate user's eyeball 200. A part of the light emitted from light sources 113a and 113b and reflected by eyeball 200 is collected by light receiving lens 114 on eye imaging element 115. Fig. 4(A) is a schematic diagram of an eye image captured by eye imaging element 115 (optical image of the eye projected on eye imaging element 115), and Fig. 4(B) is a diagram showing the output intensity of eye imaging element 115. Fig. 5 is a flowchart of the gaze detection process.

[0035] 5 starts, in step S1, CPU 121 drives light sources 113a and 113b using light source drive circuit 118 so as to emit infrared light toward user's eyeball 200. An optical image of the user's eye illuminated by the infrared light passes through light receiving lens 114 and is formed on eye imaging element 115, which then performs photoelectric conversion. This provides an electrical signal of the eye image that can be processed.

[0036] In step S2, the CPU 121 acquires an eye image (image data, image signal) from the eye imaging element 115 via the line of sight detection circuit 123.

[0037] In step S3, the CPU 121 detects the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 113a and 113b and the pupil center c from the eye image obtained in step S2.

[0038] Infrared light emitted from light sources 113a and 113b illuminates cornea 201 of user's eyeball 200. At this time, corneal reflection images Pd and Pe formed by a part of the infrared light reflected from the surface of cornea 201 are collected by light receiving lens 114 and focused on eye imaging element 115 to become corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from ends a and b of pupil 202 are also focused on eye imaging element 115 to become pupil end images a' and b' in the eye image.

[0039] FIG. 4B shows luminance information (luminance distribution) of the region α in the eye image of FIG. 4A. In FIG. 4B, the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe, and the coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb. As shown in FIG. 4B, an extremely high level of luminance is obtained at the coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from the coordinates Xa to the coordinates Xb, which corresponds to the region of the pupil 202 (the region of the pupil image obtained by focusing the light beam from the pupil 202 on the eye imaging element 115), an extremely low level of luminance is obtained except for the coordinates Xd, Xe. A luminance intermediate between the above two types of luminance is obtained in the region of the iris 203 outside the pupil 202 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 203). For example, a luminance intermediate between the above two types of luminance is obtained in a region where the X coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and a region where the X coordinate is less than coordinate Xb.

[0040] From the brightness distribution shown in FIG. 4(B), the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. For example, when the brightness is extremely low, The coordinates of the corneal reflection images Pd', Pe' can be obtained as high coordinates, and the coordinates of the pupil edge images a', b' can be obtained as extremely low coordinates. When the rotation angle θx of the optical axis of the eyeball 200 relative to the optical axis of the light receiving lens 114 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by focusing the light beam from the pupil center c on the eye imaging element 115 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this way, the coordinates of the corneal reflection images Pd', Pe' and the coordinate of the pupil center image c' can be estimated.

[0041] In step S4, CPU 121 calculates imaging magnification β of the eye image. Imaging magnification β is determined by the position of eyeball 200 relative to light receiving lens 114, and can be calculated using a function of the distance (Xd-Xe) between corneal reflection images Pd', Pe'.

[0042] In step S5, CPU 121 calculates the rotation angle of the optical axis of eyeball 200 relative to the optical axis of light receiving lens 114. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 201 approximately coincide with each other. For this reason, if the standard distance from the center of curvature O of cornea 201 to the center c of pupil 202 is Oc, then the rotation angle θx of eyeball 200 in the ZX plane (plane perpendicular to the Y axis) can be calculated by the following Equation 1. The rotation angle θy of eyeball 200 in the ZY plane (plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0043] In step S6, the CPU 121 reads out parameters necessary for gaze detection from the memory unit 122. For example, the CPU 121 reads out parameters m, Ax, Bx, Ay, By, nx, and ny. The parameter m is a constant determined by the configuration of the optical system for performing the gaze detection process, and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c on the display unit 112. It is assumed that the parameter m is determined in advance and stored in the memory unit 122. The parameters Ax, Bx, Ay, and By are gaze correction parameters for correcting individual differences in the gaze, and are acquired by performing calibration of the gaze detection. The parameter Ax is an offset value in the X-axis direction, the parameter Bx is a sensitivity coefficient in the X-axis direction, the parameter Ay is an offset value in the Y-axis direction, and the parameter By is a sensitivity coefficient in the Y-axis direction. It is assumed that the gaze correction parameters Ax, Bx, Ay, and By are stored in the memory unit 122 before the gaze detection process starts. In the gaze detection process for the right eye, parameters nx=nRx and ny=nRy are acquired, and in the gaze detection process for the left eye, parameters nx=nLx and ny=nLy are acquired. The parameter nRx is a correction coefficient in the X-axis direction for obtaining gaze information for the right eye, and the parameter nRy is a correction coefficient in the Y-axis direction for obtaining gaze information for the right eye. The parameter nLx is a correction coefficient in the X-axis direction for obtaining gaze information for the left eye, and the parameter nLy is a correction coefficient in the Y-axis direction for obtaining gaze information for the left eye. It is assumed that the parameters nRx, nRy, nLx, and nLy are determined in advance and stored in the memory unit 122.

[0044] In step S7, CPU 121 estimates the user's viewpoint on display unit 112 using the rotation angles θx, θy calculated in step S5 and the parameters m, Ax, Bx, Ay, By, nx, and ny read in step S6. The viewpoint can be regarded as the position where the gaze is fixed, the position where the user is looking, or the gaze position. If the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3. Hx=m×(Ax×θx+Bx)×nx (Formula 2) Hy=m×(Ay×θy+By)×ny (Formula 3)

[0045] In step S8, the CPU 121 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 122, and ends the line-of-sight detection process.

[0046] The gaze detection method is not limited to the above method, and may be any method that acquires gaze information from an eye image. The gaze may be detected by a method that does not use an eye image, such as a method that detects an electrooculogram without using an eye image and detects the gaze based on the electrooculogram. As the final gaze information, information indicating the gaze direction (direction of gaze) may be obtained instead of information indicating the viewpoint. For example, processing may be performed up to obtaining rotation angles (Ax×θx+Bx)×nx, (Ay×θy+By)×ny without obtaining the coordinates (Hx, Hy) of the viewpoint. As the final gaze information, information indicating the viewpoint of both eyes (intersection of the gaze of the right eye and the gaze of the left eye, position viewed by both eyes) may be obtained.

[0047] <Overall processing explanation> The overall processing of the display system 100 (controller 120) will be described with reference to Figs. 6 and 7. Fig. 6 is a flowchart of the overall processing. For example, when the user wears the HMD 110 and starts up the HMD 110 and the controller 120, the overall processing of Fig. 6 starts. Fig. 7 is a block diagram showing the functional configuration of the display system 100. Each functional unit in Fig. 7 is realized by the CPU 121 of the controller 120.

[0048] 6 starts, in step S101, the outside world image acquisition unit 302 acquires an image of the outside world (outside world image) captured by the imaging element 116. Although not shown in FIGS. 6 and 7, the CPU 121 performs predetermined image processing on the acquired outside world image and displays it on the display unit 112.

[0049] In step S102, the gaze detection unit 301 performs the gaze detection process of FIG. 5 to obtain the user's gaze information.

[0050] In step S103, CPU 121 detects an object (target object) that the user is looking at based on the gaze information acquired in step S102. For example, object detection unit 303 detects an object from the external world image acquired in step S101. Then, target object detection unit 304 detects the target object from the objects detected by object detection unit 303 based on the gaze information acquired in step S102. For example, the target object is detected based on information obtained from a plurality of gaze information in time series (for example, a trajectory of the gaze, saccades of the eyeball, a time during which the gaze is fixed, and the number of times the object is fixed). Target object information, which is a detection result of the target object, is acquired by the process of step S103. Details of the target object information will be described later.

[0051] In step S104, the reference information acquiring unit 305 acquires interest level reference information corresponding to the target object based on the target object information acquired in step S103 (reference information acquiring process). In the first embodiment, the reference information acquiring unit 305 acquires interest level reference information corresponding to the target object from the memory unit 122. Details of the interest level reference information will be described later.

[0052] For example, assume that the target object information indicates a position of the target object (a position in real space, a position of the target object in a coordinate system (world coordinate system) of the real space). Assume that the position of the object corresponding to the interest level reference information is associated with the interest level reference information stored in the memory unit 122. In this case, the reference information acquisition unit 305 may acquire, from the memory unit 122, the interest level reference information associated with the position closest to the position indicated by the target object information.

[0053] The attention object information may indicate the position and type of the attention object. The position and type of the object corresponding to the interest level reference information may be associated with the interest level reference information stored in the memory unit 122. In this case, the reference information acquisition unit 305 may acquire, from the memory unit 122, interest level reference information associated with the position of the object of the type indicated by the attention object information that is closest to the position indicated by the attention object information.

[0054] The target object information may include identification information (e.g., an identifier) ​​for identifying (specifying) the target object. The interest level reference information stored in the memory unit 122 may be associated with identification information of an object corresponding to the interest level reference information. In this case, the reference information acquiring unit 305 may acquire, from the memory unit 122, interest level reference information associated with the same identification information as the identification information included in the target object information.

[0055] Note that a plurality of pieces of interest level reference information corresponding to the plurality of objects may be stored in the server 130. In this case, the reference information acquiring unit 305 may acquire the interest level reference information corresponding to the target object from the plurality of pieces of interest level reference information based on the target object information. The process of selecting the interest level reference information corresponding to the target object from the plurality of pieces of interest level reference information based on the target object information may be performed by the reference information acquiring unit 305 or the server 130. When the process of selecting the interest level reference information corresponding to the target object is performed by the server 130, for example, the reference information acquiring unit 305 transmits the target object information to the server 130, and the server 130 returns the interest level reference information corresponding to the target object to the reference information acquiring unit 305.

[0056] The target object detection unit 304 may detect a three-dimensional position of the target object based on the gaze information of the right eye of the user and the gaze information of the left eye of the user, and generate target object information indicating the three-dimensional position. For example, the target object detection unit 304 determines (estimates) a three-dimensional positional relationship between the user (HMD 110) and the target object based on the gaze information of the right eye, the gaze information of the left eye, and the interpupillary distance (the distance between the display unit 112 for the right eye and the display unit 112 for the left eye). Then, the target object detection unit 304 detects (estimates) a three-dimensional position of the target object based on the three-dimensional position of the user (HMD 110) and the determined positional relationship. The three-dimensional position of the user (HMD 110) is estimated using, for example, a Global Positioning System (GPS). By using the three-dimensional position, interest level reference information corresponding to the target object can be acquired with higher accuracy than when using the two-dimensional position (a malfunction in which interest level reference information of an object different from the target object is acquired can be suppressed). The method of estimating the three-dimensional position of the user (HMD 110) is not limited to the method using GPS. For example, the three-dimensional position of the user (HMD 110) may be estimated by SLAM (Simultaneous Localization and Mapping). The three-dimensional position of the user (HMD 110) may be estimated by odometry using an IMU (Inertial Measurement Unit).

[0057] When interest level reference information is prepared only for a specific type of object (e.g., advertising medium), the process of step S104 may be performed only when the type of the target object is a specific type (e.g., when the target medium is an advertising medium). By doing so, the processing load can be reduced.

[0058] In step S105, the interest level determination unit 306 determines the user's level of interest in the attention object based on the gaze information acquired in step S102, the attention object information acquired in step S103, and the interest level reference information acquired in step S104. The process of step S105 (interest level determination process) will be described later in detail.

[0059] In step S106, the interest level recording unit 307 records the interest level determined in step S105. The interest level may be transmitted to the server 130 via the Internet and recorded therein. The interest level may be recorded in the memory unit 122.

[0060] <Description of interest level determination process> The interest level determination process (the process of step S105 in FIG. 6) will be described with reference to Figures 8(A), 8(B), and 9. Figure 8(A) shows a target object, and Figure 8(B) shows interest level reference information corresponding to the target object in Figure 8(A). Figure 9 is a flowchart of the interest level determination process.

[0061] The object of interest in FIG. 8(A) is an advertising medium, and includes an eye-catching area, a text area, and an image area. The eye-catching area, located at the top center of the object of interest, contains a catchphrase that is not very relevant to the product being advertised. The text area, located at the bottom center of the object of interest, contains a description of the product being advertised and the name of the company that produces the product. The image area, located at the bottom right of the object of interest, contains an image of the product being advertised.

[0062] The interest level criterion information in FIG. 8(B) indicates a criterion for determining the user's interest in the target object for each of the multiple regions of the target object. The interest level criterion information in FIG. 8(B) indicates set regions 1 to 3 (the relative positions of set regions 1 to 3 with respect to the entire target object). Set region 1 corresponds to the image region, set region 2 corresponds to the eye-catching region, and set region 3 corresponds to the text region. Furthermore, the interest level criterion information in FIG. 8(B) indicates the type (image region / eye-catching region / text region), weight, and related region for each of set regions 1 to 3. In FIG. 8(B), set region 2 is set as the related region of set region 1, and related regions are not set for set regions 2 and 3.

[0063] When the interest level determination process in FIG. 9 starts, in step S201, the interest level determination unit 306 assigns 0 to the interest level I (initialization of the interest level I).

[0064] In step S202, the interest level determination unit 306 sets a sensitivity k for the interest level I. The higher the sensitivity k, the higher the interest level I is likely to be. For example, the interest level determination unit 306 sets a value according to the attributes of the user as the sensitivity k so as to make it easier to obtain a high value for the interest level I of users who are likely to be interested (or who are desired to be interested). When the object of interest is an advertising medium and the advertising target is cosmetics, the interest level determination unit 306 may set the sensitivity k so as to make it easier to obtain a high value for the interest level I of users of the age (age group) and gender targeted by the cosmetics.

[0065] In step S203, the interest level determination unit 306 determines whether the user's gaze stays in the set area 1 for a threshold time T1 or more. This determination can also be regarded as a determination of whether the user's gaze is directed toward the set area 1 for a predetermined threshold time T1 or more. For example, the attention object information indicates the area of ​​the attention object in the external image (the area of ​​the attention object on the display unit 112). The interest level determination unit 306 determines the set area 1 on the display unit 112 based on the attention object information and the interest level reference information of the attention object (relative arrangement of the set area 1 with respect to the entire attention object). Then, the interest level determination unit 306 determines whether the gaze stays in the set area 1 on the display unit 112 for a threshold time T1 or more based on the gaze information. If the interest level determination unit 306 determines that the user's gaze stays in the set area 1 for the threshold time T1 or more, the process proceeds to step S211, and if the interest level determination unit 306 determines that the user's gaze does not stay in the set area 1 for the threshold time T1 or more, the process proceeds to step S204.

[0066] In step S211, the interest level determination unit 306 updates the interest level I by using the interest level reference information of the target object (weight w1 of the set area 1). For example, the interest level determination unit 306 updates the interest level I using the following formula 4. I = I + (1 + k) × w1 (Equation 4)

[0067] In step S204, the interest level determination unit 306 determines whether or not a saccade has occurred, which moves the viewpoint into the set area 2. If the interest level determination unit 306 determines that a saccade has occurred into the set area 2, the process proceeds to step S212, and if the interest level determination unit 306 determines that a saccade has not occurred into the set area 2, the process proceeds to step S205.

[0068] In step S212, the interest level determination unit 306 uses the interest level reference information of the attention object (weight w2 of the set area 2) to update the interest level I. For example, the interest level determination unit 306 updates the interest level I using the following formula 5. I = I + (1 + k) × w2 (Equation 5)

[0069] In step S205, the interest level determination unit 306 determines whether the user's viewpoint remains in the set area 1 for the threshold time T1' or more and whether the user's viewpoint remains in the set area 2 for the threshold time T2 or more. If the interest level determination unit 306 determines that the user's viewpoint remains in the set area 1 for the threshold time T1' or more and that the user's viewpoint remains in the set area 2 for the threshold time T2 or more, the process proceeds to step S213; otherwise, the process proceeds to step S206.

[0070] In step S212, the interest level determination unit 306 uses the interest level reference information of the attention object (weights w1 and w2 of the set regions 1 and 2) to update the interest level I. For example, the interest level determination unit 306 updates the interest level I using the following formula 6. I = I + (1 + k) × (w1 + w2) (Eq. 6)

[0071] As described above, the eye-catching area corresponding to the set area 2 contains a catchphrase that is less relevant to the product being advertised. Therefore, just because a user has seen the set area 2 does not necessarily mean that the user has an interest in the target object. As in steps S204 and S205, by combining the set area 2 with other parts of the target object, it is possible to determine whether the user is interested in the target object. As described above, the set area 2 is set as a related area of ​​the set area 1. Therefore, in step S205, the set area 1 and the set area 2 are combined.

[0072] In step S206, the interest level determination unit 306 determines whether or not the user's gaze point remains in the set area 3 for a threshold time T3 or more. If the interest level determination unit 306 determines that the user's gaze point remains in the set area 3 for a threshold time T3 or more, the process proceeds to step S214, and if the interest level determination unit 306 determines that the user's gaze point does not remain in the set area 3 for a threshold time T3 or more, the interest level determination process in FIG. 9 is terminated.

[0073] In step S214, the interest level determination unit 306 uses the interest level reference information of the attention object (weight w3 of the set region 3) to update the interest level I. For example, the interest level determination unit 306 updates the interest level I using the following formula 7. I = I + (1 + k) × w3 (Equation 7)

[0074] The threshold times T1, T1', T2, and T3 may be fixed times determined in advance, or may be times that are appropriately changed. For example, the threshold times T1, T1', T2, and T3 may be times that are appropriately changed by the user, or may be times that are appropriately changed by the controller 120. The controller may determine the visibility of the target object based on an external image, and set the threshold times T1, T1', T2, and T3 to be longer as the visibility of the target object becomes lower. The threshold times T1, T1', T2, and T3 may be the same or different.

[0075] Although an example in which three set regions 1 to 3 are set has been described, the number of set regions is not particularly limited. The interest level determination process is not limited to the process shown in FIG. 9. The interest level reference information is not limited to the information shown in FIG. 8(B) as long as it indicates a standard for determining the user's interest level in an object. For example, the interest level reference information may indicate the order in which the set regions are to be viewed, or may indicate the time during which the set regions are desired to be viewed.

[0076] <Summary> As described above, according to the first embodiment, by acquiring from the storage unit and using the predetermined interest level reference information corresponding to the target object, it is possible to determine the user's interest level in the target object with high accuracy. As a result, it is possible to provide a service that presents to the user information on products in which the user is interested with high accuracy, and to acquire the number of people who are interested in the object with high accuracy.

[0077] <<Example 2>> A second embodiment of the present invention will be described. In the following, the same points as those in the first embodiment (for example, the same configuration and processing as those in the first embodiment) will be omitted, and only the points different from those in the first embodiment will be described. The overall processing of the second embodiment is the same as that of the first embodiment (FIG. 6). However, the reference information acquisition processing in step S104 is different between the first embodiment and the second embodiment. In the reference information acquisition processing in the first embodiment, interest level reference information corresponding to the target object is selected from a plurality of interest level reference information prepared in advance. In the reference information acquisition processing in the second embodiment, interest level reference information corresponding to the target object is acquired by analyzing an image of the target object. FIG. 10 is a flowchart of the reference information acquisition processing according to the second embodiment. The interest level determination processing in the second embodiment is the same as that in the first embodiment (FIG. 9).

[0078] When the reference information acquisition process in Fig. 10 starts, in step S301, the reference information acquisition unit 305 extracts the area of ​​the target object detected in step S103 from the external world image acquired in step S101 in Fig. 6. This causes an image of the target object to be acquired.

[0079] In steps S302 to S305, the reference information acquiring unit 305 analyzes the target object image acquired in step S301 to acquire interest level reference information corresponding to the target object.

[0080] In step S302, the reference information acquisition unit 305 performs semantic region division (segmentation) of the target object image acquired in step S301. By the process of step S302, setting regions 1 to 3 in FIG. 8B are set.

[0081] In step S303, the reference information acquisition unit 305 detects a characteristic part (e.g., a character or an object) from each set region set in step S302, Decide the type of setting area (image area / eye-catching area / text area).

[0082] In step S304, the reference information acquisition unit 305 evaluates the characteristic parts detected in step S303 for each set area set in step S302 to determine the weight of the set area and the related area. If the target object is an advertising medium, a large weight may be determined for a set area that is estimated to be in line with the gist of the advertisement, and a small weight may be determined for a set area that is estimated to be in line with the gist of the advertisement. For example, a large weight may be determined for an image area depicting a product that is the target of advertising, and a small weight may be determined for an eye-catching area that contains only a catchphrase that has little relevance to the product. Then, a medium weight may be determined for a text area that contains a description of the product.

[0083] In step S305, the reference information acquiring unit 305 acquires interest level reference information corresponding to the target object by integrating the process results of steps S302 to S304.

[0084] The reference information acquisition unit 305 may use a calculator (trained model) that inputs a target object image and outputs an analysis result of the target object image (for example, a processing result of any one of steps S302 to S305). The calculator may be used for one of the processes of steps S302 to S305, or for two or more of the processes. The CPU 121 may function as the calculator, or the controller 120 may have a GPU (Graphics Processing Unit) that functions as the calculator.

[0085] The server 130 may analyze the target object image and acquire interest level reference information corresponding to the target object. In this case, for example, the reference information acquisition unit 305 transmits the outside world image or the target object image to the server 130, and the server 130 returns the interest level reference information corresponding to the target object to the reference information acquisition unit 305.

[0086] As described above, according to the second embodiment, interest level reference information corresponding to the target object is obtained by analyzing the target object image, and the obtained interest level reference information is used to determine the user's interest level in the target object with high accuracy.

[0087] The above embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above embodiment.

[0088] <<Other Examples>> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0089] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) A first acquisition means for acquiring gaze information of a user; A detection means for detecting an object that the user is looking at based on the line of sight information; A second acquisition means for acquiring, from a storage unit, reference information for determining a degree of interest of the user in the object, the reference information being predetermined information corresponding to the object; a determination means for determining the interest level based on the line of sight information and the reference information; 13. An information processing device comprising: (Configuration 2) A first acquisition means for acquiring gaze information of a user; A detection means for detecting an object that the user is looking at based on the line of sight information; A second acquisition means for acquiring reference information for determining a degree of interest of the user in the object by analyzing an image of the object; a determination means for determining the interest level based on the line of sight information and the reference information; 13. An information processing device comprising: (Configuration 3) The second acquisition means acquires the reference information using a trained model that inputs the image and outputs an analysis result of the image. 3. The information processing device according to configuration 2. (Configuration 4) The first acquisition means acquires gaze information of the user's right eye and gaze information of the user's left eye, The detection means detects a three-dimensional position of the object based on line-of-sight information of the user's right eye and line-of-sight information of the user's left eye. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) A recording means for recording the degree of interest in a server. Further having 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The reference information indicates a criterion for determining the interest level for each of a plurality of regions of the object. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) An information processing device according to any one of configurations 1 to 6, A display means; A head-mounted display device comprising: (Method 1) Acquiring user gaze information; detecting an object that the user is looking at based on the gaze information; acquiring, from a storage unit, reference information for determining a degree of interest of the user in the object, the reference information being predetermined information corresponding to the object; determining the interest level based on the line of sight information and the reference information; 13. A method for controlling an information processing apparatus comprising the steps of: (Method 2) Acquiring user gaze information; detecting an object that the user is looking at based on the gaze information; obtaining reference information for determining a degree of interest of the user in the object by analyzing an image of the object; determining the interest level based on the line of sight information and the reference information; 13. A method for controlling an information processing apparatus comprising the steps of: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 6. (medium) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 6. [Explanation of symbols]

[0090] 120: Controller 121: CPU 301: Gaze detection unit 304: Attention object detection unit 305: Reference information acquisition unit 306: Interest level determination unit

Claims

1. a first acquisition means for acquiring gaze information of a user; a detection means for detecting an object that the user is looking at based on the line-of-sight information; a second acquiring means for acquiring, from a storage unit, reference information for determining the user's interest in the object, the reference information being predetermined information corresponding to the object; a determination means for determining the interest level based on the line-of-sight information and the reference information; An information processing device comprising:

2. the first acquisition means acquires gaze information of the user's right eye and gaze information of the user's left eye, The detection means detects the three-dimensional position of the object based on line-of-sight information of the user's right eye and line-of-sight information of the user's left eye.

2. The information processing apparatus according to claim 1, wherein:

3. A recording means for recording the interest level in a server. Further having 2. The information processing apparatus according to claim 1, wherein:

4. The reference information indicates a criterion for determining the interest level for each of a plurality of regions of the object.

2. The information processing apparatus according to claim 1, wherein:

5. An information processing device according to any one of claims 1 to 4; A display means; A head-mounted display device comprising:

6. acquiring user gaze information; detecting an object that the user is looking at based on the gaze information; acquiring, from a storage unit, reference information for determining the user's interest in the object, the reference information being predetermined information corresponding to the object; determining the interest level based on the line of sight information and the reference information; 1. A method for controlling an information processing device, comprising:

7. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 4.

8. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 4.

Images