Information processing apparatus

Abstract

An information processing apparatus according to the present invention comprises: a first acquisition unit configured to acquire line-of-sight information of a user; a detection unit configured to detect an object viewed by the user based on the line-of-sight information; a second acquisition unit configured to acquire, from the storage unit, reference information for determining a degree of interest of the user with respect to the object, the reference information being predetermined information corresponding to the object; and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information.

Description

Technical Field

[0001] The present invention relates to an information processing device, and more particularly to a technology for judging the degree of interest a person has in an object. Background Art

[0002] A technology for judging the degree of interest of a person in an object based on the person's line of sight information is proposed. Patent document 1 discloses a technology for counting the number of people who view an advertisement displayed on an advertisement display device based on the line of sight directions of people present in an area where the advertisement display device is installed. Patent document 2 discloses a technology for judging the degree of interest of a passenger (a passenger of a moving body) in an object to be viewed based on the duration of the passenger's viewing of the object to be viewed.

[0003] [Citation list]

[0004] [Patent Document]

[0005] [Patent Document 1] JP 2008-112401 A

[0006] [Patent Document 2] JP 2007-172378 A Summary of the invention

[0007] Problem that the invention aims to solve

[0008] In the prior art (for example, the techniques disclosed in Patent Documents 1 and 2), the degree of interest of a person in an object is judged by regarding the behavior of a person looking at the object as the behavior of the person being interested in the object. However, it cannot be considered that a person is interested in an object simply because the person looks at the object. Therefore, in the prior art, the degree of interest cannot be judged with high accuracy.

[0009] An object of the present invention is to provide a technique capable of judging a person's degree of interest in an object with high accuracy.

[0010] Solutions for solving problems

[0011] In its first aspect, the present invention provides an information processing device, comprising: a first acquisition unit, which is configured to acquire a user's line of sight information; a detection unit, which is configured to detect the object viewed by the user based on the line of sight information; a second acquisition unit, which is configured to acquire benchmark information for judging the user's interest level in the object from a storage unit, wherein the benchmark information is predetermined information corresponding to the object; and a judgment unit, which is configured to judge the interest level based on the line of sight information and the benchmark information.

[0012] In its second aspect, the present invention provides an information processing device, comprising: a first acquisition unit, which is configured to acquire a user's line of sight information; a detection unit, which is configured to detect an object viewed by the user based on the line of sight information; a second acquisition unit, which is configured to acquire benchmark information for judging the user's interest level in the object by analyzing an image of the object; and a judgment unit, which is configured to judge the interest level based on the line of sight information and the benchmark information.

[0013] In its third aspect, the present invention provides a control method for an information processing device, comprising: a step for acquiring a user's line of sight information; a step for detecting an object viewed by the user based on the line of sight information; a step for acquiring benchmark information for judging the user's interest level in the object from a storage unit, the benchmark information being predetermined information corresponding to the object; and a step for judging the interest level based on the line of sight information and the benchmark information.

[0014] In its fourth aspect, the present invention provides a control method for an information processing device, comprising: a step for acquiring a user's line of sight information; a step for detecting an object viewed by the user based on the line of sight information; a step for acquiring benchmark information for judging the user's interest level in the object by analyzing an image of the object; and a step for judging the interest level based on the line of sight information and the benchmark information.

[0015] The present invention provides a program for causing a computer to function as each unit of the above-mentioned information processing apparatus in a fifth aspect. The present invention provides a computer-readable medium storing a program for causing a computer to function as each unit of the above-mentioned information processing apparatus in a sixth aspect.

[0016] Effects of the Invention

[0017] According to the present invention, the degree of interest of a person in an object can be determined with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A and Figure 1 B is a diagram showing the appearance of the system.

[0019] Figure 2 is a block diagram showing the electrical configuration of a display system.

[0020] Figure 3 It is a diagram for describing the principle of the sight line detection method.

[0021] Figure 4 A is a diagram showing an eye image, and Figure 4B is a diagram showing the brightness distribution of the eye image.

[0022] Figure 5 This is a flowchart of the line of sight detection processing.

[0023] Figure 6 It is a flowchart of the overall process.

[0024] Figure 7 is a block diagram showing the functional configuration of a display system.

[0025] Figure 8 A is a diagram showing an object of interest, and Figure 8 B is a diagram showing interest level reference information.

[0026] Fig. 9 This is a flowchart of the interest level determination process.

[0027] Fig.10 is a flowchart of a reference information acquisition process according to the second embodiment. DETAILED DESCRIPTION

[0028] <<First Embodiment>>

[0029] A first embodiment of the present invention is described.

[0030] Note that the present invention is applicable to various electronic devices that can obtain a person's line of sight information (information related to the line of sight). For example, the present invention is applicable to both a video perspective display device and an optical perspective display device. The video perspective display device displays an image (virtual space) obtained by shooting the real space on a display surface (a display surface that does not transmit light from the real space (the outside world)) by synthesizing graphics (for example, virtual objects) as needed. In this case, the user cannot directly view the real space, but can indirectly view the real space or view graphics synthesized with the image of the real space by viewing the displayed image. The optical perspective display device displays graphics on a display surface (a display surface that transmits light from the real space), for example. In this case, the user can directly view the real space through the display surface or view the graphics displayed on the display surface.

[0031] The present invention is applicable to both head-mounted display devices and other display devices. For example, the present invention is also applicable to handheld display devices and fixed display devices. A head-mounted display device is, for example, smart glasses (augmented reality (AR) glasses) or a head-mounted display (HMD). A handheld display device is, for example, a smart phone or a tablet terminal. A display device held by a user's hand and worn on (applied to) the head is a handheld display device and is also a head-mounted display device. A smart phone mounted on a head-mounted adapter (for example, virtual reality (VR) goggles) is a head-mounted display device. The present invention is applicable to both a head-mounted display device in which a user views an image with both eyes and a head-mounted display device in which a user views an image with one eye.

[0032] The present invention is also applicable to a method that allows a user to visually recognize only a virtual space but does not allow the user to visually recognize a real space. That is, the present invention is applicable to various types of cross reality (XR), such as augmented reality (AR), mixed reality (MR), and virtual reality (VR).

[0033] The present invention is also applicable to electronic devices other than display devices. The information processing device to which the present invention is applied may be provided in the display device, or may be provided in an electronic device separated from the display device. For example, the present invention is also applicable to a controller or a personal computer (PC) connected to the display device. The present invention is also applicable to a surveillance camera.

[0034] In the first embodiment, the sight line information of a person is obtained, and the degree of interest of the person in the object is judged based on the sight line information. In the first embodiment, the object is a real object, but the object may be a virtual object. In the prior art, the degree of interest of a person in an object is judged by regarding the behavior of a person looking at the object as the behavior of the person being interested in the object. However, it cannot be considered that the person is interested in the object simply because the person looks at the object. Therefore, in the prior art, the degree of interest cannot be judged with high accuracy.

[0035] For example, advertising media (e.g., posters, signs, and digital signs) may have a large eye-catching area (eye-catching area). The eye-catching area is an area where, for example, a photo of a person (advertising model) or a slogan is arranged in a large size. Here, the eye-catching area may have a low correlation with the advertising object (e.g., a product or service). Therefore, even if a person looks at the eye-catching area, the person may not associate it with the advertising object and may not be interested in the advertising object and the advertising media. Note that the area with low correlation with the advertising object is not limited to the eye-catching area.

[0036] Therefore, in the first embodiment, the degree of interest of a person in an object is determined with high accuracy based on an appropriate determination criterion corresponding to the object. For example, the degree of interest is determined with high accuracy by the determination criterion in consideration of the content of the object (the design intention of the object).

[0037] <Configuration Description>

[0038] Figure 1 A and Figure 1 B shows the appearance of the display system 100 according to the first embodiment. Figure 1 A is a frontal stereogram, and Figure 1 B is the back stereogram. Figure 1 As shown in FIG. 1A , the display system 100 includes an HMD 110, a controller 120, and a server 130. In the first embodiment, it is assumed that the HMD 110 is used as a video see-through HMD. The controller 120 is connected to the HMD 110 in a wired manner and is connected to the server 130 in a wireless manner. The controller 120 may be connected to the HMD 110 wirelessly. Figure 2 is a block diagram showing the electrical configuration of the display system 100 .

[0039] like Figure 1 A. Figure 1 B and Figure 2 As shown, the HMD 110 includes a camera lens 111, a display unit 112, light sources 113a and 113b, a light receiving lens 114, an eye camera element 115, and a knob 140. Figure 2 As shown in FIG. 1 , the HMD 110 includes an imaging element 116, a light measuring circuit 117, a light source driving circuit 118, and a display unit driving circuit 119. Figure 1 As shown in FIG. 1B , a display unit 112 , light sources 113 a and 113 b , a light receiving lens 114 , and an eye imaging element 115 are provided for each of the right eye and the left eye of the user.

[0040] The imaging lens 111 is a lens for imaging the outside world, and the imaging element 116 is an imaging element for imaging the outside world. The imaging element 116 is arranged on a predetermined imaging plane of the imaging lens 111 .

[0041] The display unit 112 displays various images (information). For example, the display unit 112 displays an image of the outside world captured by the imaging element 116, or displays information related to an object viewed by the user. The display unit drive circuit 119 is controlled by the controller 120 (CPU 121 described below) to drive the display unit 112.

[0042] The light source 113a and the light source 113b are each a light source for irradiating the user's eyes, for example, an infrared light emitting diode that emits infrared light to which the user is not sensitive. The light source driving circuit 118 is controlled by the controller 120 (CPU 121) to drive the light sources 113a and 113b. A portion of the light emitted from the light sources 113a and 113b and reflected on the user's eyes is focused on the eye imaging element 115 by the light receiving lens 114. The light receiving lens 114 is a lens for capturing the user's eyes, and the eye imaging element 115 is an imaging element for capturing the user's eyes.

[0043] The knob 140 is a knob for adjusting the interval between the right-eye display unit 112 and the left-eye display unit 112 to match the distance between the user's pupils.

[0044] The photometric circuit 117 amplifies, logarithmically compresses, and A / D converts a signal obtained from the imaging element 116 also serving as a photometric sensor (specifically, a brightness signal corresponding to the brightness of the object field), and sends the result as object field brightness information to the controller 120 (CPU 121).

[0045] The controller 120 includes a CPU 121 , a memory unit 122 , a visual line detection circuit 123 , a low-pass filter (LPF) 124 , a display unit drive circuit 125 , operation members 126 to 128 , and a wireless communication circuit 129 .

[0046] The CPU 121 is a central processing unit of a microcomputer included in the controller 120 , and controls the entire display system 100 .

[0047] The memory unit 122 has a function of storing a video signal from the eye imaging element 115 and a function of storing a sight line correction parameter. The sight line correction parameter is a parameter for correcting individual differences in sight lines. In addition, the memory unit 122 has a function of storing interest level reference information. The interest level reference information is predetermined information corresponding to an object, and is reference information for judging the user's interest level in the object. The memory unit 122 stores a plurality of interest level reference information corresponding to a plurality of objects, respectively.

[0048] The sight line detection circuit 123 performs A / D conversion on the output of the eye imaging element 115 (eye image obtained by imaging the eye) in a state where the optical image of the eye is formed on the eye imaging element 115, and transmits the result to the CPU 121 via the LPF 124. The CPU 121 extracts feature points required for detecting the sight line from the eye image according to a predetermined algorithm described below, and detects the user's sight line from the positions of the feature points.

[0049] The operating members 126 to 128 receive operations from the user and output operation signals (signals corresponding to the operations performed by the user) to the CPU 121. For example, the operating member 126 is a touch panel capable of receiving touch operations, the operating member 127 is an operating rod capable of being pushed down in various directions, and the operating member 128 is a four-way key capable of being pushed down in various directions in four directions. The operating member 126 (touch panel) has a function of displaying an image (information). As described above, the operating member 126 has a function as a touch panel and a function as a display unit. The display unit drive circuit 125 is controlled by the CPU 121 and drives the operating member 126 (display unit). The user can use the operating members 126 to 128 to perform various operations (instructions). For example, the user can use the operating members 126 to 128 to fine-tune the position of the user interface (UI, for example, an indicator) displayed on the display unit 112.

[0050] The wireless communication circuit 129 is controlled by the CPU 121 and communicates with an external device. For example, the wireless communication circuit 129 transmits and records the degree of interest (the degree of user's interest in the object) determined by the CPU 121 to the server 130 via the Internet.

[0051] <Description of Line of Sight Detection Processing>

[0052] refer to Figure 3 , Figure 4 A. Figure 4 B and Figure 5 The line of sight detection processing (line of sight detection method) is described. The line of sight of the right eye and the line of sight of the left eye are detected by the following line of sight detection method. Figure 3 is a diagram showing the principle of a sight line detection method, and is a schematic diagram of an optical system for detecting sight lines. Figure 3 As shown, light sources 113a and 113b are arranged substantially symmetrically with respect to the optical axis of light receiving lens 114 and illuminate the user's eyeball 200. Part of the light emitted from light sources 113a and 113b and reflected on eyeball 200 is condensed on eye imaging element 115 by light receiving lens 114. Figure 4 A is a schematic diagram of an eye image captured by the eye imaging element 115 (optical image of the eye projected on the eye imaging element 115), and Figure 4 B is a graph showing the output intensity of the eye imaging element 115 . Figure 5 This is a flowchart of the line of sight detection processing.

[0053] when Figure 5When the line of sight detection processing in the embodiment starts, in step S1, the CPU 121 drives the light sources 113a and 113b using the light source driving circuit 118 to emit infrared light to the user's eyeball 200. The optical image of the user's eye irradiated by the infrared light is formed on the eye imaging element 115 through the light receiving lens 114, and is photoelectrically converted by the eye imaging element 115. Thus, an electrical signal of the eye image that can be processed is obtained.

[0054] In step S2 , the CPU 121 acquires an eye image (image data, image signal) from the eye imaging element 115 via the visual line detection circuit 123 .

[0055] In step S3, the CPU 121 detects the coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 113a and 113b and the pupil center c from the eye image obtained in step S2.

[0056] The infrared light emitted from the light sources 113a and 113b irradiates the cornea 201 of the user's eyeball 200. At this time, the cornea reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 201 are converged by the light receiving lens 114 and formed on the eye imaging element 115 to become cornea reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the ends a and b of the pupil 202 are also imaged on the eye imaging element 115 and become pupil end images a' and b' in the eye image.

[0057] Figure 4 B shows Figure 4 Brightness information (brightness distribution) of area α in the eye image of A. Figure 4 B shows the brightness distribution in the X-axis direction, with the horizontal direction of the eye image as the X-axis direction and the vertical direction as the Y-axis direction. In the first embodiment, the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil end images a' and b' in the X-axis direction are Xa and Xb. Figure 4 As shown in FIG. 1B , at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe', extremely high levels of brightness are obtained. In the region from the coordinate Xa to the coordinate Xb corresponding to the region of the pupil 202 (the region of the pupil image obtained by imaging the light beam from the pupil 202 on the eye imaging element 115), brightness having an extremely low level is obtained except for the coordinates Xd and Xe. Then, in the region of the iris 203 outside the pupil 202 (the region of the iris image outside the pupil image obtained by imaging the light beam from the iris 203), intermediate brightness between the above two types of brightness is obtained. For example, in the region where the X coordinate (coordinate in the X-axis direction) is greater than the coordinate Xa and the region where the X coordinate is less than the coordinate Xb, intermediate brightness between the two types of brightness is obtained.

[0058] The X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil end images a' and b' can be obtained from Figure 4 . For example, coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and coordinates with extremely low brightness can be obtained as the coordinates of the pupil end images a' and b'. When the rotation angle θx of the optical axis of the eyeball 200 relative to the optical axis of the light receiving lens 114 is small, the coordinate Xc of the pupil center image c' (the center of the pupil image) obtained by imaging the light beam from the pupil center c on the eye imaging element 115 can be expressed as Xc≈(Xa+Xb) / 2. That is, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil end images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0059] In step S4, the CPU 121 calculates the imaging magnification β of the eye image. The imaging magnification β is a magnification determined by the position of the eyeball 200 relative to the light receiving lens 114 and can be calculated using a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0060] In step S5, the CPU 121 calculates the rotation angle of the optical axis of the eyeball 200 relative to the optical axis of the light receiving lens 114. The X coordinate of the midpoint between the cornea reflection image Pd and the cornea reflection image Pe is substantially consistent with the X coordinate of the center of curvature O of the cornea 201. Therefore, when the standard distance from the center of curvature O of the cornea 201 to the center c of the pupil 202 is Oc, the rotation angle θx of the eyeball 200 in the ZX plane (a plane perpendicular to the Y axis) can be calculated by the following formula 1. The rotation angle θy of the eyeball 200 in the ZY plane (a plane perpendicular to the X axis) can also be calculated by a method similar to the method of calculating the rotation angle θx.

[0061] β×Oc×SINθx≈{(Xd+Xe) / 2}-Xc···(Formula 1)

[0062] In step S6, the CPU 121 reads parameters required for detecting the line of sight from the memory unit 122. For example, the CPU 121 reads parameters m, Ax, Bx, Ay, By, nx, and ny. The parameter m is a constant determined by the configuration of the optical system for performing the line of sight detection processing, and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c on the display unit 112. It is assumed that the parameter m is predetermined and stored in the memory unit 122. The parameters Ax, Bx, Ay, and By are line of sight correction parameters for correcting individual differences in the line of sight, and are acquired by performing line of sight detection calibration. The parameter Ax is an offset value in the X-axis direction, the parameter Bx is a sensitivity coefficient in the X-axis direction, the parameter Ay is an offset value in the Y-axis direction, and the parameter By is a sensitivity coefficient in the Y-axis direction. It is assumed that before the line of sight detection processing starts, the line of sight correction parameters Ax, Bx, Ay, and By are stored in the memory unit 122. In the line of sight detection processing for the right eye, parameters nx=nRx and parameters ny=nRy are obtained, and in the line of sight detection processing for the left eye, parameters nx=nLx and parameters ny=nLy are obtained. Parameter nRx is a correction coefficient in the X-axis direction for obtaining the line of sight information of the right eye, and parameter nRy is a correction coefficient in the Y-axis direction for obtaining the line of sight information of the right eye. Parameter nLx is a correction coefficient in the X-axis direction for obtaining the line of sight information of the left eye, and parameter nLy is a correction coefficient in the Y-axis direction for obtaining the line of sight information of the left eye. It is assumed that parameters nRx, nRy, nLx, and nLy are predetermined and stored in the memory unit 122.

[0063] In step S7, the CPU 121 estimates the viewpoint of the user on the display unit 112 using the rotation angles θx and θy calculated in step S5 and the parameters m, Ax, Bx, Ay, By, nx, and ny read in step S6. The viewpoint can also be regarded as a position where the line of sight is focused, a position where the user is looking, or a line of sight position. Assuming that the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the center of the pupil c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3.

[0064] Hx=m×(Ax×θx+Bx)×nx···(Formula 2)

[0065] Hy=m×(Ay×θy+By)×ny···(Formula 3)

[0066] In step S8 , the CPU 121 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 122 , and ends the visual line detection process.

[0067] Note that the line of sight detection method is not limited to the above method, and for example, any method may be used as long as the method is a method of obtaining line of sight information from an eye image. The line of sight may be detected by a method that does not use an eye image, such as a method of detecting eye potential without using an eye image and detecting the line of sight based on the eye potential. As final line of sight information, information indicating the line of sight direction (direction of the line of sight) may be obtained instead of information indicating the viewpoint. For example, processing may be performed until the rotation angle (Ax×θx+Bx)×nx, (Ay×θy+By)×ny is obtained without obtaining the coordinates (Hx, Hy) of the viewpoint. As final line of sight information, information indicating the viewpoint of the two eyes (the intersection of the right eye line of sight and the left eye line of sight, the position viewed by the two eyes) may be obtained.

[0068] <Description of overall processing>

[0069] Reference Figure 6 and Figure 7 The overall process of the display system 100 (controller 120 ) is described. Figure 6 1 is a flowchart of the overall process. For example, when the user wears the HMD 110 and starts the HMD 110 and the controller 120, the process starts. Figure 6 overall processing. Figure 7 is a block diagram showing a functional configuration of the display system 100 . Figure 7 Each functional unit in the controller 120 is implemented by the CPU 121.

[0070] when Figure 6 When the overall processing of is started, in step S101, the outside world image acquisition unit 302 acquires an image of the outside world (outside world image) captured by the imaging element 116. Figure 6 and Figure 7 Although not shown in the figure, the CPU 121 performs predetermined image processing on the acquired external image and displays the image on the display unit 112.

[0071] In step S102, the sight line detection unit 301 performs Figure 5 The user's gaze information is obtained through gaze detection processing.

[0072] In step S103, the CPU 121 detects the object (object of interest) viewed by the user based on the line of sight information acquired in step S102. For example, the object detection unit 303 detects the object from the external image acquired in step S101. Then, the object of interest detection unit 304 detects the object of interest from the objects detected by the object detection unit 303 based on the line of sight information acquired in step S102. For example, the object of interest is detected based on information obtained from multiple time series line of sight information (for example, the trajectory of the viewpoint, the saccade of the eyeball, the duration of the viewpoint stay, and the number of times the object is looked at). Through the processing of step S103, the object of interest information as the detection result of the object of interest is acquired. The details of the object of interest information are described below.

[0073] In step S104, the reference information acquisition unit 305 acquires interest level reference information corresponding to the object of interest based on the object of interest information acquired in step S103 (reference information acquisition processing). In the first embodiment, the reference information acquisition unit 305 acquires interest level reference information corresponding to the object of interest from the memory unit 122. The details of the interest level reference information are described below.

[0074] For example, it is assumed that the attention object information indicates the position of the attention object (the position in the real space, the position of the attention object in the coordinate system of the real space (the world coordinate system)). Then, it is assumed that the position of the object corresponding to the interest level reference information is associated with the interest level reference information stored in the memory unit 122. In this case, the reference information acquisition unit 305 can acquire, from the memory unit 122, the interest level reference information associated with the position closest to the position indicated by the attention object information.

[0075] The object of interest information may indicate the position and type of the object of interest. Then, the position and type of the object corresponding to the interest level reference information may be associated with the interest level reference information stored in the memory unit 122. In this case, the reference information acquisition unit 305 may acquire, from the memory unit 122, the interest level reference information associated with the position closest to the position indicated by the object of interest information among the objects of the type indicated by the object of interest information.

[0076] The object of interest information may include identification information (e.g., an identifier) ​​for identifying (specifying) the object of interest. Then, the identification information of the object corresponding to the interest level reference information may be associated with the interest level reference information stored in the memory unit 122. In this case, the reference information acquisition unit 305 may acquire, from the memory unit 122, the interest level reference information associated with the same identification information as the identification information included in the object of interest information.

[0077] A plurality of interest level reference information corresponding to a plurality of objects may be stored in the server 130. In this case, the reference information acquisition unit 305 may acquire the interest level reference information corresponding to the object of interest from the server 130. The process of selecting the interest level reference information corresponding to the object of interest from the plurality of interest level reference information based on the plurality of object of interest information may be performed by the reference information acquisition unit 305, or may be performed by the server 130. For example, when the server 130 performs the process of selecting the interest level reference information corresponding to the object of interest, the reference information acquisition unit 305 transmits the object of interest information to the server 130, and the server 130 returns the interest level reference information corresponding to the object of interest to the reference information acquisition unit 305.

[0078] The object of interest detection unit 304 can detect the three-dimensional position of the object of interest based on the line of sight information of the user's right eye and the line of sight information of the user's left eye, and generate the object of interest information indicating the three-dimensional position. For example, the object of interest detection unit 304 determines (estimates) the three-dimensional position relationship between the user (HMD 110) and the object of interest based on the line of sight information of the right eye, the line of sight information of the left eye, and the distance between the pupils (the distance between the display unit 112 of the right eye and the display unit 112 of the left eye). Then, the object of interest detection unit 304 detects (estimates) the three-dimensional position of the object of interest based on the three-dimensional position of the user (HMD 110) and the determined position relationship. The three-dimensional position of the user (HMD 110) is estimated using, for example, a global positioning system (GPS). By using the three-dimensional position, the interest level reference information corresponding to the object of interest can be acquired with higher accuracy than the case of using the two-dimensional position (the erroneous operation of acquiring the interest level reference information of an object different from the object of interest can be prevented). Note that the method of estimating the three-dimensional position of the user (HMD 110) is not limited to the method using the GPS. For example, the three-dimensional position of the user (HMD 110) can be estimated by simultaneous localization and mapping (SLAM). The three-dimensional position of the user (HMD 110) can be estimated by odometry using an inertial measurement unit (IMU).

[0079] When interest level reference information is prepared only for a specific type of object (eg, advertising media), the processing of step S104 may be performed only when the type of the object of interest is a specific type (eg, when the media of interest is advertising media). By doing so, the processing load can be reduced.

[0080] In step S105, the interest level judgment unit 306 judges the user's interest level in the object of interest based on the line of sight information acquired in step S102, the object of interest information acquired in step S103, and the interest level reference information acquired in step S104. The details of the processing of step S105 (interest level judgment processing) are described below.

[0081] In step S106, the interest level recording unit 307 transmits and records the interest level determined in step S105 to the server 130 via the Internet. The interest level may be recorded in the memory unit 122.

[0082] <Description of Interest Degree Judgment Processing>

[0083] Reference Figure 8 A. Figure 8 B and Fig. 9 Describe the interest level judgment process ( Figure 6 , (processing in step S105 in ). Figure 8 A shows the object of interest, and Figure 8 B shows the Figure 8 The interest level benchmark information corresponding to the object of interest in A. Fig. 9 This is a flowchart of the interest level determination process.

[0084] Figure 8 The object of interest in A is an advertising medium and includes a striking area, a text area, and an image area. In the striking area arranged in the upper center of the object of interest, a slogan with low relevance to the product to be advertised is described. The description of the product to be advertised and the company name of the product are described in the text area arranged in the lower center of the object of interest. The image of the product to be advertised is drawn in the image area arranged in the lower right part of the object of interest.

[0085] Figure 8 The interest level reference information in B indicates a reference for judging the user's interest level in the object of interest for each of the plurality of regions of the object of interest. Figure 8 The interest level reference information in B indicates setting areas 1 to 3 (relative arrangement of each setting area 1 to 3 with respect to the entire object of interest). Setting area 1 corresponds to the image area, setting area 2 corresponds to the eye-catching area, and setting area 3 corresponds to the text area. In addition, Figure 8 The interest level reference information in B indicates the type (image area / eye-catching area / text area), weight, and related area for each of the setting areas 1 to 3. Figure 8 In B, setting area 2 is set as a related area of ​​setting area 1, and a related area is not set in setting areas 2 and 3.

[0086] when Fig. 9 When the interest level determination processing starts, in step S201, the interest level determination unit 306 substitutes 0 into the interest level I (initialization of the interest level I).

[0087] In step S202, the interest level judgment unit 306 sets the sensitivity k of the interest level I. As the sensitivity k is larger, the interest level I tends to be larger. For example, the interest level judgment unit 306 sets a value corresponding to the attribute of the user as the sensitivity k so that a large value can be easily obtained as the interest level I of the user who may be interested (expected to be interested). When the object of interest is an advertising medium, and the advertising object is a cosmetic, the interest level judgment unit 306 can set the sensitivity k so that a large value can be easily obtained as the interest level of the user of the age (age group) and gender targeted by the cosmetic.

[0088] In step S203, the interest level judgment unit 306 judges whether the user's viewpoint has stayed in the setting area 1 for a threshold time period T1 or longer. This judgment can also be regarded as a judgment related to whether the user's line of sight is directed to the setting area 1 for a predetermined threshold time period T1 or longer. For example, the attention object information indicates the area of ​​the attention object in the external image (the area of ​​the attention object on the display unit 112). The interest level judgment unit 306 judges the setting area 1 on the display unit 112 based on the attention object information and the interest level reference information of the attention object (the relative arrangement of the setting area 1 relative to the entire attention object). Then, the interest level judgment unit 306 judges whether the viewpoint has stayed in the setting area 1 on the display unit 112 for a threshold time period T1 or longer based on the line of sight information. When it is judged that the user's viewpoint has stayed in the setting area 1 for a threshold time period T1 or longer, the interest level judgment unit 306 proceeds to step S211, and when it is judged that the user's viewpoint has not stayed in the setting area 1 for a threshold time period T1 or longer, the process proceeds to step S204.

[0089] In step S211, the interest level judgment unit 306 uses the interest level reference information of the object of interest (the weight w1 of the setting area 1) to update the interest level I. As the weight w1 is larger, the interest level judgment unit 306 updates the interest level I to a larger value. For example, the interest level judgment unit 306 updates the interest level I using the following formula 4.

[0090] I=I+(1+k)×w1···(Formula 4)

[0091] In step S204, the interest level determination unit 306 determines whether a pan in which the viewpoint moves in the setting area 2 occurs. When it is determined that a pan to the setting area 2 occurs, the interest level determination unit 306 proceeds to step S212, and when it is determined that a pan to the setting area 2 does not occur, the interest level determination unit 306 proceeds to step S205.

[0092] In step S212, the interest level judgment unit 306 uses the interest level reference information of the attention object (the weight w2 of the setting area 2) to update the interest level I. For example, the interest level judgment unit 306 updates the interest level I using Formula 5 below.

[0093] I=I+(1+k)×w2···(Formula 5)

[0094] In step S205, the interest level judgment unit 306 judges whether the user's viewpoint has stayed in the setting area 1 for a threshold time period T1' or longer and whether the user's viewpoint has stayed in the setting area 2 for a threshold time period T2 or longer. When it is judged that the user's viewpoint has stayed in the setting area 1 for a threshold time period T1' or longer and the user's viewpoint has stayed in the setting area 2 for a threshold time period T2 or longer, the interest level judgment unit 306 advances the processing to step S213, otherwise, advances the processing to step S206.

[0095] In step S212, the interest level judgment unit 306 updates the interest level I using the interest level reference information of the attention object (setting weights w1 and w2 of regions 1 and 2). For example, the interest level judgment unit 306 updates the interest level I using Formula 6 below.

[0096] I=I+(1+k)×(w1+w2)···(Formula 6)

[0097] As described above, in the eye-catching area corresponding to the setting area 2, a slogan with low relevance to the product to be advertised is described. Therefore, even if the user looks at the setting area 2, the user is not necessarily interested in the object of interest. As in steps S204 and S205, the setting area 2 is combined with other parts of the object of interest, thereby determining whether the user is interested in the object of interest. As described above, the setting area 2 is set as a related area of ​​the setting area 1. Therefore, in step S205, the setting area 1 and the setting area 2 are combined.

[0098] In step S206, the interest level judgment unit 306 judges whether the user's viewpoint has stayed in the setting area 3 for a threshold time period T3 or longer. When it is judged that the user's viewpoint has stayed in the setting area 3 for a threshold time period T3 or longer, the interest level judgment unit 306 proceeds to step S214, and when it is judged that the user's viewpoint has not stayed in the setting area 3 for a threshold time period T3 or longer, the interest level judgment unit 306 ends the process. Fig. 9 The interest level judgment process in .

[0099] In step S214, the interest level judgment unit 306 uses the interest level reference information of the attention object (the weight w3 of the setting area 3) to update the interest level I. For example, the interest level judgment unit 306 updates the interest level I using Formula 7 below.

[0100] I=I+(1+k)×w3···(Formula 7)

[0101] Note that the threshold time periods T1, T1', T2, and T3 may be predetermined fixed time periods, or may be appropriately variable time periods. For example, the threshold time periods T1, T1', T2, and T3 may be time periods appropriately changed by the user, or may be time periods appropriately changed by the controller 120. The controller may determine the visibility of the object of interest based on the external image, and as the visibility of the object of interest becomes lower, a longer time period is set as the threshold time periods T1, T1', T2, and T3. The threshold time periods T1, T1', T2, and T3 may be the same as or different from each other.

[0102] Although an example of setting three setting areas 1 to 3 is described, the number of setting areas is not particularly limited. Fig. 9 The interest level reference information only needs to indicate a reference for judging the user's interest level in the object, and is not limited to Figure 8 The information shown in B. For example, the interest level reference information may indicate the order in which the setting areas are viewed, or may indicate a time period during which the setting areas are desired to be viewed.

[0103] <Conclusion>

[0104] As described above, according to the first embodiment, the predetermined interest level reference information corresponding to the object of interest is obtained from the storage unit and used, whereby the user's interest level in the object of interest can be determined with high accuracy. As a result, a service of presenting information on products that the user is interested in with high accuracy can be provided, or the number of people interested in the object can be obtained with high accuracy.

[0105] <<Second Embodiment>>

[0106] A second embodiment of the present invention is described. Note that in the following, descriptions of points common to the first embodiment (e.g., configurations and processes common to the first embodiment) are omitted, and points different from the first embodiment are described. The overall processing of the second embodiment is similar to that of the first embodiment ( Figure 6 ) is the same as the overall processing of the first embodiment and the second embodiment. However, the reference information acquisition processing in step S104 is different between the first embodiment and the second embodiment. In the reference information acquisition processing of the first embodiment, the interest level reference information corresponding to the object of interest is selected from a plurality of interest level reference information prepared in advance. In the reference information acquisition processing of the second embodiment, the interest level reference information corresponding to the object of interest is acquired by analyzing the image of the object of interest. Fig.10 1 is a flowchart of the reference information acquisition process according to the second embodiment. The interest level determination process of the second embodiment is the same as that of the first embodiment ( Fig. 9 ) is processed in the same manner as the interest level determination process.

[0107] when Fig.10 When the reference information acquisition process starts, in step S301, the reference information acquisition unit 305 obtains Figure 6 The region of the object of interest detected in step S103 is extracted from the external image acquired in step S101. As a result, an object of interest image (image of the object of interest) is acquired.

[0108] In steps S302 to S305 , the reference information acquisition unit 305 acquires interest level reference information corresponding to the object of interest by analyzing the object of interest image acquired in step S301 .

[0109] In step S302, the reference information acquisition unit 305 performs semantic region division (segmentation) on the attention object image acquired in step S301. Figure 8 Set areas 1 to 3 in B.

[0110] In step S303 , the reference information acquisition unit 305 determines the type of the setting area (image area / eye-catching area / text area) by detecting a feature portion (eg, a character or an object) from the setting area for each setting area set in step S302 .

[0111] In step S304, the reference information acquisition unit 305 determines the weight of the setting area and the relevant area by evaluating the feature part detected in step S303 for each setting area set in step S302. When the object of interest is an advertising medium, a large weight may be determined in the setting area estimated to meet the advertising purpose, and a small weight may be determined in the setting area estimated not to meet the advertising purpose. For example, a large weight may be determined in an image area where a product to be advertised is drawn, and a small weight may be determined in a striking area where only a slogan with low relevance to the product is described. Then, a medium weight may be determined in a text area describing the description of the product.

[0112] In step S305 , the reference information acquisition unit 305 acquires interest level reference information corresponding to the object of interest by integrating the processing results of steps S302 to S304 .

[0113] Note that the reference information acquisition unit 305 may use an arithmetic unit (learned model) for receiving an input of an object of interest image and outputting an analysis result of the object of interest image (e.g., any processing result of steps S302 to S305). The arithmetic unit may be used for one process of steps S302 to S305, or the arithmetic unit may be used for two or more processes. The CPU 121 may be used as an arithmetic unit, and the controller 120 may include a graphics processing unit (GPU) used as an arithmetic unit.

[0114] The server 130 may analyze the object of interest image and obtain interest level reference information corresponding to the object of interest. In this case, for example, the reference information acquisition unit 305 transmits the external image or the object of interest image to the server 130, and the server 130 returns the interest level reference information corresponding to the object of interest to the reference information acquisition unit 305.

[0115] As described above, according to the second embodiment, interest degree reference information corresponding to the attention object is acquired by analyzing the attention object image, and by using the acquired interest degree reference information, the user's interest degree in the attention object can be determined with high accuracy.

[0116] Note that the above embodiments (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the configurations of the above embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above embodiments are also included in the present invention.

[0117] <<Other embodiments>>

[0118] The present invention can also be implemented by supplying a program for implementing one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium and causing one or more processors in a computer of the system or device to read and execute the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0119] The disclosure of this embodiment includes the following configurations, methods, programs, and media.

[0120] (Configuration 1)

[0121] An information processing device, comprising:

[0122] A first acquisition unit, configured to acquire sight line information of a user;

[0123] a detection unit, configured to detect an object viewed by the user based on the line of sight information;

[0124] a second acquisition unit configured to acquire, from a storage unit, reference information for determining the user's interest level in the object, the reference information being predetermined information corresponding to the object; and

[0125] A determination unit is configured to determine the interest level based on the line of sight information and the reference information.

[0126] (Configuration 2)

[0127] An information processing device, comprising:

[0128] A first acquisition unit, configured to acquire sight line information of a user;

[0129] a detection unit, configured to detect an object viewed by the user based on the line of sight information;

[0130] a second acquisition unit configured to acquire reference information for determining the user's interest in the object by analyzing the image of the object; and

[0131] A determination unit is configured to determine the interest level based on the line of sight information and the reference information.

[0132] (Configuration 3)

[0133] The information processing device according to configuration 2, wherein:

[0134] The second acquisition unit acquires the reference information by using a learned model for receiving an input of the image and outputting an analysis result of the image.

[0135] (Configuration 4)

[0136] An information processing device according to any one of configurations 1 to 3, wherein:

[0137] The first acquisition unit acquires the sight line information of the user's right eye and the sight line information of the user's left eye, and

[0138] The detection unit detects a three-dimensional position of the object based on visual line information of the right eye of the user and visual line information of the left eye of the user.

[0139] (Configuration 5)

[0140] The information processing device according to any one of configurations 1 to 4, further comprising:

[0141] A recording unit is configured to record the interest level in a server.

[0142] (Configuration 6)

[0143] The information processing device according to any one of configurations 1 to 5, wherein:

[0144] The reference information indicates a reference for judging the degree of interest for each of a plurality of regions of the object.

[0145] (Configuration 7)

[0146] A head mounted display device, comprising:

[0147] The information processing device according to any one of configurations 1 to 6; and

[0148] Display unit.

[0149] (Method 1)

[0150] A method for controlling an information processing device, comprising:

[0151] Steps for obtaining user's sight line information;

[0152] A step for detecting an object viewed by the user based on the line of sight information;

[0153] a step of acquiring, from a storage unit, reference information for determining the degree of interest of the user in the object, the reference information being predetermined information corresponding to the object; and

[0154] A step for determining the interest level based on the line of sight information and the reference information.

[0155] (Method 2)

[0156] A method for controlling an information processing device, comprising:

[0157] Steps for obtaining user's sight line information;

[0158] A step for detecting an object viewed by the user based on the line of sight information;

[0159] A step for acquiring reference information for judging the user's interest in the object by analyzing the image of the object; and

[0160] A step for determining the interest level based on the line of sight information and the reference information.

[0161] (program)

[0162] A program causing a computer to function as each unit of the information processing apparatus according to any one of configurations 1 to 6.

[0163] (medium)

[0164] A computer-readable medium storing a program causing a computer to function as each unit of the information processing apparatus according to any one of configurations 1 to 6.

[0165] The present invention is not limited to the above-described embodiments, and various variations and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Therefore, the appended claims are attached to disclose the scope of the present invention.

[0166] This application claims priority based on Japanese Patent Application No. 2022-140049 filed on September 2, 2022, the disclosure of which is hereby incorporated by reference in its entirety.

[0167] [reference numerals list]

[0168] 120 Controller

[0169] 121 CPU

[0170] 301 Line of sight detection unit

[0171] 304 Focus on object detection unit

[0172] 305 Benchmark Information Acquisition Unit

[0173] 306 Interest level judgment unit

Claims

1. An information processing device, comprising: A first acquisition unit, configured to acquire sight line information of a user; a detection unit, configured to detect an object viewed by the user based on the line of sight information; a second acquisition unit configured to acquire, from a storage unit, reference information for determining the user's interest in the object, wherein the reference information is predetermined information corresponding to the object; as well as A determination unit is configured to determine the interest level based on the line of sight information and the reference information.

2. An information processing device, comprising: A first acquisition unit, configured to acquire sight line information of a user; a detection unit, configured to detect an object viewed by the user based on the line of sight information; a second acquisition unit configured to acquire reference information for judging the user's interest in the object by analyzing the image of the object; as well as A determination unit is configured to determine the interest level based on the line of sight information and the reference information.

3. The information processing device according to claim 2, wherein: The second acquisition unit acquires the reference information by using a learned model for receiving an input of the image and outputting an analysis result of the image.

4. The information processing device according to claim 1, wherein: The first acquisition unit acquires the sight line information of the user's right eye and the sight line information of the user's left eye, and The detection unit detects a three-dimensional position of the object based on visual line information of the right eye of the user and visual line information of the left eye of the user.

5. The information processing device according to claim 1, further comprising: A recording unit is configured to record the interest level in a server.

6. The information processing device according to claim 1, wherein: The reference information indicates a reference for judging the degree of interest for each of a plurality of regions of the object.

7. A head mounted display device, comprising: The information processing device according to any one of claims 1 to 6; as well as Display unit.

8. A method for controlling an information processing device, comprising: Steps for obtaining user's sight line information; A step for detecting an object viewed by the user based on the line of sight information; a step for acquiring, from a storage unit, reference information for determining the user's interest level in the object, the reference information being predetermined information corresponding to the object; as well as A step for determining the interest level based on the line of sight information and the reference information.

9. A method for controlling an information processing device, comprising: Steps for obtaining user's sight line information; A step for detecting an object viewed by the user based on the line of sight information; A step for acquiring reference information for judging the user's interest in the object by analyzing the image of the object; as well as A step for determining the interest level based on the line of sight information and the reference information.

10. A program for causing a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 6. 11 . A computer-readable medium storing a program for causing a computer to function as each unit of the information processing apparatus according to claim 1 .

Images