INFORMATION PROCESSING DEVICE

Abstract

An information processing apparatus according to the present invention includes a first acquisition unit configured to acquire line-of-sight information of a user, a detection unit configured to detect an object viewed by the user based on the line-of-sight information, a second acquisition unit configured to acquire, from a storage unit, reference information for determining a degree of interest of the user with respect to the object, the reference information being predetermined information corresponding to the object, and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information.

Description

TECHNICAL FIELD

[0001] The present invention relates to an information processing apparatus and, more particularly, to a technique for determining a person's degree of interest in an object. BACKGROUND OF THE INVENTION

[0002] A technique for determining a person's degree of interest in an object based on the person's line-of-sight information is proposed. Patent Application Publication No. 1 discloses a technique for counting the number of people viewing an advertisement content displayed on an advertisement display device based on a line-of-sight direction of a person located in an area where the advertisement display device is installed. Patent Application Publication No. 2 discloses a method for determining a passenger's degree of interest in an object to be viewed based on the duration of the passenger's viewing of the object (passenger of a moving body). PATENT LITERATURE [PTL 1] JP 2008-112401 A [PTL 2] JP 2007-172378 A SUMMARY OF THE INVENTION TECHNICAL PROBLEM

[0003] In conventional techniques (for example, the techniques disclosed in PTL 1 and 2), the degree of interest or degree of interest of a person in an object is determined by assuming that the person is interested in the object when the person observes the object. However, it cannot be assumed that a person is interested in an object simply because the person observes the object. Therefore, the degree of interest cannot be determined with high accuracy in conventional techniques.

[0004] An object of the present invention is to provide a technique for determining the degree of interest of a person in an object with high accuracy. SOLUTION TO THE PROBLEM

[0005] The present invention provides, in a first aspect, an information processing apparatus comprising: a first acquisition unit configured to acquire line-of-sight information of a user, a detection unit configured to detect an object viewed by the user based on the line-of-sight information, a second acquisition unit configured to acquire, from a storage unit, reference information for determining a degree of interest of the user with respect to the object, the reference information being predetermined information corresponding to the object, and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information.

[0006] The present invention provides, in a second aspect, an information processing apparatus comprising: a first acquisition unit configured to acquire line-of-sight information of a user, a detection unit configured to detect an object viewed by the user based on the line-of-sight information, a second acquisition unit configured to acquire reference information for determining a degree of interest of the user with respect to the object by analyzing an image of the object, and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information.

[0007] The present invention, in a third aspect, provides a control method for an information processing apparatus, comprising: a step of acquiring line-of-sight information of a user, a step of detecting an object viewed by the user based on the line-of-sight information, a step of acquiring reference information from a storage unit for determining a degree of interest of the user with respect to the object, the reference information being predetermined information corresponding to the object, and a step of determining the degree of interest based on the line-of-sight information and the reference information.

[0008] The present invention, in a fourth aspect, provides a control method of an information processing apparatus, comprising: a step of acquiring line-of-sight information of a user, a step of detecting an object viewed by the user based on the line-of-sight information, a step of acquiring reference information for determining a degree of interest of the user with respect to the object by analyzing an image of the object, and a step of determining the degree of interest based on the line-of-sight information and the reference information.

[0009] The present invention, in a fifth aspect, provides a program for causing a computer to function as each unit of the above-described information processing apparatus.

[0010] The present invention, in a sixth aspect, provides a computer-readable medium storing a program for causing a computer to function as each unit of the above-described information processing apparatus. ADVANTAGEOUS EFFECTS OF THE INVENTION

[0011] According to the present invention, the degree of interest of a person for an object can be determined with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1A and Fig. 1B show external views of a display system. Fig. 2 shows a block diagram illustrating an electrical configuration of the display system. Fig. 3 shows a diagram for describing a principle of a line-of-sight detection method. Fig. 4A shows a view illustrating an eye image, and Fig. Figure 4B shows a view illustrating a brightness distribution of the eye image. Fig. 5 shows a flowchart of line-of-sight determination processing. Fig. 6 shows a flowchart of an overall processing. Fig. 7 shows a block diagram illustrating a functional configuration of a display system. Fig. Figure 8A shows a diagram illustrating an object of interest, and Fig. Figure 8B shows a diagram illustrating interest level reference information. Fig. 9 shows a flowchart of interest degree determination processing. Fig. 10 shows a flowchart of reference information acquisition processing according to a second embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS<<Erstes Ausführungsbeispiel> >

[0012] A first embodiment of the present invention is described below.

[0013] It should be noted that the present invention is applicable to various electronic devices capable of acquiring line-of-sight information (information about a line of sight) of a person. For example, the present invention is applicable to both a video see-through display device and an optical see-through display device. The video see-through display device displays an image (virtual space) obtained by recording a real space on a display surface (display surface that does not transmit light from the real space (outside world)) by combining graphics (e.g., virtual objects) as needed. In this case, the user cannot see the real space directly, but can see the real space indirectly or see graphics in combination with the image of the real space by viewing the displayed image.The see-through optical display device displays graphics, for example, on a display surface (a display surface that transmits light from a real space). In this case, the user can view the real space directly through the display surface or view the graphics displayed on the display surface.

[0014] The present invention is applicable to both a head-mounted display device and other display devices. For example, the present invention is also applicable to a wearable display device and a stationary display device. The head-mounted display device is, for example, smart glasses (augmented reality glasses (AR glasses)) or a head-mounted display (HMD). The wearable display device is, for example, a smartphone or a tablet computer. A device held by the hands and attached to the user's head is a type of wearable display device and also a type of head-mounted display device. A smartphone mounted on a head-mounted adapter (e.g., virtual reality glasses (VR glasses)) is a type of head-mounted display device.The present invention is applicable to both a head-mounted display device in which a user views an image with both eyes and a head-mounted display device in which a user views an image with one eye.

[0015] The present invention is also applicable to a method that allows a user to visually recognize only a virtual space without allowing the user to visually recognize a real space. That is, the present invention is applicable to various types of cross reality (XR), such as augmented reality (AR), mixed reality (MR), and virtual reality (VR).

[0016] The present invention is also applicable to electronic devices other than the display device. The information processing device to which the present invention is applied may be provided in a display device or may be provided in an electronic device separate from the display device. For example, the present invention is also applicable to a controller or a personal computer (PC) connected to a display device. The present invention is also applicable to a surveillance camera.

[0017] In the first embodiment, the person's line-of-sight information is acquired, and the person's degree of interest in the object is determined based on the line-of-sight information. In the first embodiment, the object is a real object, but the object may also be a virtual object. In conventional techniques, the person's degree of interest in an object is determined by assuming that the person has an interest in the object when the person observes the object. However, it cannot be assumed that a person has an interest in an object simply because they observe the object. Therefore, the degree of interest cannot be determined with high accuracy in conventional techniques.

[0018] For example, an advertising medium (e.g., posters, signage, and digital signage) may have a large area for an eye-catcher (eye-catching area). The eye-catching area is, for example, an area where a photo of a person (advertising model) or a slogan is arranged in a large format. Here, the eye-catching area may have little relevance to an advertising goal (e.g., a product or service). Therefore, even if the person looks at the eye-catching area, they may not make a connection with the advertising goal and have no interest in the advertising goal and the advertising medium. It should be noted that the area with little relevance to the advertising goal is not limited to the eye-catching area.

[0019] Therefore, in the first embodiment, the person's degree of interest in the object is determined with high accuracy based on a suitable determination reference according to the object. For example, the degree of interest is determined with high accuracy by a determination reference taking into account the content of the object (intention of the object's design). <konfigurationsbeschreibung>

[0020] Fig. 1A and Fig. 1B illustrate an appearance of a display system 100 according to the first embodiment. Fig. 1A shows a front view and Fig. 1B a view from the rear. As in Fig. 1A, the display system 100 includes an HMD 110, a controller 120, and a server 130. In the first embodiment, it is assumed that the HMD 110 is used as a video view-through HMD. The controller 120 is connected to the HMD 110 via a cable and wirelessly connected to the server 130. The controller 120 may be wirelessly connected to the HMD 110. Fig. 2 shows a block diagram illustrating an electrical configuration of the display system 100.

[0021] As in the Fig. 1A, Fig. 1B and Fig. 2, the HMD 110 includes an imaging lens 111, display units 112, light sources 113a and 113b, a light receiving lens 114, eye imaging elements 115, and a button 140. Furthermore, the HMD 110 includes, as shown in Fig. 2, an imaging element 116, a photometry circuit 117, a light source drive unit 118 and a display unit drive circuit 119. As in Fig. 1B, the display unit 112, the light sources 113a and 113b, the light receiving lens 114, and the eye imaging element 115 are provided for the right and left eyes of the user.

[0022] The imaging lens 111 is a lens for imaging the outside world, and the imaging element 116 is an imaging element for imaging the outside world. The imaging element 116 is arranged on a planned imaging plane of the imaging lens 111.

[0023] The display unit 112 displays various images (information). For example, the display unit 112 displays an image of the outside world captured by the imaging element 116 or displays information about an object viewed by the user. The display unit drive circuit 119 is controlled by the controller 120 (a CPU 121 described below) to drive the display unit 112.

[0024] Each of the light sources 113a and 113b is a light source that illuminates the user's eye, and is, for example, an infrared light-emitting diode that emits infrared light insensitive to the user. The light source driving circuit 118 is controlled by the controller 120 (CPU 121) to drive the light sources 113a and 113b. Part of the light emitted by the light sources 113a and 113b and reflected by the user's eyes is condensed by the light-receiving lens 114 onto the eye imaging elements 115. The light-receiving lens 114 is a lens for imaging the user's eyes, and the eye imaging elements 115 are imaging elements for imaging the user's eyes.

[0025] The knob 140 is a knob for adjusting the interval between the right eye display unit 112 and the left eye display unit 112 to adjust the distance between the user's pupils.

[0026] The photometry circuit 117 performs amplification, logarithmic compression, and A / D conversion of a signal obtained from the imaging element 116, which also serves as a photometric sensor, specifically, a brightness signal corresponding to the brightness of the recording target field, and sends the result to the controller 120 (CPU 121) as brightness information of the recording target field.

[0027] The controller 120 includes the CPU 121, a memory unit 122, a line-of-sight detection circuit 123, a low-pass filter (LPF) 124, a display unit drive circuit 125, operating elements 126 to 128, and a wireless communication circuit 129.

[0028] The CPU 121 is a central processing unit of a microcomputer integrated into the control device 120 and controls the entire display system 100.

[0029] The storage unit 122 has a function of storing a video signal from the eye imaging element 115 and a function of storing a line-of-sight correction parameter. The line-of-sight correction parameter is a parameter for correcting an individual difference in the line of sight. Furthermore, the storage unit 122 has a function of storing interest degree reference information. The interest degree reference information is predetermined information corresponding to the object and is reference information for determining the user's interest degree in the object. The storage unit 122 stores a plurality of pieces of interest degree reference information, each corresponding to a plurality of objects.

[0030] The line-of-sight detection circuit 123 performs A / D conversion on the output (eye image obtained by imaging the eye) of the eye imaging element 115 in a state where the optical image of the eye is formed on the eye imaging element 115, and transmits the result to the CPU 121 via the LPF 124. The CPU 121 extracts a feature point required for detecting the line of sight from the eye image according to a predetermined algorithm described below, and detects the user's line of sight from the position of the feature point.

[0031] The operation elements 126 to 128 receive an operation from the user and output an operation signal (a signal corresponding to an operation performed by the user) to the CPU 121. For example, the operation element 126 is a touch panel capable of receiving a touch operation, the operation element 127 is an operation lever that can be pushed down in any direction, and the operation element 128 is a four-way button that can be pushed in any of the four directions. The operation element 126 (touch panel) has the function of displaying an image (information). As described above, the operation element 126 has a function as a touch panel and a function as a display unit. The display unit drive circuit 125 is controlled by the CPU 121 and drives the operation element 126 (display unit). The user can perform various operations.Perform operations (instructions) using the controls 126 to 128. For example, the user can finely adjust the position of a user interface (UI), e.g., an index, displayed on the display unit 112 using the controls 126 to 128.

[0032] The wireless communication circuit 129 is controlled by the CPU 121 and communicates with an external device. For example, the wireless communication circuit 129 transmits and records the degree of interest (the degree of the user's interest in the object) determined by the CPU 121 to the server 130 via the Internet. <Beschreibung der Sichtlinienerfassungsverarbeitung>

[0033] The line-of-sight detection processing (line-of-sight detection method) is described with reference to the Fig. 3, Fig. 4A, Fig. 4B and Fig. 5. Both the right eye line of sight and the left eye line of sight are detected by the following line of sight detection method. Fig. Figure 3 is a diagram illustrating the principle of a line-of-sight detection method and shows a schematic diagram of an optical system for detecting a line of sight. As shown in Fig. As illustrated in Figure 3, the light sources 113a and 113b are arranged substantially symmetrically with respect to the optical axis of the light-receiving lens 114 and illuminate the user's eyeballs 200. A portion of the light emitted by the light sources 113a and 113b and reflected by the eyeballs 200 is converged by the light-receiving lens 114 onto the eye-imaging elements 115. Fig. 4A shows a schematic diagram of an eye image (an optical image of an eye projected onto the eye imaging element 115) captured by the eye imaging element 115, and Fig. Figure 4B shows a diagram illustrating the output intensity of the eye imaging element 115. Fig. Figure 5 shows a flowchart of line-of-sight detection processing.

[0034] When the line-of-sight detection processing in Fig. 5 begins at step S1, the CPU 121 drives the light sources 113a and 113b using the light source driving circuit 118 to emit infrared light toward the user's eyeball 200. The optical image of the user's eye illuminated by the infrared light is formed on the eye imaging elements 115 by the light-receiving lens 114 and photoelectrically converted by the eye imaging element 115. This obtains an electrical signal of the processable eye image.

[0035] At step S2, the CPU 121 acquires an eye image (image data, image signal) from the eye imaging element 115 via the line-of-sight detection circuit 123.

[0036] At step S3, the CPU 121 acquires coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 113a and 113b and a pupil center c from the eye image obtained at step S2.

[0037] The infrared light emitted from the light sources 113a and 113b illuminates a cornea 201 of the user's eyeball 200. At this time, the corneal reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 201 are condensed by the light-receiving lens 114 and imaged on the eye imaging element 115 to become corneal reflection images Pd' and Pe' in the eye image. Similarly, light fluxes from the end portions a and b of a pupil 202 are also imaged on the eye imaging elements 115 and become pupil end images a' and b' in the eye image.

[0038] Fig. Figure 4B illustrates information about the brightness (brightness distribution) of an area α in the eye image of Fig. 4A. Fig. Figure 4B illustrates the brightness distribution in the X-axis direction, with the horizontal direction of the eye image as the X-axis direction and the vertical direction as the Y-axis direction. In the first embodiment, the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil end images a' and b' in the X-axis direction are Xa and Xb. As shown in Fig. As illustrated in FIG. 4B, brightnesses of an extremely high level are obtained at coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the range from coordinates Xa to coordinates Xb, which corresponds to the area of ​​the pupil 202 (the area of ​​the pupil image obtained by imaging the luminous flux from the pupil 202 onto the eye imaging element 115), brightness of an extremely low level is obtained except for coordinates Xd and Xe. Then, in the area of ​​an iris 203 outside the pupil 202 (the area of ​​the iris image outside the pupil image obtained by imaging the luminous flux from the iris 203), an intermediate brightness between the above two types of brightness is obtained.For example, in an area where the X coordinate (coordinate in the X-axis direction) is larger than the Xa coordinate and in an area where the X coordinate is smaller than the Xb coordinate, an average brightness between the two types of brightness is obtained.

[0039] The X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil end images a' and b' can be determined from the brightness distribution as in Fig. 4B. For example, coordinates with extremely high brightness can be obtained as coordinates of the corneal reflection images Pd' and Pe', and coordinates with extremely low brightness can be obtained as coordinates of the pupil end images a' and b'. When a rotation angle θx of the optical axis of the eyeball 200 with respect to the optical axis of the light-receiving lens 114 is small, the coordinates Xc of a pupil center image c' (the center of the pupil image) obtained by imaging the luminous flux from the pupil center c onto the eye imaging element 115 can be expressed as Xc ≈ (Xa + Xb) / 2. That is, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil end images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0040] At step S4, the CPU 121 calculates an imaging magnification β of the eye image. The imaging magnification β is a magnification determined by the position of the eyeball 200 with respect to the light-receiving lens 114 and can be calculated using a function of an interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0041] At step S5, the CPU 121 calculates the rotation angle of the optical axis of the eyeball 200 with respect to the optical axis of the light-receiving lens 114. The X coordinate of the center point between the corneal reflection image Pd and the corneal reflection image Pe substantially coincides with the X coordinate of a center of curvature O of the cornea 201. Therefore, when the standard distance from the center of curvature O of the cornea 201 to the center c of the pupil 202 is Oc, the rotation angle θx of the eyeball 200 in the ZX plane (the plane perpendicular to the Y axis) can be calculated according to the following formula 1. The rotation angle θy of the eyeball 200 in the ZY plane (the plane perpendicular to the X axis) can also be calculated using a method similar to the method for calculating the rotation angle θx. β×Oc×SINθx≈{(Xd+Xe) / 2}−Xc

[0042] At step S6, the CPU 121 reads the parameters required for detecting the line of sight from the storage unit 122. For example, the CPU 121 reads the parameters m, Ax, Bx, Ay, By, nx, and ny. The parameter m is a constant determined by the configuration of the optical system for performing the line-of-sight detection processing and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the center of the pupil c on the display unit 112. It is assumed that the parameter m is determined in advance and stored in the storage unit 122. The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in the lines of sight and are acquired by performing line-of-sight detection calibration.The parameter Ax is an offset value in the X-axis direction, the parameter Bx is a sensitivity coefficient in the X-axis direction, the parameter Ay is an offset value in the Y-axis direction, and the parameter By is a sensitivity coefficient in the Y-axis direction. It is assumed that the line-of-sight correction parameters Ax, Bx, Ay, and By are stored in the storage unit 122 before the line-of-sight detection processing starts. In the right-eye line-of-sight detection processing, the parameter nx = nRx and the parameter ny = nRy are acquired, and in the left-eye line-of-sight detection processing, the parameter nx = nLx and the parameter ny = nLy are acquired. The parameter nRx is a correction coefficient in the X-axis direction to obtain the right-eye line-of-sight information, and the parameter nRy is a correction coefficient in the Y-axis direction to obtain the right-eye line-of-sight information.The parameter nLx is a correction coefficient in the X-axis direction for obtaining the line of sight information of the left eye, and the parameter nLy is a correction coefficient in the Y-axis direction for obtaining the line of sight information of the left eye. It is assumed that the parameters nRx, nRy, nLx, and nLy are determined in advance and stored in the storage unit 122.

[0043] At step S7, the CPU 121 estimates the user's gaze point on the display unit 112 using the rotation angles θx and θy calculated at step S5 and the parameters m, Ax, Bx, Ay, By, nx, and ny read at step S6. The gaze point can also be regarded as a position where the line of sight is focused, a position where the user is looking, or a line of sight position. The coordinates (Hx, Hy) of the gaze point are assumed to be coordinates corresponding to the pupil center c. The coordinates (Hx, Hy) of the gaze point can be calculated by the following formulas 2 and 3. Hx=m×(Ax×θx+Bx)×nx Hy=m×(Ay×θy+By)×ny

[0044] At step S8, the CPU 121 stores the coordinates (Hx, Hy) of the viewpoint in the storage unit 122 and ends the line-of-sight detection processing.

[0045] Note that the line-of-sight detection method is not limited to the above method, and any method can be used, for example, as long as it is a method of acquiring the line-of-sight information from the eye image. The line of sight can be detected by a method that does not use an eye image, such as a method of detecting an eye potential and detecting a line of sight based on the eye potential without using an eye image. As the final line-of-sight information, information indicating a line-of-sight direction (direction of the line of sight) can be obtained instead of information indicating a viewpoint. For example, processing up to obtaining the rotation angle (Ax × θx + Bx) × nx, (Ay × θy + By) × ny can be performed without obtaining the coordinates (Hx, Hy) of the viewpoint.As final line of sight information, information indicating the gaze points (intersection of the line of sight of the right eye and the line of sight of the left eye, position seen by both eyes) of both eyes can be obtained. <Beschreibung der Gesamtverarbeitung>

[0046] The overall processing of the display system 100 (controller 120) is described with reference to the Fig. 6 and Fig. 7 described. Fig. Figure 6 shows a flowchart of the overall processing. For example, when the user wears the HMD 110 and activates the HMD 110 and the controller 120, the overall processing is Fig. 6 started. Fig. Figure 7 shows a block diagram illustrating a functional configuration of the display system 100. Each functional unit in Fig. 7 is realized by the CPU 121 of the control device 120.

[0047] If the overall processing of Fig. 6 is started, an outside world image acquisition unit 302 acquires an outside world image (outside world image) captured by the imaging element 116 at step S101. Although it is in the Fig. 6 and Fig. 7 is not shown, the CPU 121 performs predetermined image processing on the acquired outside world image and displays the image on the display unit 112.

[0048] At step S102, a line-of-sight detection unit 301 acquires the user's line-of-sight information by performing the line-of-sight detection processing of Fig. 5.

[0049] At step S103, the CPU 121 detects an object viewed by the user (object of interest) based on the line-of-sight information acquired at step S102. For example, an object detecting unit 303 detects an object from the outside world image acquired at step S101. Then, an object-of-interest detecting unit 304 detects the object of interest from the object detected by the object detecting unit 303 based on the line-of-sight information acquired at step S102. For example, the object of interest is detected based on information (e.g., the location of the gaze point, the saccade of the eyeball, the time the gaze point remains, and the number of times the object is viewed) obtained from a plurality of pieces of time-series line-of-sight information.Through the processing of step S103, the object of interest information, which is the detection result of the object of interest, is acquired. Details of the object of interest information are described below.

[0050] At step S104, a reference information acquisition unit 305 acquires interest degree reference information corresponding to the object of interest based on the information on the object of interest acquired at step S103 (reference information acquisition processing). In the first embodiment, the reference information acquisition unit 305 acquires the interest degree reference information corresponding to the object of interest from the storage unit 122. Details of the interest degree reference information will be described below.

[0051] For example, assume that the interest object information indicates the position (a position in real space, a position of the interest object in a real space coordinate system (world coordinate system)) of the interest object. Then, assume that the position of the object corresponding to the interest degree reference information is associated with the interest degree reference information stored in the storage unit 122. In this case, the reference information acquisition unit 305 may acquire the interest degree reference information associated with the position closest to the position indicated by the interest object information from the storage unit 122.

[0052] The object of interest information may indicate the position and type of the object of interest. Then, the position and type of the object corresponding to the interest level reference information may be linked to the interest level reference information stored in the storage unit 122. In this case, the reference information acquisition unit 305 may acquire the interest level reference information linked to the position closest to the position indicated by the object of interest information among the object types indicated by the object of interest information from the storage unit 122.

[0053] The object of interest information may include identification information (e.g., an identifier) ​​for identifying (specifying) the object of interest. Then, the object identification information corresponding to the interest level reference information may be linked to the interest level reference information stored in the storage unit 122. In this case, the reference information acquisition unit 305 may acquire the interest level reference information from the storage unit 122 linked to the identification information corresponding to the identification information included in the object of interest information.

[0054] The plurality of interest degree reference information corresponding to the plurality of objects may be stored in the server 130. In this case, the reference information acquisition unit 305 may acquire the interest degree reference information corresponding to the object of interest from the server 130. The processing of selecting the interest degree reference information corresponding to the object of interest from the plurality of pieces of interest degree reference information based on the plurality of pieces of information about the object of interest may be performed by the reference information acquisition unit 305 or may be performed by the server 130.For example, when the server 130 performs the processing of selecting the interest degree reference information corresponding to the object of interest, the reference information acquisition unit 305 transmits the information on the object of interest to the server 130, and the server 130 returns the interest degree reference information corresponding to the object of interest to the reference information acquisition unit 305.

[0055] The object of interest detection unit 304 can detect the three-dimensional position of the object of interest based on the user's right eye line of sight information and the user's left eye line of sight information, and generate information about the object of interest indicating the three-dimensional position. For example, the object of interest detection unit 304 determines (estimates) a three-dimensional positional relationship between the user (HMD 110) and the object of interest based on the right eye line of sight information, the left eye line of sight information, and the interpupillary distance (the distance between the right eye display unit 112 and the left eye display unit 112).Then, the object of interest detection unit 304 detects (estimates) the three-dimensional position of the object of interest based on the three-dimensional position of the user (HMD 110) and the determined positional relationship. The three-dimensional position of the user (HMD 110) is estimated using, for example, a global positioning system (GPS). By using the three-dimensional position, the degree of interest reference information corresponding to the object of interest can be acquired with higher accuracy than when using the two-dimensional position (it is possible to prevent a malfunction in which the degree of interest reference information of an object other than the object of interest is acquired). Note that the method for estimating the three-dimensional position of the user (HMD 110) is not limited to the method using GPS.For example, the three-dimensional position of the user (HMD 110) can be estimated by simultaneous localization and mapping (SLAM). The three-dimensional position of the user (HMD 110) can be estimated by odometry using an inertial measurement unit (IMU).

[0056] When the interest level reference information is prepared only for a specific object type (e.g., an advertising medium), the processing of step S104 can be performed only when the type of the object of interest is the specific type (e.g., when the medium of interest is the advertising medium). In this way, the processing load can be reduced.

[0057] At step S105, an interest degree determination unit 306 determines the user's interest degree in the object of interest based on the line-of-sight information acquired at step S102, the information about the object of interest acquired at step S103, and the interest degree reference information acquired at step S104. Details of the processing of step S105 (interest degree determination processing) are described below.

[0058] In step S106, an interest level recording unit 307 transmits and records the interest level determined in step S105 to the server 130 via the Internet. The interest level may be recorded in the storage unit 122. <Beschreibung der Interessensgradbestimmungsverarbeitung>

[0059] The interest degree determination processing (processing at step S105 in Fig. 6) is with reference to the Fig. 8A, Fig. 8B and Fig. 9 described.

[0060] Fig. Figure 8A illustrates the object of interest and Fig. Figure 8B illustrates interest level reference information associated with the object of interest in Fig. 8A. Fig. 9 shows a flowchart of interest degree determination processing.

[0061] The object of interest in Fig. 8A is an advertising medium and includes an eye-catching area, a text area, and an image area. In an eye-catching area, located in an upper-center portion of an object of interest, a keyword or slogan with little relevance to a product being advertised is described. A description of a product being advertised and a company name of the product are described in a text area located in a lower-center portion of the object of interest. An image of a product being advertised is drawn in an image area located in a lower-right portion of the object of interest.

[0062] The interest level reference information in Fig. 8B provide a reference for determining the user's degree of interest in the object of interest for each of the plurality of areas of the object of interest. The degree of interest reference information in Fig. 8B indicate the setting areas 1 to 3 (the relative arrangement of each of the setting areas 1 to 3 with respect to the entire objects of interest). Setting area 1 corresponds to the image area, setting area 2 corresponds to the eye-catching area, and setting area 3 corresponds to the text area. In addition, the interest level reference information in Fig. 8B shows the type (image area / eye-catching area / text area), the weight and an associated range for each of the setting areas 1 to 3. In Fig. 8B, setting area 2 is set as the corresponding area of ​​setting area 1, and the corresponding areas are not set in setting areas 2 and 3.

[0063] If the degree of interest determination processing of Fig. 9 is started at step S201, the interest degree determining unit 306 replaces an interest degree I with 0 (initialization of the interest degree I).

[0064] At step S202, the interest degree determining unit 306 sets a sensitivity k of the interest degree I. The larger the sensitivity k, the larger the interest degree I tends to be. For example, the interest degree determining unit 306 sets a value corresponding to the user's attribute as the sensitivity k so that a large value can be easily obtained as the interest degree I of the user who is likely to be interested (to be interested). When the object of interest is an advertising medium and the advertising target is a cosmetic product, the interest degree determining unit 306 can set the sensitivity k so that a large value can be easily obtained as the interest degree of the user of the age (age group) and gender targeted by the cosmetic product.

[0065] At step S203, the interest degree determination unit 306 determines whether the user's gaze remains in the setting area 1 for a threshold time period T1 or longer. This determination can also be regarded as a determination of whether the user's line of sight is directed toward the setting area 1 for a predetermined threshold time period T1 or longer. For example, the object of interest information indicates a range of the object of interest (the area of ​​the object of interest on the display units 112) in the external world image. The interest degree determination unit 306 determines the setting area 1 on the display unit 112 based on the object of interest information and the interest degree reference information of the object of interest (relative arrangement of the setting area 1 with respect to the entire object of interest).Then, based on the line of sight information, the interest degree determination unit 306 determines whether the gaze point remains in the setting area 1 on the display unit 112 for the threshold time period T1 or longer. The interest degree determination unit 306 proceeds to step S211 when it is determined that the user's gaze point remains in the setting area 1 for the threshold time period T1 or longer, and proceeds to step S204 when it is determined that the user's gaze point does not remain in the setting area 1 for the threshold time period T1 or longer.

[0066] At step S211, the interest degree determination unit 306 updates the interest degree I using the interest degree reference information (a weight w1 of the setting range 1) of the object of interest. The interest degree determination unit 306 updates the interest degree I to a larger value as the weight w1 is larger. For example, the interest degree determination unit 306 updates the interest degree I using the following formula 4. I=I+(1+k)×w1

[0067] At step S204, the interest degree determination unit 306 determines whether a saccade occurs in which the gaze point moves within the adjustment area 2. The interest degree determination unit 306 proceeds to step S212 when it is determined that the saccade occurs within the adjustment area 2, and proceeds to step S205 when it is determined that the saccade does not occur within the adjustment area 2.

[0068] At step S212, the interest degree determination unit 306 updates the interest degree I using the interest degree reference information (a weight w2 of the setting range 2) of the object of interest. For example, the interest degree determination unit 306 updates the interest degree I using the following formula 5. I=I+(1+k)×w2

[0069] At step S205, the interest degree determination unit 306 determines whether the user's gaze point remains in the setting range 1 for the threshold time period T1' or longer and the user's gaze point remains in the setting range 2 for a threshold time period T2 or longer. If it is determined that the user's gaze point remains in the setting range 1 for the threshold time period T1' or longer and the user's gaze point remains in the setting range 2 for the threshold time period T2 or longer, the interest degree determination unit 306 proceeds to step S213, and otherwise proceeds to step S206.

[0070] At step S212, the interest degree determination unit 306 updates the interest degree I using the interest degree reference information (weights w1 and w2 of the setting areas 1 and 2) of the object of interest. For example, the interest degree determination unit 306 updates the interest degree I using the following formula 6. I=I+(1+k)×(w1+w2)

[0071] As described above, in the eye-catching area corresponding to the setting area 2, a keyword that has little relevance to a product target to be promoted is described. Therefore, the user is not necessarily interested in the object of interest even when viewing the setting area 2. As in steps S204 and S205, the setting area 2 is combined with other portions of the object of interest, whereby it is possible to determine whether the user is interested in the object of interest. As described above, the setting area 2 is set as the corresponding area of ​​the setting area 1. Therefore, in step S205, the setting area 1 and the setting area 2 are combined.

[0072] At step S206, the interest degree determination unit 306 determines whether the user's gaze point remains in the setting range 3 for a threshold time period T3 or longer. The interest degree determination unit 306 proceeds to step S214 when it is determined that the user's gaze point remains in the setting range 3 for the threshold time period T3 or longer, and ends the interest degree determination processing in Fig. 9, when it is determined that the user's gaze point does not remain in the setting area 3 for the threshold time period T3 or longer.

[0073] At step S214, the interest degree determination unit 306 updates the interest degree I using the interest degree reference information (a weight w3 of the setting range 3) of the object of interest. For example, the interest degree determination unit 306 updates the interest degree I using the following formula 7. I=I+(1+k)×w3

[0074] It should be noted that the threshold time periods T1, T1', T2, and T3 may be predetermined fixed time periods or time periods that are appropriately changed. For example, the threshold time periods T1, T1', T2, and T3 may be time periods that are appropriately changed by the user or time periods that are appropriately changed by the controller 120. The controller may determine the visibility of the object of interest based on the external world image and set longer time periods than the threshold time periods T1, T1', T2, and T3 when the visibility of the object of interest is lower. The threshold time periods T1, T1', T2, and T3 may be the same or different.

[0075] Although the example described is that the three setting areas 1 to 3 are set, the number of setting areas is not particularly limited. The interest level determination processing is not limited to the Fig. 9. The interest level reference information only needs to provide a reference for determining the user's interest level in the object and is not limited to the processing described in Fig. 8B. For example, the interest level reference information may specify the order in which the setting area is viewed or the time period for which the setting area is to be displayed. <schlussfolgerung>

[0076] As described above, according to the first embodiment, the predetermined interest level reference information corresponding to the object of interest is acquired from the storage unit and used, whereby the user's interest level in the object of interest can be determined with high accuracy. This makes it possible to perform a service of presenting the user with information about a product the user is interested in with high accuracy, or to obtain the number of people interested in an object with high accuracy. <<Zweites Ausführungsbeispiel> >

[0077] A second embodiment of the present invention will be described. Note that, below, the description of the same points as those of the first embodiment (e.g., the same configuration and processing as those of the first embodiment) will be omitted, and points different from those of the first embodiment will be described. The overall processing of the second embodiment is the same as the overall processing of the first embodiment ( Fig. 6). However, the reference information acquisition processing at step S104 differs between the first embodiment and the second embodiment. In the reference information acquisition processing of the first embodiment, the interest degree reference information corresponding to the object of interest is selected from a plurality of interest degree reference information prepared in advance. In the reference information acquisition processing of the second embodiment, the interest degree reference information corresponding to the object of interest is acquired by analyzing the image of the object of interest. Fig. Fig. 10 shows a flowchart of the reference information acquisition processing according to the second embodiment. The interest degree determination processing of the second embodiment is the same as the interest degree determination processing of the first embodiment ( Fig. 9).

[0078] If the reference information acquisition processing of Fig. 10 is started at step S301, the reference information acquisition unit 305 extracts the area of ​​the object of interest detected at step S103 from the outside world image acquired at step S101 of Fig. 6 was obtained. As a result, the image of the object of interest (the image of the object of interest) is obtained.

[0079] In steps S302 to S305, the reference information acquisition unit 305 acquires the interest degree reference information corresponding to the object of interest by analyzing the image of the object of interest acquired in step S301.

[0080] In step S302, the reference information acquisition unit 305 performs semantic region division (segmentation) of the image of the object of interest acquired in step S301. The setting regions 1 to 3 in Fig. 8B are set by the processing at step S302.

[0081] At step S303, the reference information acquisition unit 305 determines the type of the setting area (image area / eye-catching area / text area) by detecting a feature part (e.g., a character or an object) from the setting area for each setting area set at step S302.

[0082] At step S304, the reference information acquisition unit 305 determines the weight of the setting area and the associated area by evaluating the feature piece acquired at step S303 for each setting area set at step S302. When the object of interest is an advertisement, a large weight may be determined in a setting area estimated to correspond to the purpose of the advertisement, and a small weight may be determined in a setting area estimated to not correspond to the purpose of the advertisement. For example, a large weight may be determined in the image area where a product to be advertised is drawn, and a small weight may be determined in an eye-catching area where only the slogan with little relevance to the product is described.Then an average weight can be determined in the text area where the description of the product is described.

[0083] At step S305, the reference information acquisition unit 305 acquires the interest degree reference information corresponding to the object of interest by integrating the processing results of steps S302 to S304.

[0084] Note that the reference information acquisition unit 305 may use a computing unit (trained model) that receives an input of the image of the object of interest and outputs an analysis result of the image of the object of interest (e.g., any processing result of steps S302 to S305). The computing unit may be used for one part of the processing of steps S302 to S305, or the computing unit may be used for two or more parts of the processing. The CPU 121 may function as the computing unit, and the controller 120 may include a graphics processing unit (GPU) that functions as the computing unit.

[0085] The server 130 can analyze the image of the object of interest and obtain the interest level reference information corresponding to the object of interest. In this case, for example, the reference information acquisition unit 305 transmits the external world image or the object of interest image to the server 130, and the server 130 returns the interest level reference information corresponding to the object of interest to the reference information acquisition unit 305.

[0086] As described above, according to the second embodiment, the interest degree reference information corresponding to the object of interest is acquired by analyzing the image of the object of interest, and the user's interest degree for the object of interest can be determined with high accuracy using the acquired interest degree reference information.

[0087] It should be noted that the above-described embodiments (including modifications) are merely examples, and configurations obtained by appropriately modifying or changing the configurations of the above-described embodiments within the scope of the present invention are also encompassed by the present invention. Configurations obtained by appropriately combining the configurations of the above-described embodiments are also encompassed by the present invention. <<Weitere Ausführungsbeispiele> >

[0088] The present invention may also be implemented by providing a program for implementing one or more functions of the above-described embodiment to a system or device via a network or storage medium, and by executing the program by one or more processors in a computer of the system or device. Furthermore, the present invention may also be implemented by a circuit (e.g., an ASIC) for implementing one or more functions.

[0089] The invention of the present embodiment includes the following configurations, methods, a program, and a medium. (Configuration 1)

[0090] Information processing device comprising: a first acquisition unit configured to acquire line-of-sight information of a user; a detection unit configured to detect an object viewed by the user based on the line-of-sight information; a second acquisition unit configured to acquire from a storage unit reference information for determining a degree of interest of the user with respect to the object, the reference information being predetermined information corresponding to the object; and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information. (Configuration 2)

[0091] Information processing device comprising: a first acquisition unit configured to acquire line-of-sight information of a user; a detection unit configured to detect an object viewed by the user based on the line-of-sight information; a second acquisition unit configured to acquire reference information for determining a degree of interest of the user with respect to the object by analyzing an image of the object; and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information. (Configuration 3)

[0092] The information processing apparatus according to configuration 2, wherein the second acquisition unit acquires the reference information using a trained model that receives an input of the image and outputs an analysis result of the image. (Configuration 4)

[0093] The information processing apparatus according to any one of configurations 1 to 3, wherein the first acquisition unit acquires line-of-sight information for a right eye of the user and line-of-sight information for a left eye of the user, and the detection unit detects a three-dimensional position of the object based on the line of sight information for the user's right eye and the line of sight information for the user's left eye. (Configuration 5)

[0094] The information processing apparatus according to any one of configurations 1 to 4, further comprising a recording unit configured to record the degree of interest in a server. (Configuration 6)

[0095] Information processing apparatus according to one of configurations 1 to 5, wherein the reference information provides a reference for determining the level of interest for each of a plurality of areas of the object. (Configuration 7)

[0096] Head-mounted display device comprising: the information processing device according to any one of configurations 1 to 6; and a display unit. (Procedure 1)

[0097] A control method for an information processing apparatus, comprising: a step of obtaining line-of-sight information of a user; a step of detecting an object viewed by the user based on the line-of-sight information; a step of obtaining reference information for determining a degree of interest of the user with respect to the object from a storage unit, wherein the reference information is predetermined information corresponding to the object; and a step of determining the degree of interest based on the line of sight information and the reference information. (Procedure 2)

[0098] A control method of an information processing apparatus, comprising: a step of obtaining line-of-sight information of a user; a step of detecting an object viewed by the user based on the line-of-sight information; a step of obtaining reference information for determining a degree of interest of the user with respect to the object by analyzing an image of the object; and a step of determining the degree of interest based on the line of sight information and the reference information. (Program)

[0099] A program that causes a computer to function as any unit of information processing apparatus according to any one of configurations 1 to 6. (Medium)

[0100] A computer-readable medium storing a program that causes a computer to function as any unit of the information processing apparatus according to any one of configurations 1 to 6.

[0101] The present invention is not limited to the above-described embodiment, and various modifications and changes may be made thereto without departing from the spirit and scope of the present invention. Accordingly, the following claims are appended to disclose the scope of the present invention.

[0102] This application claims priority based on Japanese Patent Application No. 2022-140049 filed on September 2, 2022, and all contents disclosed therein are incorporated by reference. [List of reference symbols] 120 Control device 121 CPU 301 Line of Sight Detection Unit 304 Recording unit for an object of interest 305 Reference information retrieval unit 306 Interest level determination unit QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2008-112401 A

[0002] JP 2007-172378 A

[0002] JP 2022-140049

[0102] < / schlussfolgerung> < / konfigurationsbeschreibung>

Claims

[1] Information processing device comprising: a first acquisition unit configured to acquire line-of-sight information of a user; a detection unit configured to detect an object viewed by the user based on the line-of-sight information; a second acquisition unit configured to acquire from a storage unit reference information for determining a degree of interest of the user with respect to the object, the reference information being predetermined information corresponding to the object; and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information. [2] Information processing device comprising: a first acquisition unit configured to acquire line-of-sight information of a user; a detection unit configured to detect an object viewed by the user based on the line-of-sight information; a second acquisition unit configured to acquire reference information for determining a degree of interest of the user in the object by analyzing an image of the object; and a determination unit configured to determine the degree of interest based on the line-of-sight information and the reference information. [3] The information processing apparatus according to claim 2, wherein the second acquisition unit acquires the reference information using a trained model that receives an input of the image and outputs an analysis result of the image. [4] Information processing apparatus according to claim 1, wherein the first acquisition unit acquires line-of-sight information for a right eye of the user and line-of-sight information for a left eye of the user, and the detection unit detects a three-dimensional position of the object based on the line of sight information for the user's right eye and the line of sight information for the user's left eye. [5] The information processing apparatus according to claim 1, further comprising a recording unit configured to record the degree of interest in a server. [6] The information processing apparatus according to claim 1, wherein the reference information indicates a reference for determining the degree of interest for each of a plurality of areas of the object. [7] A head-mounted display device comprising: the information processing apparatus according to any one of claims 1 to 6; and a display unit. [8] A control method for an information processing apparatus, comprising: a step of obtaining line-of-sight information of a user; a step of detecting an object viewed by the user based on the line-of-sight information; a step of obtaining reference information for determining a degree of interest of the user with respect to the object from a storage unit, wherein the reference information is predetermined information corresponding to the object; and a step of determining the degree of interest based on the line of sight information and the reference information. [9] A control method of an information processing apparatus, comprising: a step of obtaining line-of-sight information of a user; a step of detecting an object viewed by the user based on the line-of-sight information; a step of obtaining reference information for determining a degree of interest of the user with respect to the object by analyzing an image of the object; and a step of determining the degree of interest based on the line of sight information and the reference information. [10] A program for causing a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 6. [11] A computer-readable medium storing a program for causing a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 6.

Images