Electronic apparatus, information processor, and control method for electronic apparatus

Abstract

To provide an electronic apparatus which collects and presents information more closely matching user interest.SOLUTION: An electronic apparatus has: estimation means for estimating a visible range of a user; specification means for specifying an object viewed by the user, on the basis of the visible range; and collection means for collecting information relating to a visible object which is an object specified by the specification means.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an electronic device, an information processing device, and a method for controlling an electronic device. [Background technology]

[0002] A user's gaze is used as a means of estimating a user's interest without the user's input of a search word or a condition filter. For example, by detecting where the user is looking on a screen such as a head mounted display (HMD) or a personal computer (PC), it is possible to estimate an object in which the user is interested. Patent Document 1 discloses a method of calculating the degree of a user's preference for a product based on the user's gaze movement history for the product and the product purchase history, and of suggesting a recipe using the product. Patent Document 2 discloses a method of quantifying the interest of a survey subject in the content displayed at a viewing location during a viewing time based on the viewing time during which the survey subject viewed advertising content. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-174446 A [Patent Document 2] JP 2008-243058 A Summary of the Invention [Problem to be solved by the invention]

[0004] When presenting related products according to products that a user is interested in, it is difficult to estimate related products that the user is likely to be interested in unless it is known what aspects of the products the user is interested in. In other words, it is difficult to appropriately collect and present information related to the subject (product) that the user is interested in.

[0005] Therefore, an object of the present invention is to provide an electronic device that collects and presents information that more closely matches the user's interests. [Means for solving the problem]

[0006] The electronic device of the present invention is characterized by having an estimation means for estimating a user's visible range, an identification means for identifying an object that the user is viewing based on the visible range, and a collection means for collecting information related to the viewed object, which is the object identified by the identification means. Effect of the Invention

[0007] According to the present invention, it is possible to collect and present information that is more in line with the user's interests. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of an HMD. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of an HMD. [Diagram 3] FIG. 11 is a diagram illustrating identification of a visually recognized object. [Figure 4] 11 is a flowchart illustrating a process of identifying a viewed object. [Diagram 5] 10 is a diagram illustrating the specification of visual attribute information in the first embodiment. FIG. [Figure 6] 11 is a flowchart illustrating a process of identifying visual attribute information. [Figure 7] FIG. 11 is a block diagram showing an example of the configuration of an information processing system according to a second embodiment. [Figure 8] FIG. 11 is a diagram illustrating the specification of visual attribute information in the second embodiment. [Figure 9] 13 is a flowchart illustrating a collected information presentation process according to the second embodiment. [Figure 10] 11 is a flowchart illustrating a process of transmitting visual attribute information to a server. [Figure 11]FIG. 11 is a diagram showing an example of information collected based on visual attribute information. [Figure 12] 11 is a flowchart illustrating an information collection process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <Embodiment 1> 1(A) and 1(B) are external views of an HMD 100 as an example of an electronic device to which the present invention can be applied. Fig. 1(A) is a front perspective view of the HMD 100. Fig. 1(B) is a view showing the rear side of the HMD 100 (the side in contact with the user's eyes).

[0011] The HMD 100 includes an eyeball information acquisition unit 101, a display unit 102, a display lens 103, and a power button 104. The eyeball information acquisition unit 101 includes a left eyeball information acquisition unit 101L for the left eye and a right eyeball information acquisition unit 101R for the right eye. The eyeball information acquisition unit 101 can acquire eyeball information of a user looking through the viewfinder.

[0012] The display unit 102 includes a left eye display unit 102L for the left eye and a right eye display unit 102R for the right eye. The display unit 102 displays images, various information, and a graphical user interface (GUI) of a setting screen.

[0013] The display lens 103 includes a left-eye display lens 103L for the left eye and a right-eye display lens 103R for the right eye. The display lens 103 includes a zoom lens and a focus lens protected by a barrier member, and is designed to focus on the display unit 102. The power button 104 is a user interface for turning the power on and off.

[0014] 2 is a block diagram showing an example of the configuration of the HMD 100. The system control unit 201 is a central processing unit of a microcomputer built into the HMD 100, and controls the entire HMD 100. The system control unit 201 executes a program recorded in a non-volatile memory 205 to realize the processing of each functional unit.

[0015] The system control unit 201 performs display control by controlling the display unit 102, the memory control unit 208, the memory 209, etc. The system control unit 201 can read information to be displayed on the display unit 102 from the memory 209 or the recording medium 211.

[0016] The recording medium 211 stores information about objects (hereinafter also referred to as object information). The object information may be acquired from an external device or the cloud on the network 220 via the network interface 213. The system control unit 201 can load the object information stored in the recording medium 211 or the object information acquired via the network interface 213 into the memory 209 and refer to it.

[0017] The object information includes information on the type of object and attribute information of the object. The attribute information is not limited to attribute information expressed by a numerical value, and may be attribute information indicating a classification. An object has one or more attributes. The attributes of an object may be predetermined by, for example, a user. The attributes of an object may be determined for each type of object.

[0018] The object information further includes information on the display position (coordinates) and display size of the object on the display image. When attribute information is displayed together with the object, the object information includes information on the display position (coordinates) and display size of the attribute information on the display image.

[0019] The non-volatile memory 205 is an electrically erasable and recordable memory, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) (registered trademark). The non-volatile memory 205 stores constants and programs for the operation of the system control unit 201. The programs stored in the non-volatile memory 205 include programs for executing the processes shown in Figs. 4, 6, 9, 10, and 12. The system memory 206 is, for example, a RAM (Random Access Memory). The system memory 206 expands constants and variables for the operation of the system control unit 201, and programs read from the non-volatile memory 205, and is used as a work memory for the system control unit 201. The system timer 207 measures the time used for various controls and the time of a built-in clock.

[0020] The display unit 102 is a display device (display) such as a liquid crystal display (LCD), and performs display according to a digital signal from the memory 209. Although the display unit 102 is shown as one functional unit in FIG. 2, it includes a left eye display unit 102L for the left eye and a right eye display unit 102R for the right eye, which correspond to the left and right eyes, respectively. Output data from the system control unit 201 is written to the memory 209 via a memory control unit 208, or is written directly to the memory 209. The memory 209 stores image data to be displayed on the display unit 102. The memory 209 has a storage capacity sufficient to store moving images and audio for a predetermined period of time.

[0021] The power supply control unit 203 includes a battery detection circuit, a DC-DC converter, and a switch circuit for switching between blocks to be energized. The power supply control unit 203 can detect whether a battery is installed, the type of battery, and the remaining battery power. Based on these detection results, the power supply control unit 203 controls the DC-DC converter under the instruction of the system control unit 201, and supplies the voltage used to each unit including the recording medium 211.

[0022] The power supply unit 204 includes primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries, and an AC adapter, etc. The storage interface 210 is a communication interface with a recording medium 211. The recording medium 211 is a recording medium such as a memory card or a hard disk, and includes a semiconductor memory, a magnetic disk, etc.

[0023] The operation unit 202 is an operation member for inputting various instructions to the HMD 100, and notifies the system control unit 201 of information related to operations from the user. The operation unit 202 includes, for example, the power button 104. When the system control unit 201 receives a notification that the power button 104 has been pressed, it instructs the power control unit 203 to transition the HMD 100 to a power-off state if the HMD 100 is in a power-on state, and to transition the HMD 100 to a power-on state if the HMD 100 is in a power-off state.

[0024] The network interface 213 is an interface for the HMD 100 to communicate with a network 220 such as a local area network (LAN) or the Internet based on instructions from the system control unit 201. The wireless interface 221 is an interface for communicating with external devices by wireless communication methods such as Wi-Fi and Bluetooth (registered trademark).

[0025] The eyeball information acquisition unit 101 includes an event sensor 215, an event data calculation unit 216, and a line of sight It includes a detection lens 217 and an infrared light emitting diode 218. Although the eyeball information acquisition unit 101 is shown as one functional unit in Fig. 2, it includes a left eyeball information acquisition unit 101L for the left eye and a right eyeball information acquisition unit 101R for the right eye, which correspond to the left and right eyes, respectively. The eyeball information acquisition unit 101 can detect the user's eye 219 and acquire information about the user's eye (eyeball information). The infrared light emitted from the infrared light emitting diode 218 is reflected by the user's eye 219, and the reflected infrared light passes through the gaze detection lens 217 and forms an image on the imaging surface of the event sensor 215.

[0026] The event sensor 215 is an event-based vision sensor that detects a change in luminance of light incident on each pixel and outputs information about the pixel where the luminance change occurred asynchronously with other pixels. Data output from the event sensor 215 (hereinafter referred to as event data) includes, for example, the position coordinates of the pixel where the luminance change event occurred, the polarity (positive or negative) of the luminance change, and timing information corresponding to the time when the event occurred. Compared to a synchronous frame-based sensor, the event sensor 215 has reduced redundancy in the output information and has features such as high-speed operation, high dynamic range, and low power consumption.

[0027] The event data calculation unit 216 acquires (detects) information about the user's eye 219 based on the event data that is continuously output asynchronously from the event sensor 215. The event data (information about pixels that have experienced a change in luminance) is data in which information about each pixel is output asynchronously. Therefore, in order to determine the correlation between event data, the event data calculation unit 216 accumulates event data that is output from the event sensor 215 during a predetermined time and performs various calculation processes on the accumulated data. For example, the event data calculation unit 216 accumulates event data that occurs during a predetermined time and processes the accumulated event data as a set of data to acquire eye information. The event data calculation unit 216 can acquire a plurality of pieces of eye information that occur at different speeds by changing the accumulation time for accumulating the event data.

[0028] The eyeball information includes, for example, gaze position information related to gaze position (position where the user is looking), saccade information related to the direction and speed of saccades, and microsaccade information related to the frequency and amplitude of microsaccades (amount of change in gaze position). The eyeball information may include information on eye movements other than saccades and microsaccades, pupil information related to pupil size and changes in pupil diameter, and blink information related to blink speed and number of times. These pieces of information are merely examples, and the eyeball information is not limited to these pieces of information.

[0029] The event data calculation unit 216 may map the event data for the accumulation time as one frame of image data based on the event occurrence coordinates (the position coordinates of the pixel where the luminance change event occurred) and perform image processing. With this configuration, the event data calculation unit 216 can acquire eye information from one frame of image data obtained by mapping the event data for the accumulation time using a frame-based image processing method.

[0030] The visible range acquisition unit 214 estimates (acquires) the visible range of the user based on the eyeball information acquired by the event data calculation unit 216. The visible range acquisition unit 214 can determine the size (width) of the visible range or the degree of attention (overhead degree) from, for example, the frequency and amplitude of microsaccades. The visible range is synonymous with the attention range and focus range. The degree of attention is an index that is higher when the visible range is narrower and lower when the range is wider. The degree of attention is defined as the antonym of the degree of attention.

[0031] The visible range acquisition unit 214 can be configured with a neural network or the like that receives parameters related to microsaccades, blinks, pupil diameter, etc. as input and outputs the visible range and the gaze degree. However, the visible range acquisition unit 214 is not limited to a method using a neural network, and may determine the visible range and the gaze degree by other methods.

[0032] Furthermore, the visible range acquisition unit 214 may acquire the visible range using various information included in the eyeball information, not limited to microsaccades, blinks, and pupil diameter. The eyeball information includes position information, saccade direction, saccade speed, microsaccade occurrence frequency, microsaccade amplitude, pupil size, change in pupil diameter, blink speed, and blink count. The visible range acquisition unit 214 can estimate the visible range based on any information included in the eyeball information, or any combination of two or more pieces of information included in the eyeball information.

[0033] The viewed object identifying unit 212 identifies an object and attribute information viewed by the user based on the object information and the user's viewing range. When the area difference between the user's viewing range and the display range of an object present in the viewing range is smaller than a predetermined threshold, the viewed object identifying unit 212 identifies an object present in the viewing range as an object viewed by the user. In addition, when the user's gaze is directed at an object present in the viewing range for a longer period of time than a predetermined period of time, the viewed object identifying unit 212 may identify an object present in the viewing range as an object viewed by the user. In addition, when the area difference is smaller than a predetermined threshold and the time during which the user's gaze is directed (viewing time) is longer than a predetermined period of time, the viewed object identifying unit 212 may identify an object present in the viewing range as an object viewed by the user.

[0034] When attribute information is displayed together with an object, the viewed object identification unit 212 identifies the attribute information viewed by the user based on the attribute information of the object information and the user's viewing range. When the area difference between the user's viewing range and the display range of the attribute information present in the viewing range is smaller than a predetermined threshold, the viewed object identification unit 212 identifies the attribute information present in the viewing range as the attribute information viewed by the user. In addition, when the user's gaze is directed to the attribute information present in the viewing range for a longer period of time than a predetermined period of time, the viewed object identification unit 212 may identify the attribute information present in the viewing range as the attribute information viewed by the user. In addition, when the area difference is smaller than a predetermined threshold and the time the user's gaze is directed is longer than a predetermined period of time, the viewed object identification unit 212 may identify the attribute information present in the viewing range as the attribute information viewed by the user.

[0035] The conditions for identifying the object or attribute information viewed by the user are not limited to the conditions related to the area difference and the viewing time, and may be different conditions depending on the type of object. Furthermore, the conditions for identifying the object or attribute information viewed by the user may be different for each user.

[0036] The object or attribute information viewed by the user is stored in memory 209 and can be referenced by system control unit 201. System control unit 201 can also transmit the object or attribute information viewed by the user to an external device or the like via network interface 213.

[0037] Note that the first embodiment is not limited to the HMD 100, but can also be applied to electronic devices such as PCs and tablet terminals in which the distance between the user and the display unit 102 is greater than that of the HMD 100.

[0038] Identification of a viewed object will be described with reference to Fig. 3. The HMD 100 identifies a viewed object based on eyeball information, and acquires object information of the identified object. Fig. 3 shows an example of a screen 300 displayed on the display unit 102 of the HMD 100. Objects 301 to 306 are a group of images arranged in a matrix. Objects The display sizes of 301 to 306 vary depending on the object.

[0039] Line of sight movement trajectory 310 indicates how coordinates indicating the user's line of sight position (hereinafter, line of sight coordinates) have moved relative to screen 300. Line of sight movement trajectory 311 of visible range indicated by a plurality of circles (circles surrounded by dotted lines) indicates how the user's visible range has moved according to line of sight movement trajectory 310 relative to screen 300. A visual object visually recognized by the user is identified based on the display size of the object on screen 300, line of sight movement trajectory 310, and line of sight movement trajectory 311 of the visible range.

[0040] The process of identifying a viewed object will be described with reference to Fig. 4. Fig. 4 is a flowchart illustrating the process of identifying a viewed object. In step S401, the system control unit 201 acquires the coordinates of the line of sight and the visible range. The information on the coordinates of the line of sight is included in the eyeball information acquired by the eyeball information acquisition unit 101. The information on the visible range is acquired by the visible range acquisition unit 214.

[0041] In step S402, system control unit 201 acquires object information of an object displayed on display unit 102. The object information acquired in step S402 includes information on the display position and display size of the object.

[0042] In step S403, system control unit 201 determines whether or not there is an object at the line of sight coordinates. For example, when the line of sight coordinates acquired in step S401 and the display position of the object are approximately the same, system control unit 201 can determine that there is an object at the line of sight coordinates. The display position of the object here can be, for example, the coordinates of the center or center of gravity of the display range of the object (the area in which the object is displayed). If there is an object at the line of sight coordinates, the process proceeds to step S404. If there is no object at the line of sight coordinates, the process shown in FIG. 4 ends.

[0043] In step S403, system control unit 201 may determine whether or not an object is present within the user's visible range. System control unit 201 may determine that an object is present at the gaze coordinates when the gaze coordinates are included in the display range of the object or a part of the display range of the object, without being limited to the case where the gaze coordinates approximately coincide with the display position of the object.

[0044] In step S404, the system control unit 201 determines whether the area difference between the display range of the object determined to be present in the user's visible range in step S403 and the user's visible range is smaller than a predetermined threshold. If the area difference is smaller than the predetermined threshold, the process proceeds to step S405. If the area difference is equal to or larger than the predetermined threshold, the process shown in FIG. 4 ends.

[0045] In step S405, the system control unit 201 determines whether the time (viewing time) during which the user's gaze is continuously directed at the object is longer than a predetermined time. For example, when the amount of change in the gaze coordinates within the predetermined time is smaller than a predetermined value, the system control unit 201 can determine that the time during which the user's gaze is directed at the object is longer than the predetermined time. If the time during which the user's gaze is directed at the object is longer than the predetermined time, the process proceeds to step S406. If the time during which the user's gaze is directed at the object is equal to or shorter than the predetermined time, the process shown in FIG. 4 ends.

[0046] Note that the determinations in steps S404 and S405 may be omitted, or the system control unit 201 may proceed to step S406 if the condition in either step S404 or step S405 is satisfied.

[0047] In step S406, the system control unit 201 identifies the object determined in step S403 to be present within the user's visual range as a visual object, and collects information related to the identified visual object.

[0048] 3, object 301 is present in visible range 311a, and it is assumed that the area difference between the display range of object 301 and visible range 311a is smaller than a predetermined threshold. Similarly, object 306 is present in visible range 311b, and it is assumed that the area difference between the display range of object 306 and visible range 311b is smaller than a predetermined threshold. Therefore, system control unit 201 identifies object 301 (a bird) and object 306 (a fish) as visible objects.

[0049] The system control unit 201 collects information related to birds or fish in which the user is interested. The system control unit 201 can collect information on objects related to birds or fish from information on a plurality of objects stored in the recording medium 211, for example. The system control unit 201 may collect information on objects related to birds or fish from an external device via the network 220, not limited to the recording medium 211. The system control unit 201 can collect information related to birds or fish in which the user is interested, and present it to the user.

[0050] 4, the HMD 100 can identify a viewed object based on the display range (size) of an object displayed on the display unit 102, a line of sight movement trajectory 310, and a visible range movement trajectory 311, and collect and present information related to the viewed object. The HMD 100 can accurately identify a viewed object, and can collect and present information related to an object (viewed object) in which the user is interested.

[0051] A case where attribute information is displayed together with an object, and attribute information visually recognized by the user (also referred to as visually recognized attribute information) is identified will be described with reference to Fig. 5. The HMD 100 identifies attribute information visually recognized by the user from among attribute information displayed together with the visually recognized object, as identification of the object visually recognized by the user. The HMD 100 identifies attribute information visually recognized by the user based on eyeball information, and acquires the identified attribute information.

[0052] The attribute information displayed together with the object is displayed at a size that allows the user to recognize the attribute information. By displaying the attribute information at a size that allows the user to recognize the attribute information, the HMD 100 allows the user to identify the visual attribute information while recognizing the contents of the attribute information.

[0053] 5 shows an example of a screen 500 displayed on the display unit 102 of the HMD 100. Attribute information 502a to 502f indicates attribute information associated with an object 501. The display size of the attribute information 502a to 502f differs for each piece of attribute information.

[0054] The attribute information of an object is represented by a category and a parameter (numeric value), etc. In the example of Fig. 5, the object 501 is a camera, and has attributes such as an image, a model, a price, a rating, a manufacturer (manufacturer), and a pixel count. The attribute information of the model and manufacturer is represented as a category. The attribute information of the price, rating, and pixel count is represented by a numerical value.

[0055] A line of sight movement trajectory 510 indicates how the coordinates of the user's line of sight have moved relative to the screen 500. A line of sight movement trajectory 511, which is indicated by a number of circles (circles surrounded by dotted lines), indicates how the user's line of sight has moved in accordance with the line of sight movement trajectory 510 relative to the screen 500. The attribute information visually recognized by the user is identified based on the display size of the attribute information on screen 500, line of sight movement trajectory 510, and visual recognition range movement trajectory 511.

[0056] The process of identifying attribute information visually recognized by a user will be described with reference to Fig. 6. Fig. 6 is a flowchart illustrating the process of identifying visually recognized attribute information. In step S601, the system control unit 201 acquires the coordinates of the line of sight and the visible range. The information on the coordinates of the line of sight is included in the eyeball information acquired by the eyeball information acquisition unit 101. The information on the visible range is acquired by the visible range acquisition unit 214.

[0057] In step S602, system control unit 201 acquires object information of an object displayed on display unit 102 and attribute information corresponding to the object. The object information acquired in step S602 includes information on the display position and display size of the object, and information on the display position and display size of attribute information corresponding to the object.

[0058] In step S603, the system control unit 201 determines whether or not there is attribute information at the gaze coordinates (is it displayed). For example, when the gaze coordinates acquired in step S601 and the display position of the attribute information are approximately the same, the system control unit 201 can determine that there is attribute information at the gaze coordinates. The display position of the attribute information here can be, for example, the coordinates of the center or center of gravity of the display range of the attribute information (the area in which the attribute information is displayed). If there is attribute information at the gaze coordinates, the process proceeds to step S604. If there is no attribute information at the gaze coordinates, the process shown in FIG. 6 ends.

[0059] In step S603, system control unit 201 may determine whether attribute information is present within the user's visible range. System control unit 201 may determine that attribute information is present at the gaze coordinates not only when the gaze coordinates approximately coincide with the display position of the attribute information, but also when the gaze coordinates are included in the display range of the attribute information or a part of the display range of the attribute information, for example.

[0060] In step S604, the system control unit 201 determines whether the area difference between the display range of the attribute information determined to be present in the user's visible range in step S603 and the user's visible range is smaller than a predetermined threshold. If the area difference is smaller than the predetermined threshold, the process proceeds to step S605. If the area difference is equal to or larger than the predetermined threshold, the process shown in FIG. 6 ends.

[0061] In step S605, the system control unit 201 determines whether the time (viewing time) during which the user's gaze is continuously directed at the object is longer than a predetermined time. For example, when the amount of change in the gaze coordinates within the predetermined time is smaller than a predetermined value, the system control unit 201 can determine that the time during which the user's gaze is directed at the attribute information is longer than the predetermined time. If the time during which the user's gaze is directed at the attribute information is longer than the predetermined time, the process proceeds to step S606. If the time during which the user's gaze is directed at the attribute information is equal to or shorter than the predetermined time, the process shown in FIG. 6 ends.

[0062] Note that the determinations in steps S604 and S605 may be omitted, or the system control unit 201 may proceed to step S606 if the condition in either step S604 or step S605 is satisfied.

[0063] In step S606, the system control unit 201 identifies the attribute information determined to be present in the user's visual recognition range in step S603 as visually recognized attribute information, and collects information related to the identified attribute information.

[0064] In the example of Fig. 5, the attribute information 502c is present in the visible range 511a, and it is assumed that the area difference between the display range of the attribute information 502c and the visible range 511a is smaller than a predetermined threshold. Similarly, the attribute information 502d is present in the visible range 511b, and it is assumed that the area difference between the display range of the attribute information 502d and the visible range 511b is smaller than a predetermined threshold. Therefore, the viewed object identification unit 212 identifies the price of the attribute information 502c and the evaluation of the attribute information 502d as the attribute information viewed by the user.

[0065] The system control unit 201 acquires attribute information of the price and the rating in which the user is interested. In the example of FIG. 5, the attribute information of the price in which the user is interested is 100,000 yen, and the attribute information of the rating is 3.5 stars. The system control unit 201 collects information of cameras with a price of about 100,000 yen and information of cameras with a rating of about 3.5 based on the acquired attribute information. The system control unit 201 can collect information of the camera based on the attribute information viewed by the user from the recording medium 211 or an external device. The system control unit 201 can collect and present information related to the viewed camera based on the attribute information in which the user is interested.

[0066] According to the process in Fig. 6, the HMD 100 can identify visual attribute information based on the display range (size) of the attribute information displayed on the display unit 102, the line of sight movement trajectory 510, and the visual range movement trajectory 511, and collect information related to the visual object. The visual object here is an object (object corresponding to the visual attribute information) displayed together with the visual attribute information visually recognized by the user. The HMD 100 can accurately identify the visual attribute information, and can collect and present information related to the target (object corresponding to the visual attribute information) in which the user is interested.

[0067] According to the above-mentioned embodiment 1, the HMD 100 can accurately identify the object or attribute information that the user is viewing among the objects displayed on the display unit 102, and collect and present information that closely matches the user's interests.

[0068] <Embodiment 2> The second embodiment relates to a method for collecting information to be presented to a user by the HMD 100 based on an object and attribute information identified by the viewed object identification unit 212. In the second embodiment, the HMD 100 acquires information to be presented to a user from a server (information processing device). The server acquires attribute information identified as being viewed by the user based on the user's viewing range from among attribute information displayed together with the viewed object, and collects information related to the viewed object based on the acquired attribute information.

[0069] The second embodiment can also be applied to a case where the HMD 100 operates independently by providing the HMD 100 with the server configuration described in Fig. 7. Detailed description of the same processes as those in the first embodiment will be omitted.

[0070] Fig. 7 is a block diagram showing an example of the configuration of an information processing system according to embodiment 2. The information processing system includes a server 701 and an HMD 100. The HMD 100 has the same configuration as in Fig. 2, so a detailed description thereof will be omitted. Fig. 7 illustrates a part of the configuration of the HMD 100 described in Fig. 2.

[0071] The server 701 is a Web server that transmits information to be displayed on the display unit 102 to the HMD 100 via the network 220. The server 701 includes a system control unit 702, a storage 703, an object extraction unit 704, a display content control unit 705, and a network interface. The system control unit 702 controls the entire server 701.

[0072] The storage 703 stores information about objects to be displayed on the display unit 102 of the HMD 100 and presented to the user. The object extraction unit 704 extracts information about objects to be presented to the user from the information stored in the storage 703, and transmits the information to a display content control unit 705. The object extraction unit 704 may extract object information from information acquired from outside via a network interface 706, without being limited to information stored in the storage 703.

[0073] The display content control unit 705 converts the object information received from the object extraction unit 704 into a format that can be displayed on the HMD 100, and transmits the converted information to the HMD 100. The network interface 706 is an interface for communicating with the HMD 100 via a network 220 such as a LAN (Local Area Network) or the Internet. The system control unit 201 of the HMD 100 receives the object information converted by the display content control unit 705 via the network interface 706, the network 220, and the network interface 213 of the HMD 100. The system control unit 201 displays the received object information on the display unit 102.

[0074] A case where a plurality of objects are displayed together with their respective attribute information, and the attribute information visually recognized by the user is identified will be described with reference to Fig. 8. Fig. 8 shows an example of a screen 800 displayed on the display unit 102 of the HMD 100.

[0075] Objects 801 to 804 are displayed together with their respective attribute information. Each object is classified according to the object type, such as a camera or a personal computer, and is displayed together with attribute information corresponding to the type. In the example of Fig. 8, objects 801 and 804 of the camera type are displayed together with attribute information of image, model, price, rating, manufacturer, and pixel count. Objects 802 and 803 of the personal computer type are displayed together with attribute information of image, model, price, rating, manufacturer, and clock count.

[0076] A line of sight movement trajectory 810 indicates how the coordinates of the user's line of sight have moved relative to the screen 800. A line of sight movement trajectory 811, indicated by multiple circles (circles surrounded by dotted lines), indicates how the user's line of sight has moved in accordance with line of sight movement trajectory 810 relative to the screen 800.

[0077] The viewed object specifying unit 212 specifies attribute information viewed by the user (viewed attribute information) based on the display range (size) of the attribute information displayed on the display unit 102, the line of sight movement trajectory 810, and the movement trajectory 811 of the view range. In the example of Fig. 8, the viewed object specifying unit 212 specifies the price of attribute information 801a of object 801 and the manufacturer of attribute information 801b, and the price of attribute information 804a of object 804 and the evaluation of attribute information 804b, as viewed attribute information. In addition, the viewed object specifying unit 212 specifies the evaluation of attribute information 802a of object 802 and the manufacturer of attribute information 802b, and the price of attribute information 803a of object 803 and the evaluation of attribute information 803b, as viewed attribute information.

[0078] The system control unit 201 transmits the identified attribute information (visual attribute information) to the object extraction unit 704 of the server 701 via the network 220. The system control unit 201 may transmit information on the visual object corresponding to the visual attribute information together with the visual attribute information. The server 701 collects information related to the visual object based on, for example, the type of the visual object in addition to the attribute information.

[0079] 9 is a flowchart illustrating a collected information presentation process. The collected information presentation process is a process in which the server 701 collects information related to a visual object based on information received from the HMD 100, and causes the HMD 100 to present the collected information to the user.

[0080] In step S901, the system control unit 702 collects information related to the viewed object. Specifically, the system control unit 702 extracts information on an object related to the viewed object from information on a plurality of objects stored in the storage 703, based on the attribute information and the information on the viewed object received from the HMD 100. A specific example of the extracted information and details of the extraction method will be described later with reference to FIGS. 11 and 12.

[0081] In step S902, the display content control unit 705 converts the object information collected in step S901 into a display format for display on the HMD 100. In step S903, the display content control unit 705 transmits the object information, whose display format has been converted, to the HMD 100 via the network 220. In step S904, the system control unit 201 of the HMD 100 displays the information received from the server 701 on the display unit 102, thereby presenting the collected information related to the visual object to the user.

[0082] Through the above processing, the HMD 100 can display on the display unit 102 information on an object related to an object visually recognized by the user (visualized object corresponding to visual attribute information).

[0083] 10 is a flowchart illustrating a process of transmitting visual attribute information to the server 701. The HMD 100 may also transmit to the server 701 information on a visual object corresponding to the visual attribute information, such as the type of the object.

[0084] In step S1001, the viewed object identifying unit 212 identifies attribute information viewed by the user based on the user's viewing range, etc. The viewed object identifying unit 212 can identify multiple pieces of attribute information. In step S1002, the system control unit 201 acquires the attribute information viewed by the user. In step S1003, the system control unit 201 transmits the attribute information acquired in step S1002 to the server 701 via the network 220.

[0085] Through the above process, the HMD 100 transmits one or more pieces of attribute information visually recognized by the user to the server 701. The server 701 can collect information related to a visually recognized object (a visually recognized object corresponding to the visually recognized attribute information) based on the information received from the HMD 100.

[0086] FIG. 11 shows an example of information related to a viewed object collected based on the viewed attribute information. In the example of FIG. 8, the viewed attribute information for a camera is attribute information 801a (price) and attribute information 801b (maker) of object 801, and attribute information 804a (price) and attribute information 804b (rating) of object 804. Extraction condition 1101 for the price of the camera is 110,000 yen, which is the average of the price of object 801 (100,000 yen) and the price of object 804 (120,000 yen). Extraction condition 1101 for the rating of the camera is 4.1, which is the rating of object 804, since the rating of object 801 is not viewed. Extraction condition 1101 for the manufacturer of the camera is Company A, which is the manufacturer of object 801, since the manufacturer of object 804 is not viewed. Note that when there are multiple viewed attribute information for manufacturers, The extraction condition 1101 for the car can be the manufacturer that has the highest ratio among the attribute information of the manufacturers.

[0087] In the example of FIG. 8, the visually recognized attribute information for a personal computer is attribute information 802a (rating) and attribute information 802b (manufacturer) of object 802, and attribute information 803a (price) and attribute information 803b (rating) of object 803. The extraction condition 1102 for the price of the personal computer is the price of object 803, 150,000 yen, since the price of object 802 is not visually recognized. The extraction condition 1102 for the rating of the personal computer is 4.2, which is the average of the rating of object 802, 4.3, and the rating of object 803, 4.1. The extraction condition 1102 for the manufacturer of the personal computer is Company D, which is the manufacturer of object 802, since the manufacturer of object 803 is not visually recognized. Note that when there are multiple visually recognized attribute information for manufacturers, the extraction condition 1102 for the manufacturer can be the manufacturer with the largest proportion of the manufacturer's attribute information.

[0088] The object extraction unit 704 can collect information on objects related to a viewed object based on the extraction conditions 1101 for a camera and the extraction conditions 1102 for a personal computer. The object extraction unit 704 collects information on objects that satisfy the extraction conditions 1101 for a camera from among objects whose object type is a camera. Also, the object extraction unit 704 collects information on objects that satisfy the extraction conditions 1102 for a personal computer from among objects whose object type is a personal computer. In this way, the object extraction unit 704 can collect information related to a viewed object based on the type of the viewed object.

[0089] The extraction condition for attribute information representing a classification such as model or manufacturer among the viewed attribute information may be the classification with the highest proportion. Furthermore, the extraction condition for attribute information represented by a numerical value such as price, rating, number of pixels, clock count, etc. among the viewed attribute information may be the average value of the attribute information. Furthermore, the extraction condition for attribute information represented by a numerical value is not limited to the average value of the attribute information, and may be the median or mode of the attribute information, etc.

[0090] 11 is a camera object extracted from storage 703 by object extraction unit 704 based on camera extraction conditions 1101. Object 1104 is a personal computer object extracted from storage 703 by object extraction unit 704 based on personal computer extraction conditions 1102.

[0091] Object 1103 and object 1104 are information on objects collected based on attribute information viewed by the user. In this manner, the object extraction unit 704 can collect information related to the viewed object based on the type of the viewed object and the viewed attribute information. The information related to the viewed object collected by the object extraction unit 704 is transmitted to the HMD 100 and presented to the user.

[0092] 12 is a flowchart illustrating an example of information collection processing according to embodiment 2. The information collection processing in FIG. 12 illustrates a specific process of step S901 in FIG.

[0093] In step S1201, the system control unit 702 reads object information into a memory (not shown) or the like from the storage 703. In steps S1202 to S1209, the system control unit 702 collects information related to the visual object from the read information.

[0094] The system control unit 702 executes the processes of steps S1203 to S1208 for each attribute of the visual attribute information received from the HMD 100. The visual attribute information obtained is attribute information 801a, 801b, 802a, 802b, 803a, 803b, 804a, and 804b. The object type is a camera or a personal computer. The visual attribute information for cameras is attribute information 801a, 801b, 804a, and 804b, and includes attribute information of price, manufacturer, and manufacturer. The visual attribute information for personal computers is attribute information 802a, 802b, 803a, and 803b, and includes attribute information of rating, manufacturer, and price.

[0095] In step S1202, the object extraction unit 704 selects one of the attributes from the visual attribute information received from the HMD 100. At this time, the object extraction unit 704 also selects attribute information of the same type (camera or personal computer) and the same attribute. In the example of Fig. 8, when the attribute information 801a of the price of the object 801 (camera) is selected, the attribute information 804a of the price of the object 804 (camera) is also selected.

[0096] In step S1203, the object extraction unit 704 determines whether the attribute information selected in step S1202 is expressed by a category. If the attribute information is expressed by a category, the process proceeds to step S1206. If the attribute information is expressed by a numerical value instead of a category, the process proceeds to S1204.

[0097] In step S1204, the object extraction unit 704 acquires the average value of the attribute information selected in step S1202 as an extraction condition. In the example of Fig. 8, the object extraction unit 704 acquires 110,000 yen, which is the average value of 100,000 yen in the attribute information 801a and 120,000 yen in the attribute information 804a, as an extraction condition.

[0098] In step S1205, the object extraction unit 704 extracts objects having values ​​within a predetermined range including the average value acquired in step S1204 from the object information read from the storage 703 in step S1201. For example, as shown in Fig. 11, when the extraction condition 1101 (average price) for the price of a camera is 110,000 yen, the object extraction unit 704 extracts cameras with a price range of, for example, 100,000 yen to 120,000 yen as the predetermined range.

[0099] In step S1206, the object extraction unit 704 acquires, as an extraction condition, the classification with the highest ratio among the attribute information selected in step S1202. For example, when the attribute information 802b of the manufacturer of the object 802 (personal computer) is selected in step S1202, the object extraction unit 704 acquires the manufacturer D company as an extraction condition.

[0100] In step S1207, object extraction unit 704 extracts objects having the classification obtained in step S1206 from the object information read from storage 703 in step S1201. For example, as shown in Fig. 11, when extraction condition 1102 for the manufacturer of personal computers is Company D, object extraction unit 704 extracts personal computers whose manufacturer is Company D.

[0101] In step S1208, the object extraction unit 704 holds the information of the object extracted in step S1205 or step S1207 as collected information. In step S1209, the object extraction unit 704 determines whether or not there is other attribute information that has not been selected in step S1202 among the visual attribute information received from the HMD 100. If there is other attribute information, the process returns to S1202. If there is no other attribute information, the process proceeds to step S1210.

[0102] In step S1210, the object extraction unit 704 transmits the collected information held in step S1208 to the display content control unit 705. The process proceeds to step S902.

[0103] According to the above-mentioned second embodiment, the HMD 100 can accurately identify attribute information of an object displayed on the display unit 102 that the user is viewing, and collect related information according to which attribute of the object the user is interested in. Therefore, the HMD 100 can present information that is more in line with the user's interests.

[0104] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The configurations of the respective embodiments can be appropriately combined within the scope of the gist of the present invention.

[0105] In the present disclosure, for example, a description such as "at least one of XX, YY, and ZZ" means any of XX, YY, ZZ, a combination of XX and YY, a combination of XX and ZZ, a combination of YY and ZZ, or a combination of XX, YY, and ZZ.

[0106] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0107] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) An estimation means for estimating a user's visual range; An identification means for identifying an object being viewed by the user based on the viewable range; a collection means for collecting information related to a visual object, which is an object identified by the identification means; 1. An electronic device comprising: (Configuration 2) The collecting means collects information related to the viewed object based on the type of the viewed object. 2. The electronic device according to configuration 1. (Configuration 3) The identification means identifies attribute information that is being viewed by the user among attribute information displayed together with the viewed object, as the identification of the viewed object; The collecting means collects information related to the viewed object based on the attribute information identified by the identifying means. 3. The electronic device according to configuration 1 or 2. (Configuration 4) the viewed object is displayed together with attribute information corresponding to a type of the viewed object; The collecting means collects information related to the viewed object based on the type of the viewed object and the attribute information identified by the identifying means. 4. The electronic device according to configuration 3. (Configuration 5) The collecting means collects information related to the most prevalent classification as information related to the viewed object based on attribute information representing a classification among the plurality of attribute information identified by the identifying means. 5. The electronic device according to configuration 3 or 4. (Configuration 6) The collecting means collects information related to an average value, a median value, or a mode value of the attribute information expressed by numerical values ​​from among the plurality of attribute information identified by the identifying means, as information related to the visual object. 6. The electronic device according to any one of configurations 3 to 5. (Configuration 7) The attribute information displayed together with the visual object is displayed in a size that allows the user to recognize the attribute information. 7. The electronic device according to any one of configurations 3 to 6. (Configuration 8) The estimation means estimates the visible range based on at least one of gaze position information, a saccade direction, a saccade speed, a frequency of occurrence of a microsaccade, an amplitude of a microsaccade, a pupil size, a change in pupil diameter, a blink speed, and a blink count. 8. The electronic device according to any one of configurations 1 to 7. (Configuration 9) The identification means identifies an object present in the visible range as an object being viewed by the user in at least one of the following cases: (1) when an area difference between a display range of an object present in the visible range and the visible range is smaller than a predetermined threshold, and (2) when a time during which the user's line of sight is directed toward an object present in the visible range is longer than a predetermined time. 9. The electronic device according to any one of configurations 1, 2, and 8. (Configuration 10) The identification means identifies the attribute information present in the visible range as the attribute information being viewed by the user in at least one of the following cases: (1) when an area difference between the display range of the attribute information present in the visible range and the visible range is smaller than a predetermined threshold, and (2) when a time during which the user's line of sight is directed to the attribute information present in the visible range is longer than a predetermined time. 9. The electronic device according to any one of configurations 3 to 8. (Configuration 11) The presenting means further comprises a display means for displaying to the user information related to the visual object collected by the collecting means. 11. The electronic device according to any one of configurations 1 to 10. (Configuration 12) An acquisition means for acquiring attribute information that is identified as being viewed by a user based on a viewable range of the user from among attribute information displayed together with the object; a collection means for collecting information related to the object based on the attribute information acquired by the acquisition means; 13. An information processing device comprising: (method) An estimation step of estimating a user's visual range; A step of identifying an object being viewed by the user based on the view range; a collection step of collecting information related to a visual object, which is an object identified in the identification step; 13. A method for controlling an electronic device comprising: (program) 12. A program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 11. (medium) A computer is caused to function as each of the means of the electronic device according to any one of configurations 1 to 11. A computer-readable storage medium storing a program for performing the above-mentioned operations. [Explanation of symbols]

[0108] 100: HMD, 201: system control unit, 212: visual object identification unit, 214: visual range acquisition unit

Claims

1. An estimation method for estimating the user's viewing range, Based on the aforementioned viewing range, a means for identifying the object being viewed by the user, Collection means for collecting information related to a visible object which is an object identified by the aforementioned identification means, It has, The identification means, as identification of the visible object, identifies the attribute information that the user is viewing among the attribute information displayed together with the visible object. The collection means collects information related to the visible object based on the attribute information identified by the identification means. An electronic device characterized by the following features.

2. The collection means collects information related to the visible object based on the type of the visible object. The electronic device according to feature 1.

3. The aforementioned visible object is displayed along with attribute information corresponding to the type of the visible object. The collection means collects information related to the visible object based on the type of the visible object and the attribute information identified by the identification means. The electronic device according to feature 1.

4. The collection means collects information related to the classification with the highest proportion, based on the attribute information representing the classification among the multiple attribute pieces of information identified by the identification means, as information related to the visible object. The electronic device according to feature 1.

5. The collection means collects information related to the mean, median, or mode of the attribute information expressed numerically, from among the multiple attribute information identified by the identification means, as information related to the visible object. The electronic device according to feature 1.

6. The attribute information displayed along with the aforementioned visible object is displayed in a size that allows the user to recognize the attribute information. The electronic device according to feature 1.

7. The estimation means estimates the viewing range based on at least one of the following: gaze position information, saccade direction, saccade speed, microsaccade frequency, microsaccade amplitude, pupil size, pupil diameter change, blink speed, and blink count. The electronic device according to feature 1.

8. The identifying means identifies an object in the viewing range as an object being viewed by the user in at least one of the following cases: (1) when the difference in area between the display range of an object in the viewing range and the viewing range is smaller than a predetermined threshold, and (2) when the user's gaze is directed towards an object in the viewing range for a longer period of time than a predetermined time. The electronic device according to feature 1.

9. The identifying means identifies the attribute information present in the viewing range as the attribute information being viewed by the user in at least one of the following cases: (1) when the difference in area between the display range of the attribute information present in the viewing range and the viewing range is smaller than a predetermined threshold, and (2) when the user's gaze is directed towards the attribute information present in the viewing range for a longer period of time than a predetermined time. The electronic device according to feature 1.

10. The system further includes a presentation means for presenting information related to the visually accessible object collected by the collection means to the user. The electronic device according to feature 1.

11. The collection means transmits attribute information identified by the identification means and information of the visible object corresponding to the attribute information to an external information processing device, and the external information processing device receives information related to the visible object that has been collected based on the attribute information and the information of the visible object, thereby collecting information related to the visible object. The electronic device according to feature 1.

12. The estimation means estimates the viewing range using a neural network that takes parameters relating to at least one of microsaccades, blinking, and pupil diameter as input and outputs the viewing range. The electronic device according to feature 1.

13. An event sensor that asynchronously outputs event data relating to the user's eyes as information of pixels where brightness has changed, Eyeball information acquisition means that accumulates the event data output from the event sensor over a predetermined period of time to acquire eyeball information relating to the user's eyes. It further possesses, The estimation means estimates the viewing range based on the eyeball information acquired by the eyeball information acquisition means. The electronic device according to feature 1.

14. Among the attribute information displayed with the object, based on the user's viewing range, A means for obtaining attribute information identified as being viewed by the user, A collection means that collects information related to the object based on the attribute information obtained by the acquisition means, An information processing device characterized by having the following features.

15. An estimation step to estimate the user's viewing range, Based on the aforementioned viewing range, the identification step involves identifying the object that the user is viewing, A collection step to collect information related to a visible object, which is an object identified in the aforementioned specific step. It has, In the aforementioned identification step, as part of identifying the visible object, the attribute information that the user is viewing is identified from the attribute information displayed together with the visible object. In the collection step, information related to the visible object is collected based on the identified attribute information. A method for controlling electronic equipment characterized by the following features.

16. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 13.

17. A computer-readable storage medium storing a program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 13.

Images