Information processing device, head-mounted display, method for controlling the information processing device, and program

Abstract

The objective is to provide an information processing device, a head-mounted display, a control method for the information processing device, and a program that can efficiently record gaze information related to virtual objects. [Solution] The HMD 100 includes a determination means for determining the user's gaze position on the display unit 102, which displays a composite image of a real-world image acquired by the imaging unit 103 and a virtual object; a first determination means for determining whether the virtual object and the gaze position have a predetermined relationship; and an information recording means for recording gaze information when the virtual object and the gaze position have a predetermined relationship, and not recording gaze information when the virtual object and the gaze position do not have the predetermined relationship.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a head-mounted display, a control method for an information processing apparatus, and a program.

Background Art

[0002] In recent years, the development of XR (Extended Reality) has been remarkable. XR is a technology that realizes the fusion of the real world and the virtual world, and is a general term for VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and SR (Substitute Reality). In XR, a head-mounted display (hereinafter referred to as "HMD") is often used. In this case, a user wearing an HMD (hereinafter referred to as an "HMD wearer") can experience XR by, for example, a real space of an actual outdoor scene or a virtual object being presented by superimposed display on the HMD. At this time, if the HMD can detect the line-of-sight position of the HMD wearer, it becomes possible to perform display control utilizing information regarding the detected line-of-sight position.

[0003] In this regard, fixation information indicating which part of an image or a virtual object displayed on the HMD the HMD wearer was fixating on and how is an important index indicating the degree of interest of the HMD wearer in the image or the virtual object, and has high utilization value. For example, the technique described in Patent Document 1 enables effective utilization of information on fixation time by measuring and recording the fixation time on an image displayed on the HMD.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the information recorded by the technology described in Patent Document 1 is the duration of gaze on the image displayed on the HMD, and Patent Document 1 does not mention the recording of gaze information indicating which parts the HMD wearer was gazing at and how. Furthermore, even if such gaze information were recorded, if all gaze information were recorded at all times while the HMD wearer was using the HMD, the amount of data would become enormous. In particular, in XR, gaze information related to virtual objects is an important element, so gaze information when the HMD wearer is gazing at something other than a virtual object is often unnecessary. In addition, even if all recorded gaze information is to be effectively utilized after recording, the necessary data must be extracted first, which leads to a problem of poor work efficiency.

[0006] This invention has been made in view of the above-mentioned problems. The object of this invention is to provide an information processing device, a head-mounted display, a control method for the information processing device, and a program that can efficiently record gaze information relating to virtual objects. [Means for solving the problem]

[0007] To achieve the above objective, the information processing apparatus of the present invention is characterized by comprising: determination means for determining the user's gaze position on a display unit that displays a composite image of a real-world image acquired by an imaging unit and a virtual object; first determination means for determining whether the virtual object and the gaze position are in a predetermined relationship; and information recording means for recording gaze information when the virtual object and the gaze position are in the predetermined relationship, and not recording gaze information when the virtual object and the gaze position are not in the predetermined relationship. [Effects of the Invention]

[0008] According to the present invention, it is possible to efficiently record gaze information related to virtual objects. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the appearance of the HMD (Head-Mounted Display). [Figure 2] This is a block diagram of HMD. [Figure 3] This block diagram shows the hardware configuration for running each processing program on the HMD. [Figure 4] This is a flowchart showing the process of recording gaze information on an HMD (Head-Mounted Display). [Figure 5] This diagram illustrates the determination method used in the gaze information recording process of an HMD (Head-Mounted Display). [Figure 6] This diagram illustrates the settings for recording gaze information. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings. However, the configurations described in these embodiments are merely illustrative, and the scope of the present invention is not limited to the configurations described in these embodiments. For example, each part constituting the present invention can be replaced with any configuration that can perform a similar function. In addition, any additional components may be added. Furthermore, any two or more configurations (features) from these embodiments can be combined.

[0011] Referring to Figures 1 and 2, an example of the configuration of an HMD 100 to which the present invention is applied will be described. Figure 1 is a diagram showing the external appearance of the HMD 100. As shown in Figure 1, the HMD 100 is a display device that is worn on the user's head. In the following, the user wearing the HMD 100 may be referred to as the "HMD wearer," as in the background technology section above. The HMD 100 has a head-mounting member 101, a display unit 102, and an imaging unit 103. The head-mounting member 101 is for stably fixing the HMD 100 to the HMD wearer's head and is designed to operate so that the HMD 100 does not shift in accordance with the movement of the HMD wearer's head. The head-mounting member 101 is configured to cover the user's head, but it may also be configured to be worn over the user's ears.

[0012] The display unit 102 displays virtual objects or a composite image of a real-world image and a virtual object in front of the user wearing the HMD. The display unit 102 is a liquid crystal display, but may also be, for example, an organic EL (Electro-luminescence) display. The display unit 102 includes a liquid crystal panel, a driver circuit to control the liquid crystal panel, and a memory to hold the image to be displayed. The display unit 102 may be an opaque display unit or an optically transmissive display unit that allows the HMD wearer to directly see the actual surroundings. However, in this embodiment, as will be described later, the display unit 102 is an opaque display unit in order to explain an example in which a composite image of a real-world image and a virtual object is displayed on the display unit 102. The imaging unit 103 is a device (camera unit) that captures the environment around the HMD wearer, that is, the real-world space. The imaging unit 103 captures the environment in front of the HMD wearer as the real-world space. The area in front of the HMD wearer may be directly in front of the HMD wearer's head.

[0013] Figure 2 is a block diagram of the HMD100. As shown in Figure 2, in addition to the imaging unit 103 and display unit 102 described above, the HMD100 includes a gaze detection unit 201, a posture detection unit 202, an operating member 203, an information processing unit 204, and an image processing unit 205. The information processing unit 204 also includes a captured image acquisition unit 206, a virtual object holding unit 207, a virtual object generation unit 208, a gaze information acquisition unit 209, a posture information acquisition unit 210, and a permission / failure setting recording unit 211. The image processing unit 205 also includes a virtual object selection unit 212, a composite image generation unit 213, a gaze information recording unit 214, and an image recording unit 215.

[0014] The imaging unit 103 includes an optical system, an image sensor, a driver circuit to control the image sensor, an A / D conversion circuit to convert the signal acquired by the image sensor into a digital signal, and a development circuit to develop the digital signal converted by the A / D conversion circuit into a captured image. The captured image data output from the imaging unit 103 is acquired by the captured image acquisition unit 206 of the information processing unit 204 and transferred to the composite image generation unit 213 of the image processing unit 205. The composite image generation unit 213 performs a composite processing of the captured image data and the data of a virtual object, which will be described later. This generates a composite image of the captured image in real space and the virtual object. The captured image acquisition unit 206 also transfers the captured image data to the virtual object generation unit 208, causing the virtual object generation unit 208 to generate a virtual object.

[0015] The gaze detection unit 201 is a device for detecting the gaze direction of the user wearing the HMD. The gaze detection unit 201 may, for example, be a device that detects the gaze direction from the relationship between the reflected image of infrared light due to corneal reflection and the pupil, obtained by irradiating the user's eyeball with infrared light using an infrared light-emitting diode. Such devices are also used in conventional single-lens reflex cameras. The gaze information, including the user's gaze direction detected by the gaze detection unit 201, is acquired by the gaze information acquisition unit 209 of the information processing unit 204 and transferred to the gaze information recording unit 214 of the image processing unit 205.

[0016] The gaze information recording unit 214 determines whether to record the gaze information, which will be described later, and records the gaze information. The result of the determination of whether to record the gaze information is transmitted to the image recording unit 215 of the image processing unit 205. The recording by the gaze information recording unit 214 is performed on a non-volatile memory device (not shown). This is also true for the recording by the acceptance / rejection setting recording unit 211 and the recording by the image recording unit 215. The image recording unit 215 records the data of the composite image generated by the composite image generation unit 213 according to the determination result of whether to record the gaze information transmitted from the gaze information recording unit 214.

[0017] The posture detection unit 202 is an element for detecting information for acquiring the posture of the HMD 100, such as an angular velocity sensor, an acceleration sensor, a geomagnetic sensor, etc. The information for acquiring the posture of the HMD 100 is acquired by the posture information acquisition unit 210 of the information processing unit 204 when detected by the posture detection unit 202. The posture information acquisition unit 210 acquires the posture information. The acquisition of the posture information is performed by calculation using a Kalman filter or a complementary filter, which is a known posture estimation method. However, any method that can estimate and calculate the posture of the HMD 100 can be used. In addition, the direction of the Yaw direction of the HMD 100 can also be calculated by the geomagnetic sensor or the calculation of the posture information. Further, the HMD 100 may be configured such that by mounting a GPS (Global Positioning System) sensor as the posture detection unit 202, the absolute position information of the HMD 100 can be included in the posture information. The posture information acquired by the posture information acquisition unit 210 is transmitted to the fixation information recording unit 214.

[0018] The operation member 203 is a member that allows the user to perform device operations and function settings such as a power switch and buttons for controlling device settings. In the operation member 203, the fixation information recording permission setting information indicating whether to record fixation information, which is set as an instruction from the user, is transmitted to the permission setting recording unit 211 of the information processing unit 204. The permission setting recording unit 211 records the fixation information recording permission setting information. Note that in the operation member 203, a determination as to whether to record fixation information may be made by its switch operation. Further, the operation member 203 may be configured such that the user can instruct whether to record fixation information via a lever member such as a joystick and a menu screen displayed on the display unit 102.

[0019] The virtual object holding unit 207 holds data for the virtual space. This data includes data for virtual objects (shape information and position / orientation information) that make up the virtual space, and data for light sources that illuminate the virtual space. Some of the data for the virtual space is selected by the virtual object selection unit 212 of the image processing unit 205 and sent to the composite image generation unit 213. The virtual object holding unit 207 also sends virtual object data to the virtual object generation unit 208 to generate another virtual object. The virtual object generation unit 208 generates another virtual object based on the captured image data transferred from the captured image acquisition unit 206 and the virtual object data sent from the virtual object holding unit 207. This data for the other virtual object is sent to the virtual object selection unit 212 of the image processing unit 205.

[0020] The virtual object selection unit 212 selects virtual objects to be shown to the HMD wearer from the data transmitted from the virtual object holding unit 207 and the virtual object generation unit 208. The data of these selected virtual objects is transmitted to the composite image generation unit 213 as data for the virtual objects to be displayed. Furthermore, the data for the virtual objects to be displayed is also transmitted to the gaze information recording unit 214 via the composite image generation unit 213. The data for the virtual objects to be displayed includes data such as display object data, placement position data in the display field of view, overlap order data with each virtual object, and gaze information recording setting data. Details of the gaze information recording setting data will be described later.

[0021] The composite image generation unit 213 creates a composite image using the data of the captured image transferred from the captured image acquisition unit 206 and the data of the virtual object transmitted from the virtual object selection unit 212. Further, the composite image generation unit 213 transmits the data of the created composite image to the display unit 102. In this way, on the display unit 102, the composite image is displayed, and at that time, the position where the virtual object is displayed also moves according to the movement of the HMD 100. Furthermore, the composite image generation unit 213 transmits the data of the created composite image to the image recording unit 215.

[0022] Next, referring to FIG. 3, the hardware configuration for executing each processing program in the HMD 100 will be described. FIG. 3 is a block diagram showing the hardware configuration for executing each processing program in the HMD 100. As shown in FIG. 3, the HMD 100 includes a bus 300, a CPU 301, a ROM 302, a RAM 303, an input I / F 304, and an output I / F 305. The CPU 301 comprehensively controls each device connected via the bus 300. The CPU 301 reads and executes each processing program and the like stored in the ROM 302, which is a read-only memory.

[0023] The operating system (OS), each processing program, device driver, etc. are stored in the ROM 302, temporarily stored in the RAM 303, and appropriately executed by the CPU 301. The input I / F 304 inputs a signal from an external device (for example, a server, etc.) to the HMD 100 as an input signal in a format that can be processed by the HMD 100. The output I / F 305 outputs a signal from the HMD 100 to the external device (for example, a server, etc.) as an output signal in a format that can be processed by the external device.

[0024] Next, the gaze information recording process of the HMD100 will be described with reference to Figure 4. Figure 4 is a flowchart of the gaze information recording process of the HMD100. The flowchart in Figure 4 (control method of the information processing device) is realized by the CPU 301 (computer) loading the program stored in the ROM 302 into the RAM 303 and executing it. The flowchart in Figure 4 starts when the power switch of the HMD100 is operated and a virtual object is displayed on the display screen of the display unit 102.

[0025] In step S400, the CPU 301 (fifth determination means) determines whether to record gaze information using the gaze information recording unit 214. This determination is made based on the gaze information recording permission / failure setting information recorded in the permission / failure setting recording unit 211. If the gaze information recording unit 214 determines to record gaze information, the process proceeds to step S401. On the other hand, if the gaze information recording unit 214 determines not to record gaze information, the gaze information is not recorded, and the flowchart in Figure 4 ends. The gaze information recording permission / failure setting information recorded in the permission / failure setting recording unit 211 is set by the operating member 203 as an instruction from the user, but is not limited to this. For example, the gaze information recording permission / failure setting information may be automatically switched depending on the usage environment of the HMD 100.

[0026] In step S401, the CPU 301 (determination means) determines the gaze position, which is the position of the HMD wearer's gaze on the display screen of the display unit 102, using the gaze information acquisition unit 209 (determination step). This determination is made based on the gaze information of the HMD wearer output from the gaze detection unit 201. In step S402, the CPU 301 (first determination means) determines whether the gaze position is near a virtual object (a predetermined relationship) (determination step). This determination is made based on the gaze position determined in step S401 and the position of the virtual object displayed on the display screen of the display unit 102. If the CPU 301 determines that the gaze position is near a virtual object, the process proceeds to step S403. If multiple virtual objects are displayed on the display screen of the display unit 102, the CPU 301 acquires information indicating which virtual object the gaze position is near. If the CPU 301 determines that the gaze position is not near a virtual object, the gaze information is not recorded, and the flowchart in Figure 4 ends.

[0027] Here, we will explain how to determine whether the gaze position is near a virtual object. Figure 5 is a diagram illustrating the determination method performed in the gaze information recording process of the HMD. 500 is the display screen of the display unit 102. The display screen 500 displays the first virtual object 501 and the second virtual object 502. 503 is the gaze position. The gaze position 503 moves on the display screen 500 according to the direction of the user's gaze. The coordinate information of the gaze position 503 on the display screen 500 is acquired and displayed on the display screen 500 so that it can be seen by the user, as shown in Figure 5(a), but it may be set not to be displayed.

[0028] The method for determining whether a gaze position is near a virtual object will be explained below with reference to Figure 5(b). 501b is the range used when determining whether a gaze position is near the first virtual object 501 (hereinafter referred to as the "first range 501b"). 502b is the range used when determining whether a gaze position is near the second virtual object 502 (hereinafter referred to as the "second range 502b"). In the case of Figure 5(b), gaze position 503b is within the first range 501b. Therefore, gaze position 503b is determined to be near the first virtual object 501. Although not shown in the figure, if a gaze position is within the second range 502b, that gaze position is determined to be near the second virtual object 502.

[0029] The data indicating the range used to determine whether the gaze position is near a virtual object (hereinafter referred to as the "reference range") is included in the virtual object's data as pre-set information for each virtual object. Furthermore, it is desirable that the reference range be set to include the area from the virtual object to a first predetermined distance outward from its edge, as shown in the first range 501b and second range 502b in Figure 5b. This is to allow the recording of gaze information to continue even if the user's gaze slightly deviates from the virtual object due to fluctuations or other reasons while the user is gazing at the edge of the virtual object. In addition, when multiple virtual objects are displayed on the display screen 500, the reference ranges may overlap. In such cases, the CPU 301 avoids misjudgment by prioritizing the virtual object displayed closer to the user when determining whether the gaze position is near the virtual object.

[0030] Furthermore, while the reference range is pre-set for each virtual object as described above, it is not limited to this. For example, it may be transformed in real time according to conditions such as the size and orientation of the virtual object, or it may be adjustable by the user. Also, while the determination of whether the gaze position is near a virtual object is performed using the reference range, it is not limited to this. For example, if the distance from the center coordinates of the virtual object to the gaze position is shorter than a second predetermined distance (a predetermined relationship), it may be determined that the gaze position is near the virtual object.

[0031] Let's return to the explanation of Figure 4. In step S403, the CPU 301 (acquisition means) acquires gaze information recording setting data (recording setting data) from the virtual object data using the composite image generation unit 213. Here, we will explain the gaze information recording setting data. Figure 6 is a diagram for explaining the gaze information recording setting data. The gaze information recording setting data shows the setting values ​​related to the recording of gaze information. For example, if multiple virtual objects are displayed on the display screen 500, the data is organized in a data table as shown in Figure 6. In this data table, values ​​are pre-set for each virtual object for each item: whether gaze information recording is necessary, gaze information recording area, and gaze information type.

[0032] The "Requirement to Record Attention Information" item indicates whether or not to record attention information. The "Attention Information Recording Area" item indicates which area within the virtual object will record attention information. The "Attention Information Type" item indicates the type of data to be recorded as attention information. The "Attention Information Type" item has two possible values: "Detailed" (the second setting value) and "Simplified".

[0033] Returning to the explanation of Figure 4, in step S404, the CPU 301 (second determination means) determines whether to record gaze information for the virtual object (hereinafter referred to as the "gaze object") for which the gaze position was determined to be near the virtual object in step S402. This determination is made based on the value of the gaze information recording requirement item, which is pre-set in the gaze information recording setting data acquired in step S403. If the CPU 301 determines that gaze information should be recorded, that is, if the value of the gaze information recording requirement item is "required" (first setting value), the process proceeds to step S405. On the other hand, if the CPU 301 determines that gaze information should not be recorded, that is, if the value of the gaze information recording requirement item is "not" and not "required", the gaze information will not be recorded, and the flowchart in Figure 4 will end.

[0034] In step S405, the CPU 301 obtains the value of the gaze information recording area item, which is pre-set in the gaze information recording setting data obtained in step S403, for the gaze object. In step S406, the CPU 301 (third determination means) determines whether the gaze position of the gaze object is within the gaze information recording area. This determination is made based on the gaze position determined in step S401 and the position indicated by the value of the gaze information recording area item obtained in step S405. If the CPU 301 determines that the gaze position of the gaze object is within the gaze information recording area, the process proceeds to step S407. On the other hand, if the CPU 301 determines that the gaze position of the gaze object is not within the gaze information recording area, the gaze information is not recorded, and the flowchart in Figure 4 ends.

[0035] Here, with reference to Figures 5(c)(d) and 6, we will explain how to determine whether the gaze position is within the gaze information recording area. 503c in Figure 5(c) is the gaze position. The gaze information recording setting data for the first virtual object 501 is shown in row 600 of Figure 6. Therefore, the virtual object "car 1" refers to the first virtual object 501. The gaze information recording setting data for the second virtual object 502 is shown in row 601 of Figure 6. Therefore, the virtual object "car 2" refers to the second virtual object 502.

[0036] In this explanation, we assume that when the object being watched is the first virtual object 501, it is determined in step S404 to record the watching information. According to row 600 of Figure 6, the value of the item in the watching information recording area of ​​the first virtual object 501 is "front part," and that area is 501c in Figure 5(c). Therefore, in the case of Figure 5(c), it is determined that the watching position 503c is within the watching information recording area of ​​the first virtual object 501, which is the object being watched.

[0037] The values ​​of the items in the attention information recording area are pre-set for each virtual object, indicating which of the multiple areas obtained by dividing the virtual object is being observed (a specific area). For example, if the virtual object is a "car," the values ​​of the items in the attention information recording area could be "front," "right side," or "headlight." As mentioned above, the values ​​of the items in the attention information recording area are pre-set by the names of parts of the virtual object, but are not limited to this; for example, they may also be pre-set by the position, shape, and size within the virtual object.

[0038] Furthermore, the value of the item in the gaze information recording area can be pre-set to indicate the entire area of ​​the virtual object. In Figure 5(d), 502d is the area that indicates the entire area of ​​the second virtual object 502. Therefore, as shown in row 601 of Figure 6, if the value of the item in the gaze information recording area is pre-set to "entire area", gaze information will be recorded if the gaze position is in area 502d. In this way, with the HMD100, the user can pre-set for each virtual object which area within the virtual object they want to record gaze information for. Note that the value of the item in the gaze information recording area is pre-set as described above, but is not limited to this. For example, it may be changed in real time according to conditions such as the size and orientation of the virtual object, or it may be made adjustable by the user.

[0039] Returning to the explanation of Figure 4, in step S407, the CPU 301 (fourth determination means) determines whether the value of the gaze information type item included in the gaze information recording setting data acquired in step S403 is "detailed" for the gaze object. If the CPU 301 determines that the value of the gaze information type item included in the gaze information recording setting data acquired in step S403 is not "detailed," that is, it determines that it is "simple," the process proceeds to step S409, which will be described later. On the other hand, if the CPU 301 determines that the value of the gaze information type item included in the gaze information recording setting data acquired in step S403 is "detailed," the process proceeds to step S408.

[0040] In step S408, the CPU 301 (information recording means) records detailed gaze information as gaze information using the gaze information recording unit 214 (information recording step). The detailed gaze information includes the following information (1) to (3) at the time of execution of the flowchart in Figure 4. (1) Coordinate information on the display screen 500 for the gaze position determined in step S401. (2) Posture information acquired by the posture information acquisition unit 210 (including azimuth information and absolute position information detected by the posture detection unit 202). (3) Data of the virtual object to be displayed.

[0041] However, the information included in the detailed gaze information is not limited to the information described in (1) to (3) above. For example, it may include one or more of the information described in (1) to (3) above, or it may include other information. The other information refers to information that can identify which part of the virtual object the HMD wearer was looking at, and at what posture and angle, at the time the flowchart in Figure 4 was executed.

[0042] In step S409, the CPU 301 (information recording means) records simplified gaze information as gaze information using the gaze information recording unit 214 (information recording step). Simplified gaze information is information indicating that, at the time of execution of the flowchart in Figure 4, the gaze position on the display screen 500 determined in step S401 was determined to be within the gaze information recording area in step S406. However, the information included in the simplified gaze information is not limited to the above information, and any information that can identify that the HMD wearer was gazing at the gaze information recording area of ​​the virtual object at the time of execution of the flowchart in Figure 4 is sufficient.

[0043] The flowchart in Figure 4 is executed periodically until the HMD 100's operation ends when the power switch is operated again. However, the execution period (a fixed period) of the flowchart in Figure 4 is the update period of the display unit 102. As a result, at least the processing in step S402 is repeated at the update period of the display unit 102, unless the gaze information recording unit 214 determines in step S400 that it is not recording gaze information. Note that the execution period of the flowchart in Figure 4 is not limited to this, and it can be executed at any period depending on the necessary requirements such as the granularity of the gaze information to be recorded.

[0044] Based on the above, the HMD100 can efficiently record gaze information about virtual objects by recording or not recording gaze information depending on the positional and distance relationship between the virtual object and the gaze position. Furthermore, the HMD100 can record gaze information about virtual objects only when the HMD wearer is gazing at the virtual object.

[0045] Furthermore, the HMD100 can record or not record gaze information for each virtual object based on the gaze information recording setting data contained in the virtual object's data, specifically the "Gaze Information Recording Necessity" item. The HMD100 can also record gaze information for each virtual object if the gaze position is within a specific area of ​​the virtual object, based on the "Gaze Information Recording Area" item in the gaze information recording setting data contained in the virtual object's data. Additionally, the HMD100 can change the type of gaze information recorded for each virtual object based on the "Gaze Information Type" item in the gaze information recording setting data contained in the virtual object's data. In this way, the HMD100 can efficiently record gaze information related to virtual objects using the gaze information recording setting data contained in the virtual object's data.

[0046] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. For example, in this embodiment, it was explained that there are two types of values ​​for the gaze information type item included in the gaze information recording setting data: "detailed" and "simplified," but it is not limited to these, and may be further subdivided. In addition, the values ​​for the gaze information type item included in the gaze information recording setting data may include, for example, "gaze position" and "posture information." In this case, the user can pre-define the gaze information to be recorded through individual settings or multiple settings.

[0047] Furthermore, in this embodiment, the recording target controlled according to the positional and distance relationships between the virtual object and the gaze position is gaze information, but is not limited to this; for example, it may also be data of a composite image generated by the composite image generation unit 213. In this case, the CPU 301 (image recording means) may record composite image data only when gaze information is stored by the image recording unit 215 according to the positional and distance relationships between the virtual object and the gaze position.

[0048] Furthermore, if the same virtual object is displayed on multiple people's HMDs, the recording of gaze information may be switched depending on whether multiple people are looking at that same virtual object.

[0049] Furthermore, if the HMD100 is made smaller and lighter, the processing system and other components of the HMD100 may be separated, with the other components located within the HMD100, while the processing system is located in a small external box computer as an external configuration. In such a configuration, the small external box computer corresponds to the information processing unit. Note that the external configuration is not limited to a small external box computer; for example, it could be a portable computer such as a notebook PC, tablet PC, or smartphone, or a stationary computer such as a desktop PC.

[0050] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiment to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a dedicated processor (e.g., an ASIC, FPGA, or other circuit) that implements one or more functions. Moreover, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Note that "processor" refers to a processor in a broad sense and includes general-purpose processors and dedicated processors. Furthermore, the operation of the processor may not only be performed by a single processor, but may also be performed by multiple processors located in physically separate locations working together.

[0051] This embodiment includes the following configurations, methods, and programs. (Configuration 1) A determination means for determining the user's gaze position on a display unit that shows a composite image of a real-world image acquired by the imaging unit and a virtual object, A first determination means for determining whether the virtual object and the gaze position are in a predetermined relationship, An information processing apparatus comprising: an information recording means that records gaze information when the virtual object and the gaze position are in the predetermined relationship, and does not record the gaze information when the virtual object and the gaze position are not in the predetermined relationship. (Configuration 2) The information processing apparatus according to Configuration 1, characterized in that the predetermined relationship is that the gaze position is near the virtual object. (Configuration 3) The information processing device according to Configuration 2, characterized in that the vicinity of the virtual object includes the area of ​​the virtual object and the area extending outward from the edge of the virtual object to a first predetermined distance. (Configuration 4) The information processing device according to Configuration 2, characterized in that the gaze position is near the virtual object, and the distance from the center coordinates of the virtual object to the gaze position is shorter than a second predetermined distance. (Configuration 5) The information processing apparatus according to any one of Configurations 1 to 4, characterized in that the first determination means determines whether the virtual object and the gaze position are in the predetermined relationship using data indicating a reference range included in the data of the virtual object. (Configuration 6) The information processing apparatus according to Configuration 5, characterized in that, when the reference ranges overlap, the first determination means prioritizes using the reference range included in the data of the virtual object displayed closer on the display unit to determine whether the virtual object and the gaze position are in the predetermined relationship. (Configuration 7) Acquisition means for acquiring recording setting data contained in the data of the virtual object that has a predetermined relationship with the gaze position, The system includes a second determination means for determining whether the value of the recording requirement item included in the recording setting data is a first setting value, The information processing apparatus according to any one of configurations 1 to 6, characterized in that the information recording means does not record the gaze information when the value of the recording requirement item is not the first set value, even when the virtual object and the gaze position are in the predetermined relationship. (Configuration 8) A third determination means for determining whether the gaze position is within a specific area of ​​the virtual object indicated by the value of the recording area item included in the recording setting data, The information processing device according to configuration 7, characterized in that the information recording means does not record the gaze information when the value of the item indicating whether or not recording is required is the first set value, but the gaze position is not within the specific area of ​​the virtual object indicated by the value of the item for the recording area. (Configuration 9) A fourth determination means for determining whether the value of an item of information type included in the record setting data is a second setting value, The information processing device according to configuration 8, characterized in that the information recording means records detailed attention information as attention information when the value of the item of the information type is the second set value, and records simplified attention information as attention information when the value of the item of the information type is not the second set value. (Configuration 10) The information processing apparatus according to Configuration 9, characterized in that the detailed gaze information includes at least one of the coordinate information of the gaze position, the orientation information of the apparatus having the imaging unit and the display unit, and the data of the virtual object displayed on the display unit. (Configuration 11) The information processing device according to Configuration 9 or 10, characterized in that the simplified gaze information is information indicating that the gaze position is within a specific area of ​​the virtual object. (Configuration 12) When a plurality of virtual objects are displayed on the display unit, the recording setting data contained in each of the data of the plurality of virtual objects is set in advance for each of the plurality of virtual objects, characterized in that the information processing device according to any one of Configurations 7 to 11. (Configuration 13) The information processing apparatus according to any one of Configurations 1 to 12, characterized in that the determination of the gaze position by the determination means is performed at regular intervals. (Configuration 14) The information processing apparatus according to Configuration 13, characterized in that the constant period is the update period of the display unit. (Configuration 15) A fifth determination means for determining whether or not to record the gaze information according to user settings, The information processing device according to any one of configurations 1 to 14, characterized in that the information recording means does not record the gaze information if the user has set it not to record the gaze information. (Configuration 16) An information processing apparatus according to any one of Configurations 1 to 15, comprising: an image recording means that records data of the composite image when the virtual object and the gaze position are in the predetermined relationship, and does not record data of the composite image when the virtual object and the gaze position are not in the predetermined relationship. (Configuration 17) The information processing apparatus according to any one of Configurations 1 to 16, characterized in that the position in which the virtual object is displayed on the display unit moves in accordance with the movement of the imaging unit and the apparatus having the display unit. (Configuration 18) A head-mounted display comprising an information processing device according to any one of Configurations 1 to 17, an imaging unit, and a display unit. (Method 1) A method for controlling an information processing device, A determination step to determine the user's gaze position on the display unit, which shows a composite image of a real-world image acquired by the imaging unit and a virtual object, A determination step of determining whether the virtual object and the gaze position are in a predetermined relationship, A control method for an information processing device, characterized by comprising: an information recording step of recording gaze information when the virtual object and the gaze position are in the predetermined relationship, and not recording the gaze information when the virtual object and the gaze position are not in the predetermined relationship. (Program 1) A program that causes a computer to execute each of the means of the information processing device described in any one of Configurations 1 to 17. [Explanation of Symbols]

[0052] 100 HMDs (Human-Mounted Displays) 102 Display section 103 Imaging Unit 301 CPU (Decision-making means) (First determination means) (Information recording means)

Claims

1. A determination means for determining the user's gaze position on a display unit that shows a composite image of a real-world image acquired by an imaging unit and a virtual object, A first determination means for determining whether the virtual object and the gaze position are in a predetermined relationship, An information processing apparatus comprising: an information recording means that records gaze information when the virtual object and the gaze position are in the predetermined relationship, and does not record the gaze information when the virtual object and the gaze position are not in the predetermined relationship.

2. The information processing apparatus according to claim 1, characterized in that the predetermined relationship is that the gaze position is near the virtual object.

3. The information processing apparatus according to claim 2, characterized in that the vicinity of the virtual object includes the area of ​​the virtual object and the area extending outward from the edge of the virtual object to a first predetermined distance.

4. The information processing apparatus according to claim 2, characterized in that the gaze position being near the virtual object is such that the distance from the center coordinates of the virtual object to the gaze position is shorter than a second predetermined distance.

5. The information processing apparatus according to claim 1, characterized in that the first determination means determines whether the virtual object and the gaze position are in the predetermined relationship using data indicating a reference range included in the data of the virtual object.

6. The information processing apparatus according to claim 5, characterized in that, when the reference ranges overlap, the first determination means prioritizes using the reference range included in the data of the virtual object displayed closer on the display unit to determine whether the virtual object and the gaze position are in the predetermined relationship.

7. An acquisition means for acquiring recording setting data contained in the data of the virtual object that has a predetermined relationship with the gaze position, The system includes a second determination means for determining whether the value of the recording requirement item included in the recording setting data is a first setting value, The information processing apparatus according to claim 1, characterized in that the information recording means does not record the gaze information when the value of the recording requirement item is not the first set value, even when the virtual object and the gaze position are in the predetermined relationship.

8. The system includes a third determination means for determining whether the gaze position is within a specific area of ​​the virtual object indicated by the value of the recording area item included in the recording setting data, The information processing apparatus according to claim 7, characterized in that the information recording means does not record the gaze information when the value of the item indicating whether or not recording is required is the first set value, but the gaze position is not within the specific area of ​​the virtual object indicated by the value of the item for the recording area.

9. The system includes a fourth determination means for determining whether the value of an item of information type included in the record setting data is a second setting value, The information processing apparatus according to claim 8, characterized in that the information recording means records detailed attention information as attention information when the value of the item of the information type is the second set value, and records simplified attention information as attention information when the value of the item of the information type is not the second set value.

10. The information processing apparatus according to claim 9, characterized in that the detailed gaze information includes at least one of the coordinate information of the gaze position, the orientation information of the apparatus having the imaging unit and the display unit, and the data of the virtual object displayed on the display unit.

11. The information processing apparatus according to claim 9, characterized in that the simplified gaze information is information indicating that the gaze position is located within a specific area of ​​the virtual object.

12. The information processing apparatus according to claim 7, characterized in that when a plurality of virtual objects are displayed on the display unit, the recording setting data contained in the data of each of the plurality of virtual objects is set in advance for each of the plurality of virtual objects.

13. The information processing apparatus according to claim 1, characterized in that the determination of the gaze position by the determination means is performed at regular intervals.

14. The information processing apparatus according to claim 13, characterized in that the aforementioned constant period is the update period of the display unit.

15. The system includes a fifth determination means that determines whether or not to record the aforementioned gaze information according to user settings, The information processing apparatus according to claim 1, characterized in that the information recording means does not record the gaze information if the user has set it not to record the gaze information.

16. The information processing apparatus according to claim 1, further comprising: an image recording means that records data of the composite image when the virtual object and the gaze position are in the predetermined relationship, and does not record data of the composite image when the virtual object and the gaze position are not in the predetermined relationship.

17. The information processing apparatus according to claim 1, characterized in that the position in which the virtual object is displayed on the display unit moves in accordance with the movement of the imaging unit and the apparatus having the display unit.

18. A head-mounted display comprising the information processing device described in claim 1, the imaging unit, and the display unit.

19. A method for controlling an information processing device, A determination step to determine the user's gaze position on the display unit, which shows a composite image of a real-world image acquired by the imaging unit and a virtual object, A determination step of determining whether the virtual object and the gaze position are in a predetermined relationship, A control method for an information processing device, characterized by comprising: an information recording step of recording gaze information when the virtual object and the gaze position are in the predetermined relationship, and not recording the gaze information when the virtual object and the gaze position are not in the predetermined relationship.

20. A program for causing a computer to execute each of the means of the information processing apparatus described in claim 1.

Images