Head-mounted display device

Abstract

To provide a technique capable of reducing the power consumption of a head-mounted display device, while preventing a user wearing the head-mounted display device from feeling deterioration in image quality and finding discomfort.SOLUTION: A head-mounted display device of the present invention has: imaging means that picks up an image of a real space; line-of-sight detection means that detects the line of sight of a user wearing the head-mounted display device; motion detection means that detects the motion of the user; and control means that controls the operation of the imaging means to control the power consumption of the head-mounted display device on the basis of a result of the detection made by the motion detection means and a result of the detection made by the line-of-sight detection means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for controlling power consumption of a head-mounted display device. [Background technology]

[0002] In recent years, MR (Mixed Reality) technology has become known as a technology that seamlessly blends the real world and the virtual world in real time. MR technology is used, for example, in a video see-through HMD (Head Mounted Display). In a video see-through HMD using MR technology, a real space corresponding to the field of view of a user wearing the HMD is captured by a video camera or the like. Then, a composite image obtained by superimposing a CG (Computer Graphics) image on the captured image of the real space is presented (displayed) to the user.

[0003] Video see-through HMDs using MR technology require an image sensor to capture the real space that serves as the background for the synthesized image, and an image sensor to determine the position of the CG image. An image sensor to detect the user's line of sight may also be installed. Thus, an HMD is equipped with multiple image sensors. Since multiple image sensors are always capturing images while the user is wearing the HMD, the HMD consumes a lot of power.

[0004] Patent document 1 discloses a technology that determines whether the position and / or posture of a head-mounted display device is changing, and stops the operation of multiple image processing circuits mounted on the head-mounted display device in a predetermined order according to the determination result. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-25942 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology disclosed in Patent Document 1, the order in which the operations of the multiple image processing circuits are stopped is determined solely based on whether the position and / or posture of the head-mounted display device is changing, which can cause a user wearing an HMD to feel a deterioration in image quality or a sense of discomfort.

[0007] The present invention aims to provide a technology that can reduce the power consumption of a head-mounted display device while preventing a user wearing the device from feeling a deterioration in image quality or discomfort. [Means for solving the problem]

[0008] A first aspect of the present invention is a head-mounted display device comprising: an imaging means for imaging real space; a gaze detection means for detecting the gaze of a user wearing the head-mounted display device on their head; a motion detection means for detecting the movement of the user; and a control means for controlling the operation of the imaging means so as to control the power consumption of the head-mounted display device based on the detection results of the motion detection means and the detection results of the gaze detection means.

[0009] A second aspect of the present invention is a control method for a head-mounted display device, the control method comprising: an imaging step of imaging a real space; and a visual field detection step of detecting a line of sight of a user wearing the head-mounted display device on their head. This control method is characterized by having a line detection step, a motion detection step that detects the movement of the user, and a control step that controls the operation of the imaging step so as to control the power consumption of the head-mounted display device based on the detection results of the motion detection step and the detection results of the gaze detection step.

[0010] A third aspect of the present invention is a program for causing a computer to function as each of the means of the head-mounted display device described above. [Effects of the Invention]

[0011] According to the present invention, it is possible to reduce the power consumption of a head-mounted display device while preventing a user wearing the head-mounted display device from feeling a deterioration in image quality or a sense of discomfort. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing a configuration of a head-mounted display device. [Figure 2] FIG. 2 is a schematic diagram showing a user's field of view. [Figure 3] 10 is a flowchart showing the operation of the head-mounted display device. [Figure 4] 10 is a flowchart illustrating control of a background image capturing unit according to the first embodiment. [Figure 5] 10 is a table illustrating control of a background image capturing unit according to the first embodiment. [Figure 6] 6 is a flowchart illustrating control of a marker image capturing unit according to the first embodiment. [Figure 7] 10 is a table illustrating control of a marker image capturing unit according to the first embodiment. [Figure 8] 10 is a flowchart illustrating control of a background image capturing unit according to the second embodiment. [Figure 9] 10 is a flowchart showing control of a marker image capturing unit according to the second embodiment. [Figure 10] 11 is a flowchart illustrating control of a background image capturing unit according to the third embodiment. [Figure 11] 11 is a flowchart illustrating control of a marker image capturing unit according to the third embodiment. [Figure 12] 10 is a flowchart showing a synthetic image generation process according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Example 1 A first embodiment of the present invention will be described.

[0014] (Configuration of head-mounted display device) FIG. 1 is a block diagram showing the configuration of a head-mounted display device 100 according to a first embodiment. The head-mounted display device 100 is a video see-through type HMD (Head Mounted Display) using MR (Mixed Reality) technology. The head-mounted display device 100 presents (displays) to a user a composite image in which an image of a virtual object is superimposed on an image of real space corresponding to the field of view of the user wearing the head-mounted display device 100 on his / her head. The image of the virtual object is, for example, CG (Computer Graphics). The position and orientation of the image of the virtual object change depending on the position and orientation of the head-mounted display device 100.

[0015] Background imaging unit 101 is an imaging device that captures an image of real space that serves as the background in the composite image, and includes an imaging element such as a CMOS image sensor or a CCD sensor. The image signal of the real space obtained by background imaging unit 101 is corrected by image processing unit 104, and the corrected image signal is stored in memory 106, which is a temporary storage memory. Note that background imaging unit 101 includes left eye background imaging unit 101a and right eye background imaging unit 101b, which independently capture images. For example, left eye background imaging unit 101a obtains an image of real space corresponding to the field of view of the left eye of a user wearing head-mounted display device 100, and right eye background imaging unit 101b obtains an image of real space corresponding to the field of view of the right eye of the user.

[0016] The marker imaging unit 102 is an imaging device that captures an image of real space and captures an image of a marker placed at a position that serves as a reference for the position of a virtual object in real space. The virtual object may be placed at a position different from the position of the marker, but in Example 1, it is placed at the position of the marker. The marker imaging unit 102 has an imaging element such as a CMOS image sensor or a CCD sensor. An image signal of the real space obtained by the marker imaging unit 102 is corrected by an image processing unit 104, and the corrected image signal is stored in a memory 106. The marker imaging unit 102 has a left eye marker imaging unit 102a and a right eye marker imaging unit 102b, and the left eye marker imaging unit 102a and the right eye marker imaging unit 102b perform imaging independently. For example, the left eye marker imaging unit 102a obtains an image of the real space corresponding to the field of view of the left eye of a user wearing the head-mounted display device 100, and the right eye marker imaging unit 102b obtains an image of the real space corresponding to the field of view of the right eye of the user.

[0017] The eye imaging unit 103 is an imaging device that captures an image of the eye of a user wearing the head-mounted display device 100, and has an imaging element such as a CMOS image sensor or a CCD sensor. The eye image signal obtained by the eye imaging unit 103 is corrected by the image processing unit 104, and the corrected image signal is stored in the memory 106. The eye imaging unit 103 has a left eye imaging unit 103a and a right eye imaging unit 103b, which independently capture images. For example, the left eye imaging unit 103a captures an image of the left eye of a user wearing the head-mounted display device 100, and the right eye imaging unit 103b captures an image of the right eye of the user.

[0018] The image processing unit 104 performs various types of image processing on the image signals obtained by the background imaging unit 101, the marker imaging unit 102, and the eye imaging unit 103. For example, the image processing unit 104 performs image processing such as pixel defect correction, shading correction, and color correction.

[0019] The control unit 105 controls each unit of the head-mounted display device 100. For example, each process of the head-mounted display device 100 is realized by reading and executing a program stored in the memory 106.

[0020] The memory 106 stores various data. For example, the memory 106 stores image signals obtained by the imaging units provided in the head-mounted display device 100, user's gaze position information (described later) obtained by the gaze detection unit 108, and user's movement amount information (described later) obtained by the movement amount detection unit 107. The memory 106 also records data of a composite image (described later) to be displayed on the display unit 111. The memory 106 has a storage capacity sufficient to store these data.

[0021] The movement amount detection unit 107 detects the movement of the user wearing the head-mounted display device 100. For example, the movement amount detection unit 107 has a sensor such as a gyro sensor, an acceleration sensor, or a magnetic sensor, and acquires position and orientation information indicating the position and orientation of the head-mounted display device 100 based on the output signal of the sensor. The position and orientation of the head-mounted display device 100 may be interpreted as the position and orientation of the head of the user wearing the head-mounted display device 100. Furthermore, the movement amount detection unit 107 acquires movement amount information indicating the amount of movement of the head-mounted display device 100 (the user's head) based on the current position and orientation information and the past position and orientation information. For example, the movement amount detection unit 107 acquires movement amount information indicating the amount of movement of the head-mounted display device 100 (the user's head) from a predetermined time ago to the present based on the current position and orientation information and the position and orientation information from a predetermined time ago from the present. The acquired position and orientation information and movement amount information are stored in the memory 106.

[0022] The line of sight detection unit 108 detects the line of sight of the user wearing the head-mounted display device 100 based on the image of the eye obtained by the eye image pickup unit 103. The gaze of the left eye can be detected based on the image of the left eye captured by the right eye imaging unit 103b, and the gaze of the right eye can be detected based on the image of the right eye captured by the right eye imaging unit 103b. The information (information about the gaze) obtained by gaze detection is not particularly limited, but in the first embodiment, gaze position information indicating the position at which the user is looking (gaze position) is acquired. The method of gaze detection is not particularly limited. In the first embodiment, template matching is used to detect the image patterns of the pupil and white of the eye in the eye image. This allows information such as the size and center coordinates of the eye and the center coordinates of the pupil to be obtained in the eye image. Then, the gaze position is estimated based on the amount of deviation between the center coordinates of the eye and the center coordinates of the pupil.

[0023] The virtual object generation unit 109 generates an image of a virtual object. For example, it generates an image of a virtual object to be placed at the position of a marker (a position based on the position of the marker) imaged by the marker image capture unit 102. It may also generate an image of a virtual object (such as a menu) to be placed at a position not based on the real space (a position not based on the position of the marker). For example, the virtual object generation unit 109 detects a marker from the image of real space obtained by the marker image capture unit 102, and generates an image of a virtual object at a position and orientation based on the detection result. It may generate a left-eye virtual object image (an image for the left eye that is an image of a virtual object) based on the image of real space obtained by the left-eye marker image capture unit 102a. It may generate a right-eye virtual object image (an image for the right eye that is an image of a virtual object) based on the image of real space obtained by the right-eye marker image capture unit 102b. The position and orientation information obtained by the movement amount detection unit 107 may be used to generate the image of the virtual object. The image (image data) of the virtual object is stored in the memory 106.

[0024] The image synthesis unit 110 reads out the image of real space obtained by the background imaging unit 101 and the image of the virtual object obtained by the virtual object generation unit 109 from the memory 106. Then, the image synthesis unit 110 generates a synthetic image by overlaying the image of the virtual object on the image of real space, and stores the synthetic image (image data) in the memory 106. For example, the image synthesis unit 110 may generate a left-eye synthetic image by overlaying a left-eye virtual object image on the image of real space obtained by the left-eye background imaging unit 101a, or generate a right-eye synthetic image by overlaying a right-eye virtual object image on the image of real space obtained by the right-eye background imaging unit 101b.

[0025] When generating a composite image, the image synthesis unit 110 may calculate the size of the area of ​​the image of the virtual object within a range visible to the user wearing the head-mounted display device 100, and store the calculated value in the memory 106. The size of the area of ​​the image of the virtual object within a range visible to the user is, for example, the ratio of the area (number of pixels) of the virtual object in the composite image to the entire area (total number of pixels) of the composite image (display ratio of the virtual object). The ratio of the area of ​​real space in the composite image where the virtual object is not superimposed to the entire area of ​​the composite image (display ratio of the real space) may also be calculated and stored in the memory 106.

[0026] The display unit 111 is configured with an LCD (liquid crystal panel), an organic EL panel, or the like, and displays the composite image generated by the image composition unit 110. The display unit 111 has a left eye display unit 111a and a right eye display unit 111b, which perform display independently. For example, the left eye display unit 111a presents (displays) a left eye composite image to the left eye of a user wearing the head-mounted display device 100, and the right eye display unit 111b presents (displays) a right eye composite image to the right eye of the user. The method by which the display unit 111 presents the composite image to the user is not particularly limited. For example, the display unit 111 may directly project the composite image onto the user's retina using a laser or the like.

[0027] (user's perspective) FIG. 2 is a schematic diagram showing the field of view of a user 200 wearing the head-mounted display device 100. The visual field of the user 200 includes a central visual field 201 including the gaze position, and a peripheral visual field 202 outside the central visual field 201. In the central visual field 201, the user 200 is capable of high-resolution recognition and can distinguish fine differences. In the peripheral visual field 202, the user 200 is only capable of low-resolution recognition and can only distinguish rough movements. For this reason, the resolution of the area including the user's gaze position in the composite image cannot be reduced, but the resolution of the area away from the gaze position may be reduced.

[0028] (Head-mounted display operation) Fig. 3 is a flowchart showing the operation of the head-mounted display device 100. The operation of Fig. 3 starts in a state where the user is wearing the head-mounted display device 100 and is viewing a composite image.

[0029] In step S301, the control unit 105 controls the eye imaging unit 103 to acquire an image of the user's eye and store the image in the memory 106. Then, the control unit 105 controls the gaze detection unit 108 to acquire information about the user's gaze position from the image of the eye stored in the memory 106 and store the information about the gaze position in the memory 106.

[0030] In step S302, the control unit 105 controls the movement amount detection unit 107 to acquire information about the amount of movement of the user, and stores the information about the amount of movement in the memory .

[0031] In step S303, control unit 105 controls background imaging unit 101 to acquire an image of real space. Then, control unit 105 controls image processing unit 104 to perform image processing on the acquired image of real space, and stores the processed image in memory 106. Details of the control of background imaging unit 101 in step S303 will be described later with reference to FIG. 4.

[0032] In step S304, the control unit 105 controls the marker imaging unit 102 to acquire an image of the real space. Then, the control unit 105 controls the image processing unit 104 to perform image processing on the acquired image of the real space, and stores the processed image in the memory 106. Details of the control of the marker imaging unit 102 in step S304 will be described later with reference to FIG. 6.

[0033] In step S305, control unit 105 controls virtual object generation unit 109 to generate an image of a virtual object from the image of real space stored in memory 106 in step S304, and stores the image in memory 106. Then, control unit 105 controls image synthesis unit 110 to generate a composite image from the image of the virtual object stored in memory 106 and the image of real space stored in memory 106 in step S303, and stores the composite image in memory 106.

[0034] In step S306, the control unit 105 causes the display unit 111 to display the composite image stored in the memory 106 in step S305.

[0035] In step S307, the control unit 105 determines whether or not to terminate the operation of the head-mounted display device 100. If the control unit 105 determines that the operation of the head-mounted display device 100 is to be terminated, the control unit 105 terminates the operation of Fig. 3, and if the control unit 105 determines that the operation of the head-mounted display device 100 is not to be terminated, the control unit 105 proceeds to step S301. For example, when the control unit 105 detects that the user has removed the head-mounted display device 100 from the head, the control unit 105 determines that the operation of the head-mounted display device 100 is to be terminated.

[0036] (Control of background imaging unit) FIG. 4 is a flowchart showing the control of the background image capture unit 101 in step S303 of FIG. It is.

[0037] In step S401, the control unit 105 determines whether the user wearing the head-mounted display device 100 is moving, based on the movement amount information acquired in step S302 of Fig. 3. If the control unit 105 determines that the user is not moving, the process proceeds to step S402, and if the control unit 105 determines that the user is moving, the process proceeds to step S403. For example, if the movement amount indicated by the movement amount information is smaller than a threshold, the control unit 105 determines that the user is not moving, and if the movement amount indicated by the movement amount information is larger than the threshold, the control unit 105 determines that the user is moving. If the movement amount indicated by the movement amount information is equal to the threshold, the control unit 105 may determine that the user is not moving, or may determine that the user is moving.

[0038] In step S402, the control unit 105 determines whether the user is looking at real space or a virtual object based on the gaze position information acquired in step S301 of Fig. 3. This determination may be interpreted as determining whether the gaze position indicated by the gaze position information is in the real space area or the virtual object area in the displayed composite image. If the control unit 105 determines that the user is looking at real space, the process proceeds to step S403; if the control unit 105 determines that the user is looking at a virtual object, the process proceeds to step S404.

[0039] In step S403, control unit 105 sets the frame rate of the image captured by background imaging unit 101 (the frame rate of the image captured by background imaging unit 101) to "high frame rate."

[0040] In step S404, control unit 105 sets the frame rate of the image captured by background imaging unit 101 (the frame rate of the image captured by background imaging unit 101) to a "low frame rate" lower than the "high frame rate."

[0041] In step S405, the control unit 105 controls the background image capturing unit 101 so as to capture an image of the real space at the frame rate set in step S403 or step S404.

[0042] FIG. 5 shows the correspondence relationship between information on the user's gaze position, information on the user's movement amount, and the frame rate of background image capturing unit 101.

[0043] When the user is moving, the frame rate of background imaging unit 101 is set (controlled) to a high frame rate regardless of the user's gaze position. When the user is moving, the image of real space displayed on display unit 111 is constantly changing. By setting the frame rate of background imaging unit 101 to a high frame rate, it is possible to reduce the delay in the change in the image of real space in response to the user's movement, thereby reducing the sense of discomfort felt by the user.

[0044] When the user is not moving (is stationary), the frame rate of background image capturing unit 101 is switched depending on the position of the user's line of sight.

[0045] When the user's line of sight is in the real space area (when the user is looking at the real space), the user captures an image of the real space in the central visual field 201. In the central visual field 201, the user 200 is capable of high-resolution recognition and can distinguish fine differences. Setting the frame rate of the background imaging unit 101 to a high frame rate can reduce the sense of discomfort felt by the user.

[0046] When the user's gaze position is in the area of ​​the virtual object (when the user is looking at the virtual object), the user captures an image of the virtual object in central vision 201. The image of real space is captured in peripheral vision 202. In peripheral vision 202, user 200 can only perceive low-resolution images and can only distinguish rough movements. Therefore, even if the frame rate of background imaging unit 101 is set to a low frame rate, the user does not feel uncomfortable. By setting the frame rate of background imaging unit 101 to a low frame rate, the power consumption of head-mounted display device 100 can be reduced.

[0047] In the first embodiment, when the user is not moving and is looking at a virtual object, the frame rate of the background image capturing unit 101 is set to a low frame rate, and otherwise the frame rate of the background image capturing unit 101 is set to a high frame rate. This makes it possible to reduce the power consumption of the head-mounted display device 100 while preventing the user wearing the head-mounted display device 100 from feeling a deterioration in image quality or a sense of discomfort.

[0048] (Control of marker imaging unit) FIG. 6 is a flowchart showing the control of the marker image capturing unit 102 in step S304 of FIG.

[0049] In step S601, the control unit 105 determines whether or not the user wearing the head-mounted display device 100 is moving, based on the movement amount information acquired in step S302 of Fig. 3. If the control unit 105 determines that the user is not moving, the process proceeds to step S602, and if the control unit 105 determines that the user is moving, the process proceeds to step S603.

[0050] In step S602, the control unit 105 determines whether the user is looking at real space or a virtual object based on the gaze position information acquired in step S301 of Fig. 3. If the control unit 105 determines that the user is looking at real space, the process proceeds to step S604, and if the control unit 105 determines that the user is looking at a virtual object, the process proceeds to step S603.

[0051] In step S603, the control unit 105 sets the frame rate of the image captured by the marker image capturing unit 102 (the frame rate of the image captured by the marker image capturing unit 102) to "high frame rate."

[0052] In step S604, the control unit 105 sets the frame rate of the image captured by the marker image capturing unit 102 (the frame rate of the image captured by the marker image capturing unit 102) to a "low frame rate" lower than the "high frame rate."

[0053] In step S605, the control unit 105 controls the marker image capturing unit 102 so as to capture an image of the real space at the frame rate set in step S603 or step S604.

[0054] 7 shows the correspondence relationship between the user's gaze position information, the user's movement amount information, and the frame rate of the marker imaging unit 102. As described above, the image of the virtual object is generated at a position and orientation based on the image (image of the marker) obtained by the marker imaging unit 102.

[0055] When the user is moving, the frame rate of the marker image capturing unit 102 is set (controlled) to a high frame rate regardless of the user's line of sight position. When the user is moving, the image of the virtual object displayed on the display unit 111 is constantly changing. By setting the frame rate of 02 to a high frame rate, the delay in changes in the image of the virtual object in response to the user's movements can be reduced, reducing the sense of discomfort felt by the user.

[0056] When the user is not moving (is stationary), the frame rate of the marker imaging unit 102 is switched depending on the position of the user's line of sight.

[0057] When the user's line of sight is in the area of ​​the virtual object (when the user is looking at the virtual object), the user captures the image of the virtual object in the central visual field 201. In the central visual field 201, the user 200 is capable of high-resolution recognition and can distinguish fine differences. Setting the frame rate of the marker imaging unit 102 to a high frame rate can reduce the sense of discomfort felt by the user.

[0058] When the user's gaze position is in the real space area (when the user is looking at the real space), the user captures an image of the real space in the central visual field 201. The image of the virtual object is captured in the peripheral visual field 202. In the peripheral visual field 202, the user 200 can only recognize low-resolution images and can only distinguish rough movements. Therefore, even if the frame rate of the marker image capture unit 102 is set to a low frame rate, the user does not feel uncomfortable. By setting the frame rate of the marker image capture unit 102 to a low frame rate, the power consumption of the head-mounted display device 100 can be reduced.

[0059] In the first embodiment, when the user is not moving and is looking at the real space, the frame rate of the marker image capturing unit 102 is set to a low frame rate, and otherwise the frame rate of the marker image capturing unit 102 is set to a high frame rate. This also makes it possible to reduce the power consumption of the head-mounted display device 100 while suppressing the user wearing the head-mounted display device 100 from feeling degradation in image quality or discomfort.

[0060] Although an example has been described in which the power consumption of the head-mounted display device 100 is controlled by controlling the frame rate of the imaging units (background imaging unit 101 and marker imaging unit 102), the method of controlling the power consumption of the head-mounted display device 100 is not limited to this. It is sufficient to control the operation of the imaging units so as to control the power consumption of the head-mounted display device 100, and for example, it is also possible to control the resolution of an image captured by the imaging unit or the angle of view of the image. The resolution can be reduced by treating multiple pixels as one pixel or thinning out pixels. Then, by reducing the resolution, the power consumption of the head-mounted display device 100 can be reduced. Furthermore, by narrowing the angle of view, the power consumption of the head-mounted display device 100 can be reduced. The power consumption of the head-mounted display device 100 can also be reduced by reducing the readout speed of signals from the imaging units.

[0061] Furthermore, if the user's head movement can be detected as a gesture, it is considered that the user is not paying much attention to the composite image while making the gesture. Therefore, the gesture movement (for example, the user's movement of only shaking their head up and down or left and right) may not be regarded as the user's movement, and the process may proceed from step S401 to step S402, or from step S601 to step S602.

[0062] Furthermore, it is also possible to control only one of the background image capturing unit 101 and the marker image capturing unit 102. The control of the marker image capturing unit 102 may be applied to an optical see-through type HMD.

[0063] Although the example of switching the frame rate between a high frame rate and a low frame rate has been described, the operation of the imaging unit may be switched between three or more operations. The power consumption of the head-mounted display device 100 may be switched between different power consumption levels. For example, when the user is not moving, lower power consumption may be set than when the user is moving. When the user is not moving, the power consumption may be switched depending on whether the user is viewing a virtual object or real space.

[0064] <Example 2> A second embodiment of the present invention will be described. In the second embodiment, it is determined whether or not the user is looking at an image of a virtual object (such as a menu) placed at a position not based on the real space (a position not based on the position of a marker), thereby reducing the power consumption of the head-mounted display device 100. Note that, in the following, a description of the same configuration and processing as in the first embodiment will be omitted, and a description will be given of the configuration and processing different from the first embodiment.

[0065] (Control of background imaging unit) 8 is a flowchart showing the control of the background image capturing unit 101 according to the embodiment 2. The control in FIG. 8 is performed in step S303 in FIG.

[0066] In step S801, the control unit 105 determines whether the user is looking at a specific virtual object based on the gaze position information acquired in step S301 of Fig. 3. The specific virtual object is a virtual object (such as a menu) that is placed at a position that is not based on the real space (a position that is not based on the position of a marker). Hereinafter, this specific virtual object will be referred to as "additional information." If the control unit 105 determines that the user is not looking at the additional information, the process proceeds to step S802, and if the control unit 105 determines that the user is looking at the additional information, the process proceeds to step S805.

[0067] Steps S802 to S806 are the same as steps S401 to S405 in FIG.

[0068] According to the above control, when the user is viewing additional information, the frame rate of the background imaging unit 101 is set to a low frame rate without being based on the movement amount information, thereby reducing the power consumption of the head-mounted display device 100. When the user is viewing additional information, the user is not viewing the real space, and therefore the user does not feel uncomfortable even if the frame rate of the background imaging unit 101 is set to a low frame rate.

[0069] (Control of marker imaging unit) 9 is a flowchart showing the control of the marker image capturing unit 102 according to the embodiment 2. The control in FIG. 9 is performed in step S304 in FIG.

[0070] In step S901, the control unit 105 determines whether or not the user is looking at additional information (a specific virtual object) based on the gaze position information acquired in step S301 of Fig. 3. If the control unit 105 determines that the user is not looking at additional information, the process proceeds to step S902, and if the control unit 105 determines that the user is looking at additional information, the process proceeds to step S905.

[0071] Steps S902 to S906 are the same as steps S601 to S605 in FIG.

[0072] According to the above control, when the user is viewing additional information, the frame rate of the marker image capturing unit 102 is set to a low frame rate without being based on the movement amount information, thereby reducing the power consumption of the head-mounted display device 100. When the user is viewing additional information, the user is not viewing the virtual object placed at the position of the marker (a position based on the position of the marker), and therefore the user does not feel uncomfortable even if the frame rate of the background image capturing unit 101 is set to a low frame rate.

[0073] Example 3 A third embodiment of the present invention will be described. In the third embodiment, power consumption of the head-mounted display device 100 is reduced based on the display ratio of the virtual object (or the display ratio of the real space). As described in the first embodiment, the display ratio of the virtual object is the ratio of the area (number of pixels) of the virtual object in the composite image to the entire area (total number of pixels) of the composite image, and indicates the size of the area of ​​the image of the virtual object within the range visible to the user. The display ratio of the real space is the ratio of the area of ​​the real space in the composite image where the virtual object is not superimposed to the entire area of ​​the composite image, and indicates the size of the area of ​​the image of the real space within the range visible to the user. Note that, hereinafter, a description of the same configuration and processing as in the first embodiment will be omitted, and a description will be given of configurations and processing different from the first embodiment.

[0074] (Control of background imaging unit) 10 is a flowchart showing the control of the background image capturing unit 101 according to the embodiment 3. The control in FIG. 10 is performed in step S303 in FIG.

[0075] Step S1001 is the same as step S401 in Fig. 4, and step S1002 is the same as step S402. In step S1002, if the control unit 105 determines that the user is looking at real space, the process proceeds to step S1004, and if the control unit 105 determines that the user is looking at a virtual object, the process proceeds to step S1003.

[0076] In step S1003, control unit 105 determines whether the display ratio of the virtual object is greater than a threshold. If control unit 105 determines that the display ratio of the virtual object is smaller than the threshold, control unit 105 proceeds to step S1004, and if control unit 105 determines that the display ratio of the virtual object is greater than the threshold, control unit 105 proceeds to step S1005. If the display ratio of the virtual object is equal to the threshold, control unit 105 may proceed to step S1004 or may proceed to step S1005. The composite image is generated in step S305 of FIG. 3 (a step after the control (step S303) of FIG. 10). In step S1003, for example, the display ratio calculated when the previous composite image (the composite image of one frame before) was generated is used.

[0077] In step S1003, it may be determined whether the display ratio of the real space is greater than a threshold value. In this case, if the display ratio of the real space is greater than the threshold value, the process proceeds to step S1004, and if the display ratio of the real space is less than the threshold value, the process proceeds to step S1005.

[0078] Steps S1004 to S1006 are the same as steps S403 to S405 in FIG.

[0079] Even if the user's gaze position is within the area of ​​a virtual object, if the display ratio of the virtual object is small (the display ratio of real space is large), the area of ​​real space will be included in the user's central visual field 201. In such a case, if the frame rate of the image of real space (the frame rate of the background imaging unit 101) is set to a low frame rate, the user may perceive a deterioration in image quality or feel uncomfortable. For example, the difference between the frame rate of the image of real space and the frame rate of the virtual object increases, causing a positional deviation between the real space and the virtual object, which causes a sense of discomfort to the user.

[0080] Therefore, in the control of FIG. 10, even if the user's line of sight is in the area of ​​a virtual object, if the display ratio of the virtual object is small, the frame rate of the background image capture unit 101 is set to a high frame rate. By doing so, it is possible to reduce the sense of discomfort felt by the user. In addition, even if the user's line of sight is in the area of ​​a virtual object, if the display ratio of the virtual object is small, the frame rate of the background image capture unit 101 is set to a high frame rate. If the display ratio is large, the frame rate of the background image capturing unit 101 is set to a low frame rate. By doing so, the power consumption of the head-mounted display device 100 can be reduced.

[0081] (Control of marker imaging unit) 11 is a flowchart showing the control of the marker image capturing unit 102 according to the embodiment 3. The control in FIG. 11 is performed in step S304 in FIG.

[0082] Step S1101 is the same as step S601 in Fig. 6, and step S1102 is the same as step S602. In step S1102, if the control unit 105 determines that the user is looking at real space, the process proceeds to step S1103, and if the control unit 105 determines that the user is looking at a virtual object, the process proceeds to step S1104.

[0083] In step S1103, the control unit 105 determines whether the display ratio of the real space is greater than a threshold. If the control unit 105 determines that the display ratio of the real space is smaller than the threshold, the control unit 105 proceeds to step S1104, and if the control unit 105 determines that the display ratio of the real space is greater than the threshold, the control unit 105 proceeds to step S1105. If the display ratio of the real space is equal to the threshold, the control unit 105 may proceed to step S1104 or may proceed to step S1105. The composite image is generated in step S305 of FIG. 3 (a step subsequent to the control (step S304) of FIG. 11). In step S1103, for example, the display ratio calculated when the previous composite image (the composite image of one frame before) was generated is used.

[0084] In step S1103, it may be determined whether the display ratio of the virtual object is greater than a threshold value. In this case, if the display ratio of the virtual object is greater than the threshold value, the process proceeds to step S1104, and if the display ratio of the virtual object is less than the threshold value, the process proceeds to step S1105.

[0085] Steps S1104 to S1106 are the same as steps S603 to S605 in FIG.

[0086] Even if the user's gaze position is within the real space area, if the display ratio of the real space is small (the display ratio of the virtual object is large), the area of ​​the virtual object will be within the user's central visual field 201. In such a case, if the frame rate of the image of the virtual object (the frame rate of the marker imaging unit 102) is set to a low frame rate, the user may feel a deterioration in image quality or may feel uncomfortable. For example, the difference between the frame rate of the image of the real space and the frame rate of the virtual object increases, causing a positional deviation between the real space and the virtual object, which may cause a sense of discomfort to the user.

[0087] Therefore, in the control of FIG. 11, even if the user's gaze position is in the real space area, if the display ratio of the real space is small, the frame rate of the marker image capture unit 102 is set to a high frame rate. This can reduce the sense of discomfort felt by the user. Also, if the user's gaze position is in the real space area and the display ratio of the real space is large, the frame rate of the marker image capture unit 102 is set to a low frame rate. This can reduce the power consumption of the head-mounted display device 100.

[0088] Example 4 A fourth embodiment of the present invention will be described. In the fourth embodiment, the number of times an image of a virtual object is generated and the number of times a composite image is generated are reduced, thereby reducing the power consumption of the head-mounted display device 100. Note that, in the following, a description of the same configuration and processing as in the first embodiment will be omitted, and only configurations and processing different from those in the first embodiment will be described.

[0089] 12 is a flowchart showing a synthetic image generation process according to Example 4. The generation process in FIG. 12 is performed in step S305 in FIG.

[0090] In step S1201, control unit 105 determines whether or not to generate an image of a virtual object (whether or not to use the previous image of the virtual object without generating a new image of the virtual object) based on the movement amount information acquired in step S302 of Fig. 3. If control unit 105 determines that an image of the virtual object is to be generated, the process proceeds to step S1202, and if control unit 105 determines that an image of the virtual object is not to be generated, the process proceeds to step S1203.

[0091] For example, control unit 105 determines to generate an image of a virtual object when the user is moving, and determines not to generate an image of a virtual object when the user is not moving. If the virtual object for which the determination of whether to generate an image is made is a changeable (e.g., movable) virtual object, the state of the virtual object may be taken into consideration. For example, when the user is not moving, it may determine to generate an image of the virtual object if it is a timing when the virtual object changes, and it may determine not to generate an image of the virtual object if it is a timing when the virtual object does not change.

[0092] In the generation process of Fig. 12, when the user is not moving (and the virtual object is not changing), an image of the virtual object is not generated (the previous image of the virtual object is used instead of generating a new image). By not generating an image of the virtual object, it is possible to reduce the power consumption of head-mounted display device 100. Since the image of the virtual object generated when the user is not moving (and the virtual object is not changing) is the same as the previous image (one frame before), using the previous image does not give the user a sense of discomfort.

[0093] In step S1202, the control unit 105 controls the virtual object generation unit 109 to generate an image of a virtual object from the image of real space stored in the memory in step S304, and stores the image in the memory .

[0094] In step S1203, control unit 105 determines whether or not to generate a composite image (whether or not to use the previous composite image without generating a new one) based on the imaging result of background imaging unit 101. If control unit 105 determines to generate a composite image, it proceeds to step S1204, and if control unit 105 determines not to generate a composite image, it ends the generation process of FIG. 12. If the generation process of FIG. 12 ends without generating a composite image, in step S306 of FIG. 3, control unit 105 causes display unit 111 to display the composite image generated previously (one frame before).

[0095] For example, control unit 105 determines to generate a composite image when there is a change in the image of real space obtained by background imaging unit 101, and determines not to generate a composite image when there is no change in the image of real space obtained by background imaging unit 101. The method of determining whether there is a change in the image of real space is not particularly limited. For example, it may be determined whether there is a change in the image of real space from the previous time to the current time by comparing the previous composite image (one frame before) with the current composite image (current frame) and determining whether the difference between the composite images is greater than a threshold value.

[0096] In the generation process of FIG. 12, if the previous image of the virtual object is used and there is no change in the image of real space that serves as the background of the composite image, the composite image is not generated (the previous composite image is used without generating a new composite image). By not generating a composite image, it is possible to reduce the power consumption of the head-mounted display device 100. If the previous image of the virtual object is used and there is no change in the image of real space that serves as the background of the composite image, the composite image is not generated (the previous composite image is used without generating a new composite image). The composite image to be displayed is the same as the previous one (one frame before), so the user does not feel uncomfortable even if the previous composite image is used.

[0097] In step S1204, control unit 105 controls image synthesis unit 110 to generate a synthetic image from the image of the virtual object stored in memory 106 and the image of real space, and stores the synthetic image in memory 106. If the processing of step S1202 has been performed, the current image of the virtual object (the image generated in step S1202) is used, and if the processing of step S1202 has not been performed, the previous image of the virtual object is used.

[0098] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0099] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0100] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0101] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0102] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A head-mounted display device, an imaging means for imaging a real space; a gaze detection means for detecting the gaze of a user wearing the head-mounted display device on their head; a motion detection means for detecting the user's motion; a control means for controlling the operation of the imaging means so as to control the power consumption of the head-mounted display device based on the detection result of the movement detection means and the detection result of the line-of-sight detection means; A head-mounted display device comprising: (Configuration 2) the head-mounted display device displays a composite image in which an image of a virtual object is superimposed on the image of the real space captured by the imaging means, When the user is not moving and the user is looking at the virtual object and the control means controls the operation of the imaging means so as to control the power consumption of the head-mounted display device to be lower than the power consumption of the head-mounted display device when the user is not moving and the user is looking at the real space. 2. The head-mounted display device according to configuration 1, (Configuration 3) a detection unit that detects a position that serves as a reference for the position of a virtual object from the image of the real space captured by the imaging unit; the head-mounted display device displays an image of the virtual object such that the virtual object is placed at a position based on the position detected by the detection means; When the user is not moving and is looking at the real space, the control means controls the operation of the imaging means so as to control the power consumption of the head-mounted display device to be lower than the power consumption of the head-mounted display device when the user is not moving and is looking at the virtual object. 2. The head-mounted display device according to configuration 1, (Configuration 4) the head-mounted display device displays an image of a virtual object; When the user is looking at a specific virtual object, the control means controls the operation of the imaging means without being based on a detection result of the motion detection means. 2. The head-mounted display device according to configuration 1, (Configuration 5) the head-mounted display device displays a composite image in which an image of the virtual object is superimposed on an image of the real space captured by the imaging means, The control means controls the operation of the imaging means so that power consumption of the head-mounted display device is controlled to be lower in each of a case where the user is looking at the specific virtual object and a case where the user is not moving and is looking at a virtual object other than the specific virtual object than power consumption of the head-mounted display device in a case where the user is not moving and is looking at the real space. 5. The head-mounted display device according to configuration 4. (Configuration 6) the imaging device further includes a detection unit that detects a position that serves as a reference for the position of a virtual object other than the specific virtual object from the image of the real space captured by the imaging unit, the head-mounted display device displays an image of a virtual object other than the specific virtual object so that the virtual object is placed at a position based on the position detected by the detection means, and displays an image of the specific virtual object regardless of the position detected by the detection means; In each of a case where the user is looking at the specific virtual object and a case where the user is not moving and is looking at the real space, the control means controls the operation of the imaging means so as to control the power consumption of the head-mounted display device to be lower than the power consumption of the head-mounted display device in a case where the user is not moving and is looking at the virtual object other than the specific virtual object. 5. The head-mounted display device according to configuration 4. (Configuration 7) The specific virtual object is a virtual object that is placed at a position that is not based on the real space. 7. The head-mounted display device according to any one of configurations 4 to 6, wherein: (Configuration 8) the head-mounted display device displays an image of a virtual object; The control means receives the detection result of the movement detection means, the detection result of the line of sight detection means, and The operation of the imaging means is controlled based on the size of an area of ​​the image of the virtual object within a range visible to the user. 2. The head-mounted display device according to configuration 1, (Configuration 9) the head-mounted display device displays a composite image in which an image of the virtual object is superimposed on an image of the real space captured by the imaging means, The control means when the size of the area of ​​the image of the virtual object is smaller than a threshold value even when the user is not moving and is looking at the virtual object, the power consumption of the head-mounted display device is not controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and is looking at the real space, When the user is not moving, the user is looking at the virtual object, and the size of the area of ​​the image of the virtual object is larger than the threshold value, the operation of the imaging means is controlled so that the power consumption of the head-mounted display device is controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and the user is looking at the real space. 9. The head-mounted display device according to configuration 8, (Configuration 10) the imaging device further includes a detection unit that detects a position that serves as a reference for the position of the virtual object from the image of the real space captured by the imaging unit, the head-mounted display device displays an image of the virtual object such that the virtual object is placed at a position based on the position detected by the detection means; The control means when the size of the area of ​​the image of the virtual object is larger than a threshold value even when the user is not moving and is looking at the real space, the power consumption of the head-mounted display device is not controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and is looking at the virtual object, When the user is not moving, the user is looking at the real space, and the size of the area of ​​the image of the virtual object is smaller than the threshold value, the operation of the imaging means is controlled so that the power consumption of the head-mounted display device is controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and the user is looking at the virtual object. 9. The head-mounted display device according to configuration 8, (Configuration 11) The control means controls at least one of a frame rate, a resolution, and an angle of view of an image captured by the imaging means, thereby controlling power consumption of the head-mounted display device. 11. The head-mounted display device according to any one of configurations 1 to 10. (Configuration 12) the head-mounted display device generates and displays an image of a virtual object; The control means further controls whether to use a previous image of the virtual object without generating a new image based on the detection result of the motion detection means. 2. The head-mounted display device according to configuration 1, (Configuration 13) When the user is not moving, the control means performs control so that a previous image of the virtual object is used instead of generating a new image of the virtual object. 13. The head-mounted display device according to configuration 12. (Configuration 14) the head-mounted display device generates and displays a composite image by superimposing an image of the virtual object on the image of the real space captured by the imaging means; When a previous image of the virtual object is used without generating a new image, the control means further controls whether or not to use the previous composite image without generating a new composite image based on the image capturing result of the imaging means. 14. The head-mounted display device according to configuration 12 or 13. (Configuration 15) When there is no change in the image of the real space captured by the imaging means, the control means performs control so as to use the previous composite image without generating a new composite image. 15. The head-mounted display device according to configuration 14. (Configuration 16) The control means controls the operation of the imaging means without regarding the user's movement of only shaking the head as the user's movement. 16. The head-mounted display device according to any one of configurations 1 to 15. (method) A method for controlling a head-mounted display device, comprising: an imaging step of imaging a real space; a gaze detection step of detecting a gaze of a user wearing the head-mounted display device on their head; a motion detection step of detecting a motion of the user; a control step of controlling an operation of the imaging step so as to control power consumption of the head-mounted display device based on the detection result of the movement detection step and the detection result of the line-of-sight detection step; A control method comprising: (program) A program for causing a computer to function as each means of the head-mounted display device according to any one of configurations 1 to 16. [Explanation of symbols]

[0103] 100: Head-mounted display device 101: Background image capturing unit 102: Marker image capturing unit 105: Control unit 107: Movement amount detection unit 108: Line of sight detection unit

Claims

1. A head-mounted display device, an imaging means for imaging a real space; a gaze detection means for detecting the gaze of a user wearing the head-mounted display device on their head; a motion detection means for detecting the user's motion; a control means for controlling the operation of the imaging means so as to control the power consumption of the head-mounted display device based on the detection result of the movement detection means and the detection result of the line-of-sight detection means; A head-mounted display device comprising:

2. the head-mounted display device displays a composite image in which an image of a virtual object is superimposed on the image of the real space captured by the imaging means, When the user is not moving and is looking at the virtual object, the control means controls the operation of the imaging means so as to control the power consumption of the head-mounted display device to be lower than the power consumption of the head-mounted display device when the user is not moving and is looking at the real space.

2. The head-mounted display device according to claim 1.

3. a detection unit that detects a position that serves as a reference for the position of a virtual object from the image of the real space captured by the imaging unit; the head-mounted display device displays an image of the virtual object such that the virtual object is placed at a position based on the position detected by the detection means; When the user is not moving and is looking at the real space, the control means controls the operation of the imaging means so as to control the power consumption of the head-mounted display device to be lower than the power consumption of the head-mounted display device when the user is not moving and is looking at the virtual object.

2. The head-mounted display device according to claim 1.

4. the head-mounted display device displays an image of a virtual object; When the user is looking at a specific virtual object, the control means controls the operation of the imaging means without being based on a detection result of the motion detection means.

2. The head-mounted display device according to claim 1.

5. the head-mounted display device displays a composite image in which an image of the virtual object is superimposed on an image of the real space captured by the imaging means, The control means controls the operation of the imaging means so that power consumption of the head-mounted display device is controlled to be lower in each of a case where the user is looking at the specific virtual object and a case where the user is not moving and is looking at a virtual object other than the specific virtual object than power consumption of the head-mounted display device in a case where the user is not moving and is looking at the real space.

5. The head-mounted display device according to claim 4.

6. the imaging device further includes a detection unit that detects a position that serves as a reference for the position of a virtual object other than the specific virtual object from the image of the real space captured by the imaging unit, the head-mounted display device displays an image of a virtual object other than the specific virtual object so that the virtual object is placed at a position based on the position detected by the detection means, and displays an image of the specific virtual object regardless of the position detected by the detection means; When the user is looking at the particular virtual object and when the user is moving, and when the user is not moving and the user is looking at the real space, the control means controls the operation of the imaging means so as to control the power consumption of the head-mounted display device to be lower than the power consumption of the head-mounted display device when the user is not moving and the user is looking at the virtual object other than the specific virtual object.

5. The head-mounted display device according to claim 4.

7. The specific virtual object is a virtual object that is placed at a position that is not based on the real space.

5. The head-mounted display device according to claim 4.

8. the head-mounted display device displays an image of a virtual object; The control means controls the operation of the imaging means based on the detection result of the movement detection means, the detection result of the line of sight detection means, and the size of the area of ​​the image of the virtual object within the range visible to the user.

2. The head-mounted display device according to claim 1.

9. the head-mounted display device displays a composite image in which an image of the virtual object is superimposed on an image of the real space captured by the imaging means, The control means when the size of the area of ​​the image of the virtual object is smaller than a threshold value even when the user is not moving and is looking at the virtual object, the power consumption of the head-mounted display device is not controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and is looking at the real space, When the user is not moving, the user is looking at the virtual object, and the size of the area of ​​the image of the virtual object is larger than the threshold value, the operation of the imaging means is controlled so that the power consumption of the head-mounted display device is controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and the user is looking at the real space.

9. The head-mounted display device according to claim 8.

10. the imaging device further includes a detection unit that detects a position that serves as a reference for the position of the virtual object from the image of the real space captured by the imaging unit, the head-mounted display device displays an image of the virtual object such that the virtual object is placed at a position based on the position detected by the detection means; The control means when the size of the area of ​​the image of the virtual object is larger than a threshold value even when the user is not moving and is looking at the real space, the power consumption of the head-mounted display device is not controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and is looking at the virtual object, When the user is not moving, the user is looking at the real space, and the size of the area of ​​the image of the virtual object is smaller than the threshold value, the operation of the imaging means is controlled so that the power consumption of the head-mounted display device is controlled to be lower than the power consumption of the head-mounted display device when the user is not moving and the user is looking at the virtual object.

9. The head-mounted display device according to claim 8.

11. The control means controls the frame rate, resolution, and and at least one of the angle of view, thereby controlling the power consumption of the head-mounted display device.

2. The head-mounted display device according to claim 1.

12. the head-mounted display device generates and displays an image of a virtual object; The control means further controls whether to use a previous image of the virtual object without generating a new image based on the detection result of the motion detection means.

2. The head-mounted display device according to claim 1.

13. When the user is not moving, the control means performs control so that a previous image of the virtual object is used instead of generating a new image of the virtual object.

13. The head-mounted display device according to claim 12.

14. the head-mounted display device generates and displays a composite image by superimposing an image of the virtual object on the image of the real space captured by the imaging means; When a previous image of the virtual object is used without generating a new image, the control means further controls whether or not to use the previous composite image without generating a new composite image based on the image capturing result of the imaging means.

13. The head-mounted display device according to claim 12.

15. When there is no change in the image of the real space captured by the imaging means, the control means performs control so as to use the previous composite image without generating a new composite image.

15. The head-mounted display device according to claim 14.

16. The control means controls the operation of the imaging means without regarding the user's movement of only shaking the head as the user's movement.

2. The head-mounted display device according to claim 1.

17. A method for controlling a head-mounted display device, comprising: an imaging step of imaging a real space; a gaze detection step of detecting a gaze of a user wearing the head-mounted display device on their head; a motion detection step of detecting a motion of the user; a control step of controlling an operation of the imaging step so as to control power consumption of the head-mounted display device based on the detection result of the movement detection step and the detection result of the line-of-sight detection step; A control method comprising:

18. A program for causing a computer to function as each of the means of the head-mounted display device according to any one of claims 1 to 16.

Images