Display device, display device control method, and program

Abstract

To provide a display device, a display device control method, and a program which are capable of ensuring the safety of a user from the surroundings even if the user wearing the display device moves in real space.SOLUTION: An HMD 100 includes: first acquisition means (operation unit information acquisition unit 105) capable of acquiring position information relating to the position in real space of an operation unit 202 that performs an operation on the HMD 100; second acquisition means (operation unit information acquisition unit 105) for acquiring the movement limit of the operation unit 202; setting means (control unit 106) for setting a safety area 203 including the HMD 100 and the operation unit 202; and determination means (control unit 106) for determining whether or not the operation unit 202 is approaching a boundary 203a of the safety area 203 based on the position information when the position information of the operation unit 202 is acquired, and for determining whether or not the operation unit 202 is approaching the boundary 203a based on the movement limit when the position information of the operation unit 202 is not acquired.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a display device, a control method for a display device, and a program. [Background technology]

[0002] In recent years, head mounted displays (hereinafter referred to as "HMDs") have been utilized in various industries and fields, such as games, education, medicine, training, and design. When worn on the user's head, the HMD can project or display images of a virtual space on a display panel that corresponds to the user's field of vision. When the user moves his or her head, the HMD can change the field of vision that can be displayed in the virtual space according to the movement. This can enhance the user's sense of immersion in the virtual space.

[0003] Patent Document 1 discloses an HMD that displays a composite image, which is a composite image of a real space image and a virtual space image around the user, according to the distance between the HMD and the boundary when the HMD worn on the user's head approaches the boundary of a specific area in the real space. When the HMD displays the composite image, it can adjust the brightness level of the image and notify the user of a warning. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-175204 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the HMD described in Patent Document 1 cannot issue a warning to the user even if the user's hand, which has a wider range of motion than the head on which the HMD is worn, or the controller held in the hand approaches the boundary of a specific area. In this case, there is a problem in that it is difficult to ensure the safety of the user from the surroundings.

[0006] The present invention has been made in view of the above problems, and aims to provide a display device, a control method for the display device, and a program that can ensure the safety of a user from the surroundings even when the user wearing the display device moves in real space. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the display device of the present invention is a display device that is detachably worn by a user and displays an image of a virtual space while worn, and is characterized in that it comprises a first acquisition means that is movable relative to the display device within a real space in which the display device is used while worn and is capable of acquiring position information regarding a position in the real space of an operation unit that operates the display device, a second acquisition means that acquires a movement limit of the operation unit relative to the display device, a setting means that sets a predetermined area in the real space that includes the display device and the operation unit, and, when position information of the operation unit is acquired by the first acquisition means, determines whether the operation unit is approaching a boundary of the predetermined area based on the position information, and when position information of the operation unit is not acquired by the first acquisition means, determines whether the operation unit is approaching the boundary based on the movement limit. Effect of the Invention

[0008] According to the present invention, even if a user wearing a display device moves in real space, the safety of the user from the surroundings can be ensured. [Brief description of the drawings]

[0009] [Figure 1] FIG. 11 is a block diagram showing an example of a hardware configuration when the display device is applied to an HMD (head mounted display). [Diagram 2] FIG. 1 is a horizontal view of a user using an HMD in a room. [Diagram 3] FIG. 1 is a top view of a user using an HMD within a safety area. [Figure 4A] FIG. 13 is a diagram showing a user using an HMD in a safety area as viewed from above in a state in which position and orientation information of the operation unit is not acquired. [Figure 4B] FIG. 13 is a diagram showing a user using an HMD in a safety area as viewed from above in a state in which position and orientation information of the operation unit is not acquired. [Diagram 5] FIG. 2 is a diagram showing an example of an image visually recognized by a user on an HMD. [Figure 6] 13 is a flowchart showing a virtual reality image display process executed by the HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each of the embodiments. For example, each part constituting the present invention can be replaced with any configuration that can exert the same function. In addition, any configuration may be added.

[0011] FIG. 1 is a block diagram showing an example of a hardware configuration when the display device is applied to an HMD (head mounted display). The HMD 100 shown in FIG. 1 is detachably worn on the head of a user 201. The HMD 100 is a display device capable of displaying an image of a virtual reality space (virtual space) in a state where the HMD 100 is worn on the head of the user 201 (hereinafter referred to as a "worn state"). The HMD 100 has an image input unit 101, an image input unit 102, an image processing unit 103, a position and orientation estimation unit 104, and an operation unit information acquisition unit 105. The HMD 100 also has a control unit 106, a ROM 107, a RAM 108, a communication unit 109, an image projection unit 110, an image projection unit 111, a display unit 112, and a display unit 113. These pieces of hardware included in the HMD 100 are connected to each other so as to be able to communicate with each other via a bus 114. The HMD 100 is also used together with an operation unit 202. The operation unit 202 is a controller that performs operations on the HMD 100 while being held by the hand of the user 201. The operation on the HMD 100 is not particularly limited, and may be, for example, an operation of giving an input to the content displayed on the display unit 112 or the display unit 113. In this embodiment, the operation unit 202 is the hand of the user 201 itself. As a result, when the user 201 moves his / her arm, the operation unit 202 moves relative to the HMD 100 in real space. Since the operation unit 202 is used in a state held by the user 201, it can move relative to the HMD 100 in real space when the user 201 moves his / her arm. Note that, although the operation unit 202 is the hand of the user 201 in this embodiment, it is not limited thereto, and may be, for example, the foot of the user 201, or may be a controller held by the hand of the user 201 or a controller wrapped around the arm of the user 201.

[0012] The image input unit 101 has a lens and an imaging element, and captures a moving image of the real space corresponding to the left eye of the user 201 while the device is being worn. The image input unit 102 has a lens and an imaging element, and captures a moving image of the real space corresponding to the right eye of the user 201 while the device is being worn. The image input unit 101 and the image input unit 102 make the HMD 100 a binocular HMD that captures left and right moving images. The HMD 100 may be a monocular HMD that captures one image. The image processing unit 103 performs various signal processing such as compression / expansion processing, white balance adjustment, interpolation, edge enhancement, gamma correction, and gradation conversion on the image signals captured by the image input unit 101 and the image input unit 102 to generate a captured image. The image processing unit 103 also generates an image of a virtual reality object 501 (see FIG. 5(a) to FIG. 5(c)) arranged in the virtual reality space as a virtual reality image for the left eye and as a virtual reality image for the right eye. The image processing unit 103 can superimpose an image of a virtual reality object 501 arranged in a virtual reality space on a captured image in the real space to generate a mixed reality image for the left eye and also generate a mixed reality image for the right eye. The image processing unit 103 may be implemented as a processing circuit or a computer program executed by the control unit 106.

[0013] The position and orientation estimation unit 104 estimates the three-dimensional position and orientation of the HMD 100 in the real space as position and orientation information (position information). That is, the position and orientation estimation unit 104 acquires position and orientation information related to the position and orientation of the HMD 100 in the real space (first acquisition step). This information is necessary to display a virtual reality image on the HMD 100 in association with the movement of the head of the user 201. The method of estimating the position and orientation information of the HMD 100 by the position and orientation estimation unit 104 is not particularly limited, and for example, SLAM (Simultaneous Localization And Mapping) or other GPS or the like can be used. SLAM can estimate the three-dimensional position and orientation information in the real space by tracking the features of natural objects (for example, points indicating corners) captured in a captured image for each frame. Then, a virtual reality image depicting a predetermined virtual reality object 501 at a predetermined position and orientation can be generated using the position and orientation information of the HMD 100 estimated by the position and orientation estimation unit 104.

[0014] The operation unit information acquisition unit 105 acquires position and orientation information (position information) regarding the position and orientation of the operation unit 202 in the real space (first acquisition step). The method of acquiring the position and orientation information of the operation unit 202 by the operation unit information acquisition unit 105 is not particularly limited, and for example, a method of extracting the operation unit 202 from a captured image of the operation unit 202 and acquiring the position and orientation information of the operation unit 202 based on the extraction result can be used. Alternatively, a GPS or the like can be used. As described above, in this embodiment, the operation unit 202 is the hand of the user 201. In this case, the hand of the user 201, which is the operation unit 202, is imaged by the image input unit 101 and the image input unit 102. Then, by processing this captured image by the image processing unit 103, the joint points of the hand and the wrist of the user 201 can be detected from the captured image, and the position and orientation information of the operation unit 202 can be acquired. Furthermore, when the operation unit 202 is, for example, a controller held by the hand of the user 201, the position and orientation information of the operation unit 202 can be acquired by reading infrared rays emitted from the controller with an infrared camera of the HMD 100. In this embodiment, the position and orientation information of the HMD 100 is acquired by the position and orientation estimation unit 104 (first acquisition means), and the position and orientation information of the operation unit 202 is acquired by the operation unit information acquisition unit 105 (first acquisition means), but this is not limited to this. For example, each piece of position and orientation information may be acquired by one position information acquisition unit (first acquisition means).

[0015] The operation unit information acquisition unit 105 (second acquisition means) can also acquire the movement limit of the operation unit 202 relative to the HMD 100 (second acquisition step). In this embodiment, the operation unit information acquisition unit 105 acquires the limit position that the operation unit 202 can reach with the HMD 100 as the movement limit of the operation unit 202. The limit position is acquired based on the height of the HMD 100 when the user 201 is standing. Specifically, the user 201 inputs his / her own height to the HMD 100 in advance. Then, the operation unit information acquisition unit 105 estimates the arm length R (see FIG. 4A) of the user 201 based on the height of the user 201, and sets the arm length R with the HMD 100 as the limit position that the operation unit 202 can reach. Note that instead of the height of the user 201, the arm length R of the user 201 can also be directly input. In this way, the operation unit information acquisition unit 105 can acquire the limit position that the operation unit 202 can reach based on the user's height or arm length R. Note that, although the operation unit information acquisition unit 105 acquires the movement limit of the operation unit 202 in this embodiment, the present invention is not limited to this. For example, a part that acquires the movement limit of the operation unit 202 may be provided separately from the operation unit information acquisition unit 105.

[0016] The control unit 106 is, for example, a computer having a CPU or the like. The control unit 106 controls each hardware (device) connected to the bus 114. The control unit 106 can also set a predetermined area in the real space that includes the HMD 100 in a worn state and the operation unit 202 (setting process). The "predetermined area" is an area in the real space where the user 201 can move smoothly, for example, a safety area 203 (see FIG. 2) in which the user 201 is prevented from contacting anything other than the user 201 in the real space. The safety area 203 can be set, for example, by the user 201 specifying a predetermined position on the image of the real space displayed on the display unit 112 and the display unit 113 using the operation unit 202, and the control unit 106 accepting the specification. Thus, in this embodiment, the control unit 106 has a function as a setting means for setting the safety area 203, but is not limited thereto. For example, a part having a function as a setting means may be provided separately from the control unit 106. The control unit 106 also functions as a determination unit that performs two determinations (determination steps) described later. The first determination is whether or not the HMD 100 is approaching the boundary 203a (see FIG. 2) of the safety area 203. The second determination is whether or not the operation unit 202 is approaching the boundary 203a of the safety area 203. In this way, in this embodiment, the control unit 106 has a function as a determination unit, but is not limited thereto. For example, a part that has a function as a determination unit may be provided separately from the control unit 106. In addition, as described later, the HMD 100 issues a notification according to the determination result in the control unit 106. In this embodiment, when it is determined that at least one of the HMD 100 and the operation unit 202 is approaching the boundary 203a of the safety area 203, the notification is issued as a first notification (first warning notification). Moreover, when it is determined that the operation unit 202 is approaching the boundary 203a of the safety area 203, a second notification (second warning notification) is issued to that effect. In the HMD 100, this notification is carried out by the display unit 112 (notification means).

[0017] The ROM 107 stores a control program for the control unit 106, an operating system (OS), and various other programs necessary for implementing functions executed by various devices. These programs include, for example, programs for causing the control unit 106 to execute each part and each means (control method of the display device) of the HMD 100. The RAM 108 functions as the main memory, work area, etc. of the control unit 106. The control unit 108 can implement various operations by loading programs, etc. necessary for executing processing into the RAM 108 and executing the programs.

[0018] The communication unit 109 executes a communication control process between the external device and the communication control unit 109 via the network. The communication control process is not particularly limited, and may be, for example, Internet communication using TCP / IP. The image projection unit 110 projects a virtual reality image or a mixed reality image for the left eye generated by the image processing unit 103 onto the display unit 112. The image projection unit 111 projects a virtual reality image or a mixed reality image for the right eye generated by the image processing unit 103 onto the display unit 113. The display unit 112 is positioned in front of the left eye of the user 201 while the device is being worn, and displays the virtual reality image or the mixed reality image for the left eye projected by the image projection unit 110. The display unit 113 is positioned in front of the right eye of the user 201 while the device is being worn, and displays the virtual reality image or the mixed reality image for the right eye projected by the image projection unit 111.

[0019] FIG. 2 is a diagram showing a state in which a user is using an HMD in a room, as viewed from the horizontal direction. As shown in FIG. 2, a user 201 is standing in a room 300 wearing an HMD 100. Furniture such as desks 301 and 302 are arranged in the room 300. A safety area 203 is an area set by the control unit 106 when the user 201 uses the HMD 100. The safety area 203 avoids real space, that is, obstacles such as the desks 301 and 302 present in the room 300 and the movable range of a door 303. This allows the user 201 to use the HMD 100 safely as long as he is inside the safety area 203, even in a situation in which the field of vision is covered by the HMD 100 and the real space is not visible.

[0020] Fig. 3 is a diagram showing a user using an HMD in a safety area viewed from above. Distances D1 and D2 are shown in Fig. 3. Distance D1 is the shortest distance (horizontal distance) from the HMD 100 to the boundary 203a of the safety area 203. Distance D2 is the shortest distance (horizontal distance) from the operation unit 202 to the boundary 203a of the safety area 203.

[0021] 4A and 4B are diagrams showing a user using an HMD in a safety area from above in a state where the position and orientation information of the operation unit is not acquired. An operation range 401 in FIG. 4A is a range that the operation unit 202 can reach. This operation range 401 is a circular area with the HMD 100 as the center and the arm length R of the user 201 estimated by the operation unit information acquisition unit 105 as the radius. Note that a boundary 401a of the operation range 401 is the movement limit of the operation unit 202. An operation unit position coordinate 402 is the position of the operation unit 202 immediately before the position and orientation information of the operation unit 202 is not acquired by the operation unit information acquisition unit 105. A position coordinate P is the position of the boundary 203a of the safety area 203 closest to the HMD 100. A distance Dp is the distance between the HMD 100 and the position coordinate P. An area 403 is an area included in the operation range 401, that is, an area within the operation range 401. This area 403 is an area having a radius of distance Dp with its center at position coordinate P. In Fig. 4A, area 403 is hatched with an inclination from the upper left to the lower right.

[0022] FIG. 5 is a diagram showing an example of an image visually recognized by a user on an HMD. A virtual reality image 500A shown in FIG. 5(a) is an example of a virtual reality image displayed on the display unit 112 when neither the first nor the second notification is made. The virtual reality image 500A includes a virtual reality object 501. The virtual reality object 501 is not particularly limited, and is, for example, a CG displayed in a content (application) used by the user 201 in the HMD 100. A virtual reality image 500B shown in FIG. 5(b) is an example of a virtual reality image displayed on the display unit 112 when the first notification is made. The virtual reality image 500B includes a virtual reality object 501 and a first notification image (warning notification image) 502. The first notification image 502 is a virtual reality image (first image) displayed as the first notification, and in this embodiment, is a guardian having a shape along the boundary 203a of the safety area 203. The first notification image 502 can notify that at least one of the user 201 (HMD 100) and the operation unit 202 is approaching the boundary 203a of the safety area 203.

[0023] The virtual reality image 500C shown in FIG. 5(c) is an example of a virtual reality image displayed on the display unit 112 when the second notification is being performed. The virtual reality image 500C includes a virtual reality object 501 and a second notification image (warning notification image) 503. The second notification image 503 is a virtual reality image (second image) displayed as the second notification, and in this embodiment, is a guardian having a shape along the boundary 203a of the safety area 203. This second notification image 503 can notify that it is estimated that the operation unit 202 is approaching the boundary 203a of the safety area 203. Note that the first notification image 502 is a guardian with a solid line, and the second notification image 503 is a guardian with a broken line, unlike the first notification image 502. This allows the user 201 to quickly understand which notification image, the first notification image 502 or the second notification image 503, is being displayed. In addition, the first notification image 502 and the second notification image 503 have different shapes in this embodiment, but are not limited thereto. For example, the first notification image 502 and the second notification image 503 may have different colors, or may have different shapes and colors. In addition, a virtual reality image having a parallax according to the distance between the left and right eyes of the user 201 is displayed on the display unit 113 with respect to the virtual reality image displayed on the display unit 112. In addition, in this embodiment, the display unit 112 of the HMD 100 is used as a notification means for performing the first notification and the second notification, but is not limited thereto. For example, the first notification and the second notification may be performed by a speaker built into the HMD 100 or a sound from an external speaker. In addition, when a controller held by the user 201 is used as the operation unit 202, the first notification and the second notification may be performed by vibration from a vibrator built into the controller.

[0024] 6 is a flowchart showing a virtual reality image display process executed by the HMD. As shown in FIG 6, in step S601, the control unit 106 controls the position and orientation estimation unit 104 to estimate the position and orientation of the HMD 100, and obtains the estimation result as position and orientation information of the HMD 100.

[0025] In step S602, the control unit 106 controls the operation unit information acquisition unit 105 to acquire position and orientation information (operation unit information) of the operation unit 202. In addition, the operation unit information acquisition unit 105 calculates a motion vector of the operation unit 202 using the past position and orientation information of the operation unit 202, and acquires the calculation result. Here, the past position and orientation information of the operation unit 202 refers to the position and orientation information of the operation unit 202 a predetermined time before (for example, 0.1 to 1 second before) the timing when the position and orientation information of the operation unit 202 is acquired in step S602. Note that in step S602, if the operation unit 202 is not extracted from the captured image of the operation unit 202, that is, if the joint points of the hand or the wrist of the user 201 are not detected, the position and orientation information and the motion vector of the operation unit 202 are not acquired.

[0026] In step S603, control unit 106 controls image processing unit 103 to generate virtual reality image 500A (see FIG. 5(a)) as a virtual reality image viewed from a viewpoint corresponding to the movement of HMD 100 acquired in step S601. A virtual reality object 501 is placed in virtual reality image 500A. In step S602, position and orientation information of operation unit 202 is acquired, and if operation unit 202 is present in the images captured by image input unit 101 and image input unit 102, a CG representing the hand of operation unit 202 is also included in the virtual reality image as virtual reality object 501.

[0027] In step S604, the control unit 106 judges whether the HMD 100 is approaching the boundary 203a based on the position and orientation information of the HMD 100 acquired in step S601. Specifically, the control unit 106 judges that the HMD 100 is approaching the boundary 203a when the distance D1 (see FIG. 3) is smaller than the first threshold, and judges that the HMD 100 is not approaching the boundary 203a when the distance D1 is equal to or greater than the first threshold. As described above, the distance D1 is the shortest distance from the HMD 100 to the boundary 203a of the safety area 203. The first threshold is preferably set in advance, and may be changeable as appropriate. Then, if it is judged that the HMD 100 is approaching the boundary 203a as a result of the judgment in step S604, the process proceeds to step S607. On the other hand, if it is judged that the HMD 100 is not approaching the boundary 203a as a result of the judgment in step S604, the process proceeds to step S605.

[0028] In step S605, the control unit 106 controls the operation unit information acquisition unit 105 to determine whether or not the position coordinates of the operation unit 202 have been acquired in step S602. If it is determined in step S605 that the position coordinates of the operation unit 202 have been acquired, the process proceeds to step S606. On the other hand, if it is determined in step S605 that the position coordinates of the operation unit 202 have not been acquired, the process proceeds to step S609.

[0029] In step S606, the control unit 106 judges whether the operation unit 202 is approaching the boundary 203a of the safety area 203 based on the position and orientation information of the operation unit 202 acquired in step S602. Specifically, the control unit 106 judges that the operation unit 202 is approaching the boundary 203a when the distance D2 (see FIG. 3) is smaller than the second threshold, and judges that the operation unit 202 is not approaching the boundary 203a when the distance D2) is equal to or greater than the second threshold. As described above, the distance D2 is the shortest distance from the position of the operation unit 202 to the boundary 203a. The second threshold is preferably set in advance, and may be appropriately changeable. The second threshold may be the same size as the first threshold, or may be different. Then, as a result of the judgment in step S606, if it is judged that the operation unit 202 is approaching the boundary 203a of the safety area 203, the process proceeds to step S607. On the other hand, if it is determined in step S606 that the operation unit 202 is not approaching the boundary 203a of the safety area 203, the process proceeds to step S613.

[0030] In step S607 after step S604 or step S606, the control unit 106 controls the image processing unit 103 to generate a first notification image 502 (see FIG. 5B).

[0031] In step S608, control unit 106 controls image processing unit 103 to superimpose first alert image 502 generated in step S607 on virtual reality image 500A generated in step S603 to generate virtual reality image 500B (see FIG. 5(b)). After executing step S608, the process proceeds to step S613.

[0032] In step S609 after executing step S605, the control unit 106 controls the operation unit information acquisition unit 105 to acquire the operation range 401 of the operation unit 202 (see FIG. 4A).

[0033] In step S610, the control unit 106 determines whether or not the operation range 401 (operation unit 202) of the operation unit 202 is approaching the boundary 203a of the safety area 203. This determination is made based on the position and orientation information of the HMD 100 acquired in step S601 and the boundary 401a of the operation range 401 of the operation unit 202 acquired in step S609, i.e., the movement limit of the operation unit 202.

[0034] The determination in step S610 will be described with reference to FIG. 4B. The area 404 in FIG. 4B is an area that is inward from the boundary 203a of the safety area 203 by a distance Dq. In FIG. 4B, the area 404 is hatched with an inclination from the upper right to the lower left. The operation range 401 overlaps this area 404. The vector α is a vector that points from the HMD 100 toward the position coordinate P. The vector β is a motion vector that indicates the position of the operation unit 202 immediately before the position and orientation information of the operation unit 202 is not acquired, that is, the direction of the movement of the operation unit 202 at the operation unit position coordinate 402. When the angle θ between the vector α and the vector β is an acute angle (preferably 45 degrees or less), it is assumed that the vector β is in the same direction as the vector α. In FIG. 4B, the vector β1 corresponds to the vector β when the angle θ is an acute angle. On the other hand, when the angle θ between the vector α and the vector β is an obtuse angle, it is assumed that the vector β is not in the same direction as the vector α. In Fig. 4B, vector β2 corresponds to vector β when the angle θ is an obtuse angle. Then, in step S610, if the operation unit position coordinate 402 is included inside the boundary 401a of the operation range 401 and the direction of vector β is toward the position coordinate P, it is determined that the operation range 401 of the operation unit 202 is approaching the boundary 203a. Then, if it is determined in step S610 that the operation range 401 of the operation unit 202 is approaching the boundary 203a, the process proceeds to step S611. On the other hand, if it is determined in step S610 that the operation range 401 of the operation unit 202 is not approaching the boundary 203a, the process proceeds to step S613.

[0035] In this way, in step S610, when the position and orientation information of the HMD 100 is acquired in step S601 and the position and orientation information of the operation unit 202 is not acquired in step S602, it is determined whether the operation range 401 is approaching the boundary 203a. As described above, in this embodiment, the determination in step S610 is made based on the position and orientation information of the HMD 100 acquired in step S601 and the boundary 401a of the operation range 401 of the operation unit 202 acquired in step S609, but is not limited to this. For example, instead of the position and orientation information of the HMD 100, position and orientation information regarding the position and orientation of the user 201 himself may be used. In this case, it is preferable to acquire the position and orientation information of the user 201 based on an image of the user 201 captured by an external camera. Also, in this embodiment, the operation unit position coordinates 402 are used for the determination in step S610, but is not limited to this. For example, it is also possible to make the determination in step S610 based only on the position and orientation information of the HMD 100 and information regarding the operation range 401 of the operation unit 202.

[0036] In step S611, the control unit 106 controls the image processing unit 103 to generate the second notification image 503. After step S611 is executed, the process proceeds to step S612.

[0037] In step S612, control unit 106 controls image processing unit 103 to superimpose second alert image 503 generated in step S611 on virtual reality image 500A generated in step S603 to generate virtual reality image 500C (see FIG. 5(c)). After executing step S612, the process proceeds to step S613.

[0038] In step S613, control unit 106 controls image projection unit 110 and image projection unit 111 to project and display the virtual reality image generated in any one of step S603, step S608, or step S612 onto display unit 112 and display unit 113.

[0039] When the user 201 moves in real space while wearing the HMD 100 on his / her head, the hand of the user 201, which has a wider range of movement than the head on which the HMD 100 is worn, may approach the boundary 203a of the safety area 203. Even in this case, the HMD 100 can perform the first or second notification as described above. This allows the user 201 to confirm the first or second notification and ensure his / her safety from the surroundings. Also, as shown in FIG. 5(a), when the user 201 is not approaching the boundary 203a of the safety area 203, the display of the first notification image 502 and the second notification image 503 can be stopped to enhance the immersion of the user 201 in the virtual space. As shown in FIG. 5(b), by displaying the first notification image 502 superimposed on the virtual reality object 501, the immersion of the user 201 in the virtual space is reduced, but the boundary 203a of the safety area 203 can be recognized by the user 201. 5(c), by displaying the second alert image 503 superimposed on the virtual reality object 501, the sense of immersion of the user 201 in the virtual space is reduced, but the user 201 can be made aware of the boundary 203a of the safety area 203. The second alert image 503 makes it possible for the user 201 to be aware of the boundary 203a of the safety area 203 while changing the degree of warning compared to the first alert image 502.

[0040] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0041] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A display device that is removably worn by a user and displays an image of a virtual space while worn by the user, a first acquisition means capable of acquiring position information relating to a position in the real space of an operation unit that is movable relative to the display device in the mounted state and that operates the display device; A second acquisition means for acquiring a movement limit of the operation unit relative to the display device; a setting means for setting a predetermined area in the real space, the predetermined area including the display device and the operation unit; a determination means for determining whether or not the operation unit is approaching a boundary of the specified area based on the position information when position information of the operation unit is acquired by the first acquisition means, and for determining whether or not the operation unit is approaching the boundary based on the movement limit when position information of the operation unit is not acquired by the first acquisition means. (Configuration 2) The first acquisition means is capable of acquiring position information regarding a position of the display device in the real space, The display device described in configuration 1, characterized in that when the first acquisition means acquires position information of the display device and the first acquisition means does not acquire position information of the operation unit, the judgment means determines whether the operation unit is approaching the boundary based on the position information of the display device and the movement limit. (Configuration 3) The display device described in Configuration 2, characterized in that when the first acquisition means acquires position information of the display device, the judgment means determines whether the display device is approaching the boundary based on the position information. (Configuration 4) The determination means is determining that the display device is approaching the boundary when the shortest distance from the position of the display device to the boundary is smaller than a threshold; determining that the operation unit is approaching the boundary when the shortest distance from the position of the operation unit to the boundary is smaller than a threshold; The display device according to configuration 3, characterized in that it is determined that the operation unit is approaching the boundary when the position of the operation unit immediately before the position information of the operation unit was not acquired is inside the movement limit and the direction of the motion vector of the operation unit immediately before the position information of the operation unit was not acquired is toward the position of the boundary of the specified area that is closest to the display device. (Configuration 5) The display device according to any one of configurations 2 to 4, wherein the first acquisition means acquires the position information of the display device using SLAM (Simultaneous Localization And Mapping). (Configuration 6) The display device described in any one of configurations 1 to 5, wherein the first acquisition means extracts the operation part from a captured image of the operation part, and acquires position information of the operation part based on the extraction result. (Configuration 7) The display device according to any one of configurations 1 to 6, wherein the second acquisition means acquires, as the movement limit, a limit position that the operating unit can reach with the display device as the center. (Configuration 8) The display device according to configuration 7, wherein the second acquisition means acquires the limit position based on a height or an arm length of the user. (Configuration 9) The display device according to configuration 7, wherein the second acquisition means acquires the limit position based on a height of the display device when the user is standing. (Configuration 10) The display device described in Configuration 3 is characterized in that, if, as a result of the judgment by the judgment means, the first acquisition means acquires position information of the operation unit and it is judged that at least one of the display device and the operation unit is approaching a boundary, the display device is provided with an alarm means for notifying that fact as a first alarm, and if the first acquisition means does not acquire position information of the operation unit and it is judged that the operation unit is approaching the boundary, the display device is provided with an alarm means for notifying that fact as a second alarm. (Configuration 11) The display device described in Configuration 10, characterized in that the notification means displays a first image placed in the virtual space as the first notification, and displays a second image placed in the virtual space and different from the first image as the second notification. (Configuration 12) The display device according to configuration 11, wherein the second image is different from the first image in at least one of shape and color. (Configuration 13) The display device according to any one of configurations 1 to 12, wherein the setting means sets, as the predetermined area, an area in which the user can move smoothly within the real space. (Configuration 14) The display device according to configuration 13, wherein the setting means sets, as the specified area, an area in the real space in which contact between the user and anything other than the user is prevented. (Configuration 15) The display device according to any one of configurations 1 to 14, wherein the operation unit is the user's hand or a controller held in the hand. (Method 1) A method for controlling a display device that is removably worn by a user and displays an image of a virtual space in the worn state, comprising: a first acquisition step for acquiring position information relating to a position in the real space of an operation unit that is movable relative to the display device in the mounted state and that operates the display device; A second acquisition step of acquiring a movement limit of the operation unit relative to the display device; a setting step of setting a predetermined area in the real space, the predetermined area including the display device and the operation unit; a determination step of, when position information of the operation unit is acquired in the first acquisition step, determining whether or not the operation unit is approaching a boundary of the specified area based on the position information, and, when position information of the operation unit is not acquired in the first acquisition step, determining whether or not the operation unit is approaching the boundary based on the movement limit. (Configuration 17) A program for causing a computer to execute each means of the display device according to any one of configurations 1 to 15. [Explanation of symbols]

[0042] 100 HMD 104 Position and orientation estimation unit 105 Operation unit information acquisition unit 106 Control section 202 Operation section 203 Safety area 203a Boundary 401a boundary 502 First notification image (Warning notification image) 503 Second notification image (Warning notification image)

Claims

1. A display device that is removably attached to a user and displays an image of a virtual space in the attached state, a first acquisition means capable of acquiring position information relating to a position in the real space of an operation unit that operates the display device and that is movable relative to the display device in the real space in which the display device is used in the mounted state; A second acquisition means for acquiring a movement limit of the operation unit relative to the display device; a setting means for setting a predetermined area in the real space, the predetermined area including the display device and the operation unit; a judgment means for, when position information of the operation unit is acquired by the first acquisition means, judging whether or not the operation unit is approaching a boundary of the specified area based on the position information, and for, when position information of the operation unit is not acquired by the first acquisition means, judging whether or not the operation unit is approaching the boundary based on the movement limit.

2. the first acquisition means is capable of acquiring position information relating to a position of the display device in the real space; The display device according to claim 1, characterized in that, when the first acquisition means acquires position information of the display device and the first acquisition means does not acquire position information of the operation unit, the judgment means determines whether the operation unit is approaching the boundary based on the position information of the display device and the movement limit.

3. The display device according to claim 2, characterized in that, when position information of the display device is acquired by the first acquisition means, the determination means determines whether the display device is approaching the boundary based on the position information.

4. The determination means is determining that the display device is approaching the boundary when the shortest distance from the position of the display device to the boundary is smaller than a threshold; determining that the operation unit is approaching the boundary when the shortest distance from the position of the operation unit to the boundary is smaller than a threshold; The display device according to claim 3, characterized in that if the position of the operation unit immediately before the position information of the operation unit was not obtained is inside the movement limit, and the direction of the movement vector of the operation unit immediately before the position information of the operation unit was not obtained is toward the position of the boundary of the specified area closest to the display device, it is determined that the operation unit is approaching the boundary.

5. The display device according to claim 2 , wherein the first acquisition means acquires the position information of the display device using Simultaneous Localization and Mapping (SLAM).

6. 2 . The display device according to claim 1 , wherein the first acquisition means extracts the operation part from a captured image of the operation part, and acquires position information of the operation part based on a result of the extraction.

7. The display device according to claim 1 , wherein the second acquisition means acquires, as the movement limit, a limit position that the operation unit can reach with the display device as a center.

8. The display device according to claim 7 , wherein the second acquisition means acquires the limit position based on a height or an arm length of the user.

9. The display device according to claim 7 , wherein the second acquisition means acquires the limit position based on a height of the display device when the user is standing.

10. The display device according to claim 3, further comprising an alarm means for notifying the user of the fact as a first alarm if, as a result of the judgment by the judgment means, the first acquisition means acquires position information of the operation unit and it is judged that at least one of the display device and the operation unit is approaching a boundary, and for notifying the user of the fact as a second alarm if the first acquisition means does not acquire position information of the operation unit and it is judged that the operation unit is approaching the boundary.

11. The display device according to claim 10, characterized in that the notification means displays a first image arranged in the virtual space as the first notification, and displays a second image arranged in the virtual space and different from the first image as the second notification.

12. The display device according to claim 11 , wherein the second image is different from the first image in at least one of shape and color.

13. The display device according to claim 1 , wherein the setting means sets, as the predetermined area, an area in which the user can move smoothly within the real space.

14. The display device according to claim 13 , wherein the setting means sets, as the predetermined area, an area in the real space where contact between the user and anything other than the user is prevented.

15. 2. The display device according to claim 1, wherein the operation unit is a hand of the user or a controller held by the hand.

16. A method for controlling a display device that is detachably attached to a user and displays an image of a virtual space in the attached state, comprising: a first acquisition step for acquiring position information relating to a position in the real space of an operation unit that is movable relative to the display device in the mounted state and that performs an operation on the display device; A second acquisition step of acquiring a movement limit of the operation unit relative to the display device; a setting step of setting a predetermined area in the real space, the predetermined area including the display device and the operation unit; a determination step of, when position information of the operation unit is acquired in the first acquisition step, determining whether or not the operation unit is approaching a boundary of the specified area based on the position information, and, when position information of the operation unit is not acquired in the first acquisition step, determining whether or not the operation unit is approaching the boundary based on the movement limit.

17. A program for causing a computer to execute each of the means of the display device according to claim 1.

Images