Head-mounted display device

Abstract

The invention provides a head-mounted display device which can prevent users from colliding with each other and colliding with non-users and can provide a convenient operation space. A control unit of a head-mounted display device sets a first operation space in which a user performs a gesture operation, and controls an image projection unit to display, as a mixed reality object, an image of the first operation space and an image of an operation object in which the user performs the gesture operation via a display unit. The position of the external device and information on a second operation space in which a user of the external device performs a gesture operation are acquired through the communication unit, and whether the first operation space and the second operation space overlap is determined based on the position information of the head-mounted display device, the position of the external device, and the information on the second operation space. In the case of overlap, the control unit controls the communication unit to connect to the smartphone, and switches from the gesture operation to the operation of the smartphone.

Description

Technical Field

[0001] The present invention relates to a head-mounted display device that displays mixed reality images and can be operated by user's gestures. Background Art

[0002] In recent years, head mounted display devices (hereinafter referred to as HMD) have been developed, which are in the shape of glasses or goggles, and can make it appear as if the mixed reality objects actually exist by projecting mixed reality images on a portion corresponding to a lens with high transparency, and overlapping and displaying mixed reality objects in real space. Patent document 1 describes a system for displaying shared virtual objects and individual virtual objects in a mixed reality environment.

[0003] In addition, in order to perform operations such as starting to view and listen to content and selecting content using the HMD, there is a gesture user interface (gesture UI) that is operated by the user's hand movements. The gesture UI can operate the HMD without holding the operating device with the hand, and is highly convenient. However, since the gesture UI is accompanied by body movement, when other users operating the HMD are very close, there is a concern that the users' bodies may collide with each other when performing gesture operations. In order to avoid such collisions, Patent Document 2 describes that the HMD has a device information notification unit (marker) for determining the position of the device, and a camera is used to photograph it to detect the distance and direction of the mark, and the operating space of the HMD is adjusted so that the gesture operation spaces of the HMDs do not overlap.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application No. 2016-525741

[0007] Patent Document 2: Japanese Patent Application Publication No. 2012-108577 Summary of the invention

[0008] Technical issues to be solved by the invention

[0009] Patent document 1 does not consider avoiding collisions with others due to gesture operations. Patent document 2 does not consider avoiding collisions with people who are not using an HMD. When the method described in patent document 2 is adopted, if the HMD of another user is facing in the opposite direction or there are obstructions in the way, there is a situation where the camera cannot detect the mark set on the HMD. In this case, the distance and direction of the HMD of the other user cannot be known, and there is a possibility of collision. In addition, the adjustment of the gesture operation space is only performed on this device, so sometimes in order to make the operation space of each other farther away, the operation space for the user is adjusted to be too narrow, resulting in poor usability.

[0010] An object of the present invention is to provide an HMD that can avoid body collisions caused by gesture operations not only between users of the HMD but also between users of the HMD and people who do not use the HMD, and that has an operation space that is convenient for users to use.

[0011] Technical solutions to solve problems

[0012] The head-mounted display device of the present invention can display mixed reality images and can be operated by gestures by the user, including: an image projection unit for projecting images; a display unit for overlapping the image projected from the image projection unit with the surrounding space to display as a mixed reality object; a receiving unit for receiving position information of the head-mounted display device; a communication unit capable of communicating with an external device; and a control unit, wherein the control unit sets a first operation space for the user of the head-mounted display device to perform gesture operations, controls the image projection unit to display the mixed reality object via the display unit, wherein the mixed reality object is an image of the first operation space and an image of an operation object displayed in the first operation space for the user of the head-mounted display device to perform gesture operations, obtains the position of the external device and information about the second operation space for the user of the external device to perform gesture operations through the communication unit, determines whether the first operation space overlaps with the second operation space based on the position information of the head-mounted display device, the position of the external device and the information about the second operation space received by the receiving unit, and controls the communication unit to connect to a smartphone when the first operation space overlaps with the second operation space, switching from gesture operation to smartphone operation.

[0013] In addition, the head-mounted display device of the present invention that can display mixed reality images and can be operated by gestures by the user includes: an image projection unit that projects images; a display unit that overlaps the image projected from the image projection unit with the surrounding space to display it as a mixed reality object; a receiving unit that receives position information of the head-mounted display device; a communication unit that can communicate with an external device; and a control unit, the control unit setting a first operation space for the user of the head-mounted display device to perform gesture operations, and controlling the image projection unit to display the mixed reality object via the display unit, wherein the mixed reality object is an image in the first operation space and a gesture operation for the user of the head-mounted display device displayed in the first operation space. The image of the operating object for gesture operation is obtained by the communication unit, and the position of the external device and the information about the second operating space in which the user of the external device performs the gesture operation are obtained. Based on the position information of the head-mounted display device, the position of the external device and the information about the second operating space received by the receiving unit, whether the first operating space overlaps with the second operating space is determined. When the first operating space overlaps with the second operating space, a third operating space that is reduced in size compared to the first operating space is calculated so as not to overlap with the second operating space. When the third operating space is displayed as a mixed reality object via the display unit, the image projection unit is controlled to not display a portion of the image of the operating object.

[0014] Effects of the Invention

[0015] According to the present invention, not only collision between users of the HMD but also body collision between the user of the HMD and a person not using the HMD due to gesture operation can be avoided, and an HMD having an operation space convenient for the user can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a figure which shows the external appearance structure of HMD (Example 1).

[0017] Figure 2 It is a diagram showing the system configuration of an HMD.

[0018] Figure 3 It is a diagram for explaining the position information acquisition function of the HMD.

[0019] Figure 4 This is a diagram showing a state where a mixed reality object is displayed by the HMD.

[0020] Figure 5 The diagrams explain various methods of operating the HMD by the user.

[0021] Figure 6 FIG. 1 is a diagram showing an example of an operation object displayed to an HMD user.

[0022] Figure 7 This is a diagram showing a state where a user performs gesture operation to view and listen to content.

[0023] Figure 8 This is a diagram showing a state where two HMD users use the HMD side by side.

[0024] Fig. 9 This is a diagram showing how the gesture operation space is moved to avoid a collision.

[0025] Fig.10 This is a diagram showing a situation where the gesture operation space is reduced in order to avoid collision.

[0026] Fig.11 This is a diagram showing a situation where a gesture operation is switched to a smartphone operation.

[0027] Fig.12 This is a flowchart showing a method for adjusting the operation space when two HMD users are close to each other.

[0028] Fig.13 This is a diagram showing a state where one HMD user uses the HMD (Example 2).

[0029] Fig.14 This is a diagram showing a situation where the gesture operation space is reduced in order to avoid collision.

[0030] Fig.15 This diagram shows a case where a part of an object is not displayed in order to avoid a collision.

[0031] Fig.16 This is a flowchart for a case where the display position of an object is changed by reducing the operation space. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present invention will be described using the drawings.

[0033] Example 1

[0034] In the first embodiment, a case is described in which a collision between the bodies of a plurality of HMD users due to gesture operation is avoided when the users are close to each other.

[0035] Figure 11 is a diagram showing the appearance structure of the HMD (head mounted display device) of this embodiment. The HMD 1 includes: a system storage unit 101a, 101b storing a control unit, a memory, a storage, etc.; a camera 102a, 102b; a position information radio wave receiving unit (position information acquisition unit) 103a, 103b; an image projection unit 104a, 104b that projects an image of a mixed reality object; perspective lenses 105a, 105b that overlap the projected mixed reality object with the surrounding space and display it; and a distance image acquisition unit 106 that acquires a distance image.

[0036] Here, a binocular HMD is shown in which left and right images are displayed on left and right perspective lenses 105a by two image projection units 104a and 104b, but a monocular HMD that displays a single image may also be used.

[0037] The position information radio wave receiving units 103a and 103b receive radio wave signals from beacons described later and obtain the position of the HMD 1. The distance image acquisition unit 106 detects the user's hand movements (gesture operations). In addition, people and objects existing in front of the HMD 1 are detected, and their distances and directions are measured. As a specific example of the distance image acquisition unit 106, a TOF (Time Of Flight) camera that measures distance based on the flight time of light can be used.

[0038] In addition, the HMD 1 and the smartphone 2 having the inter-terminal communication function defined by the fifth generation mobile communication standard are connected by a communication cable 20. In addition, wireless communication such as WiFi and Bluetooth may be used instead of the communication cable 20. Furthermore, if the system storage units 101a and 101b have built-in communication functions defined by the fifth generation mobile communication standard, communication with the external smartphone 2 is not required.

[0039] Figure 2 Yes means Figure 1 The system structure of the HMD shown in FIG. The system storage unit 101a, 101b of the HMD 1 has: a memory 201 constituting a function unit of various programs being executed; a storage 202 storing various data, programs, contents, etc.; a control unit 205 for executing programs; and a smartphone communication unit 206 for communicating with the smartphone 2, which are connected by a bus 203. Furthermore, as an input unit 204, there is an input unit based on a touch sensor for touching an input button with a fingertip, a touch of the HMD frame, vibration of the HMD frame, etc.

[0040] The control unit 205 receives the user's operation via the distance image acquisition unit 106, the input unit 204, and the smartphone 2, and controls the display operation of the image (content image) performed by the image projection units 104a and 104b. In addition, the control unit 205 detects the approach of another person (other HMD) by using the distance image acquisition unit 106 and the smartphone 2, and adjusts the operation space for gesture operation to avoid collision between the users' bodies.

[0041] Figure 3 1 is a diagram for explaining the location information acquisition function of the HMD. The HMD 1 can acquire the location information of the HMD 1 using the beacons 5a and 5b for acquiring location information installed indoors. The beacons 5a and 5b for acquiring location information, for example, use Bluetooth (registered trademark) Low Energy wireless technology to send identification codes of each beacon, such as iBeacon (registered trademark). In addition, beacon devices using Bluetooth (registered trademark) are described in Japanese Patent Application Publication No. 2010-130661, etc.

[0042] The position information radio wave receiving units (position information acquisition means) 103a and 103b of HMD1 receive radio waves from beacons 5a and 5b and measure the intensity of the radio waves, thereby being able to know the distance from the two beacons 5a and 5b. At this time, HMD1 receives radio waves from the two position information radio wave receiving units 103a and 103b, and based on the difference in intensity, it is possible to know in which direction HMD1 is tilted from the beacons 5a and 5b. That is, the position information of HMD1 based on the two beacons 5a and 5b and the information on the direction of the user 3 using HMD1 can be acquired.

[0043] Furthermore, in the same room, when another user 3' is using another HMD 1', the user 3' can obtain the position information of the other HMD 1' and the direction information of the user 3' using the HMD 1' based on the beacons 5a and 5b. The user 3 and the user 3' can exchange the obtained HMD position information and direction information respectively using the short-range inter-terminal communication (smartphones 2 and 2') of the fifth generation mobile communication.

[0044] Figure 4 1 is a diagram showing a state where a mixed reality object is displayed by the HMD, and is a state where the HMD 1 is viewed from the left side. The mixed reality object 300 projected from the image projection unit 104b is displayed on the perspective lens 105b. When viewed from the eyes 31 of the HMD user, it looks like the mixed reality object 300 is floating in the space in front of the HMD user in the actual surrounding background. The mixed reality object 300 can be a representative image content, as well as various contents such as an object for gesture operation, an operation space object, etc. Hereinafter, it is referred to as an "object".

[0045] Figure 5 The diagrams explain various methods of operating the HMD by the user.

[0046] (1) A spherical object 301 is displayed in front of the eye 31 of the HMD user. The user performs a gesture to move the hand 32 by touching the object 301 or waving the hand horizontally. The distance image acquisition unit 106 detects that the user's hand 32 is located at the position of the object 301 and the gesture is performed on the object 301. Thus, the user can perform operations such as selecting the content displayed on the HMD 1, and starting / stopping viewing.

[0047] (2) An object 302 in the shape of a smartphone is displayed in front of the eye 31 of the HMD user. The user performs a gesture of moving the hand 32 with respect to the object 302 in the same manner as operating an actual smartphone. The distance image acquisition unit 106 detects that the user's hand 32 is located at the position of the object 302 and performs a gesture of operating the smartphone. The user can thereby perform operations such as starting and stopping an application, installing an application, and starting and ending communication in the same manner as operating an actual smartphone.

[0048] (3) As an operation other than gestures, there is a case where the touch panel of the smartphone 2 connected to the HMD 1 is actually operated. In front of the eyes 31 of the HMD user, an object 303 in the shape of a hand is displayed together with an object 302 in the shape of a smartphone. At this time, the object 303 is displayed in a motion manner linked to the actual operation of the smartphone 2 by the user, so the user can operate the HMD without turning his eyes toward the smartphone 2.

[0049] In addition, it is also possible to Figure 2 The HMD is operated by the input button or the touch sensor using the input unit 204. In addition, when another person 4 who does not use the HMD approaches the HMD user, the distance image acquisition unit 106 acquires information about the outer shape of the other person 4 and the distance to the other person 4. Thus, it is possible to recognize that another person is approaching nearby, and it is possible to recognize that, for example, the right hand 42 of the approaching person is extended toward the HMD user.

[0050] Next, the basics of a method for viewing mixed reality content using an HMD will be described.

[0051] Figure 6: is a diagram showing an example of an operation object displayed to the HMD user. An object (operation space object) 600 representing a spherical gesture operation space is displayed around the user 3 who is sitting and wearing the HMD 1. The operation space object 600 indicates the range of space in which the gesture operation performed by the user 3 is effective, so exceeding this range means that the operation is invalid and there is a possibility of collision with other people's bodies. Inside the operation space, objects (operation objects) 601, 602, and 603 for gesture operations for selecting mixed reality content (hereinafter referred to as content) to be viewed, starting viewing, and ending viewing are displayed.

[0052] Figure 7 This is a diagram showing a state where a user performs gesture operation to view and listen to content. For example, the user 3 performs a gesture by touching the operation object 601 with his hand 32 to select the desired content. In this way, the corresponding mixed reality content 701 can be displayed in front of the user 3, and the user can start viewing and listening.

[0053] Next, a description will be given of how to avoid collision between the bodies of a plurality of HMD users due to gesture operations when the users are close to each other.

[0054] Figure 8 This is a diagram showing a state where two HMD users use the HMD side by side. Around the HMD users 3 and 3', operation space objects 600 and 600' representing their respective gesture operation spaces are displayed. In addition, in order to avoid complexity in the drawing, the operation space objects 600 and 600' only represent the outermost circle (dotted line). At this time, each HMD 1 and 1' can display only the gesture operation space of the machine to the user, and can also display the gesture operation space of other users. This can utilize the terminal-to-terminal communication function of the fifth generation mobile communication (smartphones 2 and 2') to exchange information on the position and size of the gesture operation space with other HMDs. As a result, user 3 can not only see his own object 600, but also see the object 600' for user 3'.

[0055] In the operation space, operation objects 601 to 603 and 601 ′ to 603 ′ for performing gesture operations for selecting content, starting viewing, and ending viewing are displayed. In order to avoid complicating the drawing, the display of the content being viewed is omitted.

[0056] There is a case where the user 3, 3' swings the hand 32, 32' laterally at the corresponding object (here 601, 603') to interrupt the viewing of the content. When the two persons are close to each other, there is a concern that their hands 32, 32' may collide.

[0057] Fig. 9This is a diagram showing a situation where the gesture operation space is moved to avoid collision. By moving the gesture operation spaces (objects 600, 600') for two HMD users 3, 3' in a direction away from each other, it is possible to avoid overlapping of the operation spaces. In addition, by moving the display positions of the objects (601 to 603, 601' to 603') for gesture operation, the user is prompted to perform gesture operation using the hand 33, 33' on the side far from other users. In this way, it is possible to avoid collision between the hands of the users 3, 3' who are close to each other.

[0058] Fig.10 This is a diagram showing a situation where the gesture operation space is reduced in order to avoid collision. By reducing the size of the gesture operation space (objects 600, 600') for the two HMD users 3, 3', it is possible to avoid the gesture operation spaces from overlapping each other. In addition, the size of the objects of gesture operation (601 to 603, 601' to 603') is also reduced in a manner that allows them to be within the reduced gesture operation space, so that the display positions are brought closer. In addition, when reducing the size of the gesture operation space, it is not limited to reducing the entire spherical space. The operation space can also be limited to the lower half of the spherical space, or the shape of the operation space can be deformed into a rotating ellipsoid. In this way, it is possible to avoid collision between the hands of the users 3, 3' that are close to each other.

[0059] In the above Fig. 9 , Fig.10 When the operation space is moved or reduced in this way, the information of each other's operation space is exchanged between HMD1 and 1' through the terminal communication function (smartphones 2 and 2'), and it is determined whether the operation space overlaps, and a new operation space is determined to avoid overlap. That is, HMD1 and HMD1' can use the terminal communication function to detect the situation of being close to each other, so no matter which direction the other party's HMD is facing, even if there are obstacles on the way, it will not be missed. In addition, the two HMD1 and 1' determine the operation space in conjunction with each other, so the optimal size can be set without making the adjusted operation space too narrow.

[0060] Fig.11: This is a diagram showing the situation of switching from gesture operation to smartphone operation. In the case where the collision of the hands of users 3 and 3' cannot be avoided even if the gesture operation space is adjusted, HMD 1 and 1' terminate the display of the operation space object 600 and 600' to users 3 and 3', and notify that the gesture operation cannot be accepted. Then, users 3 and 3' switch to the operation performed by the real smartphone 2 and 2'. At this time, in front of users 3 and 3', objects 611 and 611' in the shape of smartphones based on mixed reality and objects 612 and 612' in the shape of hands that operate smartphones are displayed. When the real smartphone 2 and 2' are operated by hands 32 and 32', the display position of the hand-shaped objects 612 and 612' moves in conjunction with the movement of the operating hands 32 and 32'. As a result, users 3 and 3' can operate smartphones without turning their sight toward the real smartphones 2 and 2'. Accordingly, for example, smartphones 2 and 2' can be operated while being kept in pockets to select content, start viewing, and end viewing.

[0061] Fig.12 is a flowchart showing a method for adjusting the operating space when two HMD users are close to each other. Fig. 9 The display position of an object is changed by moving the gesture operation space described in . In addition, this process is repeatedly executed by the control unit 205 while the user 3 uses the HMD 1.

[0062] S801: The HMD 1 acquires the position information of the HMD 1 (including the direction of the HMD 1) by using the position information radio wave receiving units 103a and 103b.

[0063] S802: HMD 1 exchanges HMD position information (including orientation) and gesture operation space information (position and size) with nearby HMD 1' through 5G mobile communication inter-terminal communication via smartphone 2. Thus, the position information and operation space information of nearby HMD 1' are acquired.

[0064] S803: Based on the information of the position and size of the operation space obtained from the nearby HMD 1', determine whether the operation space of the nearby HMD 1' overlaps with the local HMD 1. End if the operation space does not overlap. If the operation space overlaps, proceed to S804.

[0065] S804: Calculate a position where the operation spaces do not overlap by moving the operation space of the own HMD 1 in a direction away from the operation space of the nearby HMD 1 ′.

[0066] S805: The calculated position information of the operation space is exchanged with the nearby HMD 1' through short-distance inter-terminal communication. At this time, the position information of the operation space calculated by the nearby HMD 1' is acquired from the nearby HMD 1'.

[0067] S806: Based on the position information of the calculated operation space obtained from the nearby HMD 1', determine whether it overlaps with the calculated operation space of the local HMD 1. If the operation spaces do not overlap, proceed to S807. If the operation spaces overlap, return to S804 and calculate the position where the operation spaces do not overlap again. Repeat the determination of S806 until the operation spaces do not overlap.

[0068] S807: Determine the position of an operating object for performing gesture operation on the content within the calculated operation space and display the object.

[0069] S808: Displaying an object representing a new gesture operation space according to the position change of the operation space.

[0070] S809: According to the position change of the operation space, a new detection area is set when the distance image acquisition unit 106 detects the gesture operation.

[0071] S810: Since the operation space does not overlap with the nearby HMD 1', the operation space, the display position of the operation object, and the detection area of ​​the gesture operation are returned to the initial values.

[0072] The above flowchart shows the case where the gesture operation space is moved to avoid collision, and the same is true when the gesture operation space is reduced. Alternatively, the two can be combined to perform adjustment.

[0073] Furthermore, in the above-described embodiment, a case where two HMD users are close to each other is assumed, but the same is true when two or more HMD users are close to each other.

[0074] According to the first embodiment, when multiple HMD users approach each other, it is possible to avoid collision of the users' bodies due to gesture operation. In this case, it is possible to reliably detect HMDs existing nearby and provide an HMD having an operation space that is easy for the users to use.

[0075] Example 2

[0076] In the second embodiment, a case where a body collision between a user using an HMD and a person not using an HMD is avoided will be described.

[0077] Fig.13This is a diagram showing a state in which one HMD user uses the HMD. User 3 performs gesture operation to view content. In the operation space object 600, operation objects 601 to 603 corresponding to the first group of content are displayed, and operation objects 604 to 606 for the second group of content are displayed side by side. For example, user 3 performs gesture operation on object 601 with hand 32, and the selected content 701 is displayed and viewed. It is assumed that a person who does not use the HMD approaches in this state.

[0078] Fig.14 This is a diagram showing a situation where the gesture operation space is reduced to avoid collision. There is a situation where another person 4 who does not use an HMD approaches the HMD user 3, enters the gesture operation space (object 600), and approaches the display position of objects 604 to 606. At this time, when the user 3 performs gesture operation on the objects 604 to 606, the body of the user 3 and the other person 4 collide.

[0079] To avoid this situation, HMD 1 recognizes that another person 4 is approaching based on the information from distance image acquisition unit 106. Then, the size of the gesture operation space (object 600) is reduced to a range where there is no collision. In addition, the operation objects 604 to 606 that are no longer in the operation space due to the reduction are moved to other positions in the operation space. In this example, they are moved to the right side of the audiovisual content 701. In this way, collision between the user 3 and another person's body can be avoided.

[0080] In the above example, the case where the gesture operation space is reduced and the operation object is moved in order to avoid the collision is described, but this situation can also be dealt with by moving the gesture operation space and reducing the operation object.

[0081] When the operation space is reduced or moved as described above, the distance image acquisition unit 106 measures the distance and direction from the other person 4 and determines a new operation space so that the position of the other person 4 does not overlap with the operation space. In addition, collision avoidance is not only for people, but also for objects.

[0082] Fig.15 is a diagram showing a situation where a part of an object is not displayed in order to avoid collision. Fig.14 As a result of reducing the gesture operation space (object 600) as shown, an object is generated that is no longer in the gesture operation space. At this time, a part of the objects (here 604 to 606) are not displayed. Then, the operation corresponding to the non-displayed object is switched to other operations such as the smartphone 2. At this time, the user 3 is notified that the object of the gesture operation is restricted. In this way, the user 3 can avoid collision with other people's bodies.

[0083] Fig.16: is a flowchart showing a method for adjusting the operation space when a person who does not use the HMD approaches the HMD user. Fig.14 The case where the display position of an object is changed by reducing the gesture operation space as described in . In addition, this process is repeatedly executed by the control unit 205 while the user 3 uses the HMD 1.

[0084] S901 : The distance image acquisition unit 106 of the HMD 1 acquires a distance image in front.

[0085] S902: Determine whether there is a person or object in the gesture operation space based on the acquired distance image. If there is no person or object in the operation space, proceed to S907. If there is a person or object in the operation space, proceed to S903.

[0086] S903: Calculate the size of the operation space so that the location of a person or object does not overlap with the operation space. In the calculation method, for example, the radius of the spherical operation space is reduced and the calculation is performed by checking whether there is overlap with a person or object.

[0087] S904: Displaying an operation object for performing gesture operation in the calculated operation space. The display position at this time is determined by a position as far away as possible from the person or object mapped on the distance image. For example, the space around the content being viewed, up, down, left, and right is recorded as a candidate for the display position in advance, and a position as far away as possible from the person or object mapped on the distance image is selected from the candidates.

[0088] S905: Displaying an object representing a new gesture operation space according to the change in the size of the operation space.

[0089] S906: According to the change in the size of the operation space, a new detection area is set when the distance image acquisition unit 106 detects the gesture operation. This completes the adjustment of the operation space.

[0090] S907: Since there is no person or object in the operation space, the size of the operation space, the display position of the operation object, and the detection area of ​​the gesture operation are returned to the initial values.

[0091] In the above flowchart, the case where the gesture operation space is reduced in order to avoid collision is shown, but the same is true when the gesture operation space is moved. Alternatively, the two can be combined to perform adjustment.

[0092] In addition, the above embodiments describe the situation where the HMD user is sitting, but it can also be used while standing or walking. In particular, when using while walking, even if others do not enter the gesture operation space, it is expected that others will enter the operation space due to continuous walking. At this time, a warning can be issued to the user by displaying a warning object or issuing a warning sound. In addition, when issuing a warning, it is also possible to set a priority when issuing a warning, such as identifying whether the person approaching is an adult or a child based on the distance image of the distance image acquisition unit, and issuing a warning earlier in the case of a child.

[0093] According to the second embodiment, a user using an HMD can avoid collision with the body of a person who is approaching and not using an HMD due to gesture operation, and an HMD having an operation space that is convenient for the user is provided.

[0094] Description of Reference Numerals

[0095] 1. 1': Head-mounted display device (HMD),

[0096] 2. 2': Smartphone,

[0097] 3.3': User,

[0098] 4: Others,

[0099] 5a, 5b: Beacons for obtaining location information,

[0100] 20: Communication cables,

[0101] 32, 32', 33, 33': the user's hand,

[0102] 101a, 101b: system storage unit,

[0103] 102a, 102b: cameras,

[0104] 103a, 103b: Position information radio wave receiving unit (position information acquisition unit),

[0105] 104a, 104b: image projection unit,

[0106] 105a, 105b: perspective lenses,

[0107] 106: distance image acquisition unit,

[0108] 201: Memory,

[0109] 202: Memory,

[0110] 203: bus,

[0111] 204: Input unit,

[0112] 205: Control Department,

[0113] 206: Smartphone Communications Department,

[0114] 300-303: Mixed reality objects,

[0115] 600, 600': Operation space object,

[0116] 601~606:Operation objects,

[0117] 611: Smartphone Objects,

[0118] 612: User's hand object,

[0119] 701: Mixed reality content.

Claims

1. A head mounted display device capable of displaying mixed reality images and capable of being operated by gestures by a user, characterized in that: include: an image projection unit for projecting an image; A display unit that overlaps the image projected from the image projection unit with the surrounding space to display it as a mixed reality object; a receiving unit for receiving position information of the head mounted display device; a communication unit capable of communicating with an external device; and Control Department, The control unit, setting a first operation space for a user of the head mounted display device to perform gesture operation, controlling the image projection unit to display the mixed reality object via the display unit, wherein the mixed reality object is an image of the first operation space and an image of an operation object displayed in the first operation space and used for gesture operation by a user of the head mounted display device, acquiring, through the communication unit, the position of the external device and information about a second operation space in which a user of the external device performs gesture operation, determining whether the first operation space overlaps with the second operation space based on the position information of the head mounted display device, the position of the external device, and the information about the second operation space received by the receiving unit, When the first operation space overlaps with the second operation space, the communication unit is controlled to connect to the smartphone, and the gesture operation is switched to the operation of the smartphone.

2. A head mounted display device capable of displaying mixed reality images and capable of being operated by gestures by a user, characterized in that: include: an image projection unit for projecting an image; A display unit that overlaps the image projected from the image projection unit with the surrounding space to display it as a mixed reality object; a receiving unit for receiving position information of the head mounted display device; a communication unit capable of communicating with an external device; and Control Department, The control unit, setting a first operation space for a user of the head mounted display device to perform gesture operation, controlling the image projection unit to display the mixed reality object via the display unit, wherein the mixed reality object is an image of the first operation space and an image of an operation object displayed in the first operation space and used for gesture operation by a user of the head mounted display device, acquiring, through the communication unit, the position of the external device and information about a second operation space in which a user of the external device performs gesture operation, determining whether the first operation space overlaps with the second operation space based on the position information of the head mounted display device, the position of the external device, and the information about the second operation space received by the receiving unit, When the first operation space overlaps with the second operation space, a third operation space that is smaller in size than the first operation space is calculated so as not to overlap with the second operation space. When the third operation space is displayed as the mixed reality object via the display unit, the image projection unit is controlled so as not to display a portion of the image of the operation object.

3. The head mounted display device according to claim 1, wherein: The control unit notifies a user of the head mounted display device that a part of the gesture operation is restricted, without displaying a part of the image of the operating object.

4. The head mounted display device according to claim 1, wherein: The control unit controls the communication unit to connect to the smartphone when a portion of the image of the operating object is not displayed, and switches the operation corresponding to the non-displayed operating object from the gesture operation to the operation of the smartphone.

5. The head mounted display device according to claim 1 or 2, wherein: The operation object is used to select content, start viewing, and end viewing.

6. The head mounted display device according to claim 1 or 2, wherein: A camera is also provided for detecting the gesture operation performed by the user of the head mounted display device.

Images