Head-mounted display, head-mounted display system, and head-mounted display method

Abstract

The purpose of the present invention is to provide: a virtual reality head-mounted display (VRHMD) that can appropriately ascertain the surrounding situation even while a VR space is being experienced, by determining whether or not the surrounding situation is one that the wearer of the VRHMD wants to ascertain even if the wearer is outside of a safe activity range, and displaying the determination on a display of the VRHMD even if the wearer is outside of the safe activity range according to the determination result; and a system including the VRHMD. Another purpose is to provide a display method regarding said display.

Description

[Technical Field]

[0001] The present invention relates to a head mounted display for virtual reality (VR), a system using the head mounted display, and a display method for the head mounted display. Note that hereinafter, the head mounted display for virtual reality (VRHMD) may be referred to as a VRHMD. [Background technology]

[0002] Virtual spaces (hereinafter sometimes referred to as VR spaces) are used in a variety of fields, including games, education, and tourism. A VRHMD is used to experience these VR spaces. A VRHMD is, for example, a device worn on the head that displays virtual space images on a goggle-like display. This device is equipped with multiple sensors, such as a camera, a sensor for measuring distance to objects, and a positioning sensor, as well as a CPU for image processing and a battery. When wearing this VRHMD and experiencing a VR space, depending on the content, the wearer may be able to move freely within the VR space. However, the actual space in which the wearer is located contains various objects (obstacles) such as walls and desks, limiting the area in which they can move. Therefore, for safety reasons, limitations on the area of ​​activity are imposed to avoid these obstacles. When the VRHMD wearer approaches a boundary within which they can safely move, the boundary is superimposed on the VRHMD display, thereby making the wearer aware of the limits of their movement.

[0003] Here, when the obstacle is fixed, it is useful to display on the display that the VRHMD wearer is approaching the boundary of the above-mentioned safe activity area, but it is also possible that an object such as a person, an animal such as a dog, or a ball may invade this safe activity area. From this perspective, there is known technology that, when a person, animal, or the like invades the safe activity area, displays the invading person, animal, or the like by superimposing it on the VRHMD display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-257716 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-143976 Summary of the Invention [Problem to be solved by the invention]

[0005] When wearing a VRHMD and experiencing a VR space, the immersive feeling makes the participant want to understand their surroundings, especially what is happening outside the safe activity zone. Some of the reasons for this include the following: - You don't want others to see you immersed in a VR space. Someone appears in the real world where the VRHMD wearer is. For some reason, you want to let the wearer know, but the wearer is engrossed and unable to speak up. · A phone call is coming in. The chime rings to indicate a visitor has arrived. There are various things that can be mentioned.

[0006] In conventional examples, if a person or other entity enters the safe activity range of the VRHMD wearer, the person or entity can be displayed by superimposing it on the VRHMD display, but if the person appears outside the safe activity range, the situation cannot be grasped.

[0007] Therefore, an object of the present invention is to provide a VRHMD and a system equipped with a VRHMD that can appropriately grasp the surrounding situation even while experiencing a VR space by determining whether the surrounding situation is one that the wearer wants to grasp, even if it is outside the safe activity range of the VRHMD wearer, and displaying the surrounding situation on the VRHMD display according to the determination result, even if it is outside the safe activity range.An object of the present invention is also to provide a display method related to this display. [Means for solving the problem]

[0008] According to a first aspect of the present invention, there is provided the following head-mounted display. That is, the head-mounted display is a head-mounted display for a virtual space. The head-mounted display includes a display, a camera, a distance detection unit, an image generation unit, a memory unit, and a control unit. The display displays an image. The camera captures an image of real space. The distance detection unit detects the distance to an object existing in real space. The image generation unit generates an image to be displayed on the display. The memory unit stores type conditions and distance conditions for the object to be displayed. The control unit then recognizes the type of object from the image captured by the camera, extracts an object that matches the type condition and distance condition, and superimposes an image showing the extracted object on an image of the virtual space and displays it on the display.

[0009] According to a second aspect of the present invention, there is provided a head-mounted display system as follows. That is, the head-mounted display system includes a camera that captures images of real space and a head-mounted display for virtual space. The head-mounted display includes a display that displays images, a distance detection unit that detects the distance to an object existing in real space, an image generation unit that generates an image to be displayed on the display, a memory unit that stores type conditions and distance conditions for the object to be displayed, and a control unit. The control unit recognizes the type of object from the image captured by the camera, extracts objects that match the type conditions and distance conditions, and superimposes an image showing the extracted object on an image of the virtual space and displays it on the display.

[0010] According to a third aspect of the present invention, there is provided a display method for a head-mounted display, which is described below. This display method is performed using a head-mounted display for a virtual space. This method includes a storage step of storing type conditions and distance conditions for an object to be displayed, an image generation step of generating an image depicting the virtual space, an imaging step of capturing an image of the real space around the head-mounted display, a distance detection step of detecting the distance to an object existing in the real space, a recognition step of recognizing the type of object from the captured image, an extraction step of extracting an object that matches the type condition and distance condition from the recognized objects, and a superimposed display step of superimposing and displaying an image of the extracted object on an image of the virtual space. [Effects of the Invention]

[0011] According to the present invention, a VRHMD and a system including the VRHMD are provided that determine whether the surrounding situation is one that the wearer wants to understand, even if it is outside the safe activity range of the VRHMD wearer, and display the surrounding situation on the VRHMD display according to the determination result, even if it is outside the safe activity range, thereby enabling the wearer to appropriately understand the surrounding situation even while experiencing a VR space. Also provided are a display method related to this display. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a VRHMD. [Figure 2] This is a diagram used to explain the actual real-world space in which a VRHMD wearer is located. [Figure 3] FIG. 2 is a diagram illustrating an example of the hardware configuration of a VRHMD. [Figure 4] FIG. 1 is a diagram used to explain an example of the configuration of a camera. [Figure 5] FIG. 1 is a diagram used to explain an example of the configuration of a camera. [Figure 6A] FIG. 10 is a diagram illustrating an example of a method for acquiring an image of the surroundings. [Figure 6B] FIG. 10 is a diagram illustrating an example of a method for acquiring an image of the surroundings. [Figure 7] FIG. 10 is a diagram for explaining an example of displaying the boundary of the safe activity area. [Figure 8] 10 is a flowchart illustrating an example of an operation flow in initial setting of a VRHMD. [Figure 9] FIG. 10 is a diagram illustrating an example of a VR space image displayed on a display. [Figure 10A] FIG. 10 is a diagram showing an example of a VR space image on which an object is displayed superimposed. [Figure 10B] FIG. 10 is a diagram showing an example of a VR space image on which an object is displayed superimposed. [Figure 11] 10 is a flowchart illustrating an example of processing during operation of the VRHMD according to the first embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a VR space image on which a virtual object representing an object is superimposed and displayed. [Figure 13] 10 is a flowchart illustrating an example of processing during operation of a VRHMD according to a second embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of a hardware configuration of a voice detection processing unit according to a third embodiment. [Figure 15] 11 is a flowchart illustrating an example of processing during operation of a VRHMD according to a third embodiment. [Figure 16] FIG. 13 is a diagram illustrating an example of boundary setting according to the fourth embodiment. [Figure 17] FIG. 10 is a diagram illustrating an example of a VR space image on which a virtual object representing an object is superimposed and displayed. [Figure 18] FIG. 13 is a diagram used to explain an example of a method for detecting an object that exists outside the field of view according to the fifth embodiment. [Figure 19] FIG. 10 is a diagram illustrating an example of a VR space image on which a virtual object representing an object is superimposed and displayed. [Figure 20] 13 is a flowchart illustrating an example of processing during operation of a VRHMD according to a fifth embodiment. [Figure 21]FIG. 13 is a diagram illustrating an example of a mode in which a smartphone is used according to the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Similar components throughout the drawings will be designated by the same reference numerals, and duplicate explanations may be omitted. According to the embodiment, an HMD (Head Mounted Display) is provided that allows users to properly grasp the surrounding situation even when outside the safety activity area. As a result, it is possible to contribute to, for example, the 9th Sustainable Development Goal (SDG) advocated by the United Nations ("Build resilient infrastructure, promote inclusive and sustainable industrialization, and build resilient technological infrastructure").

[0014] First Embodiment A first embodiment will be described with reference to FIGS. 1 to 11. First, an overview of a VRHMD will be described with reference to FIGS. 1 to 3. FIG. 1 is an example of a VRHMD according to one embodiment of the present invention, and is a diagram showing the VRHMD in a worn state and a display inside the VRHMD. FIG. 2 is a diagram used to explain the actual real space in which a person wearing a VRHMD is located.

[0015] 1, VRHMD 1 is provided with camera 200 and the like, and is worn on the head of a user. Camera 200 captures images of the real space around the wearer. Inside VRHMD 1, a display 130 is provided, and the created VR space image, the real space image captured by camera 200, and the like are displayed on display 130.

[0016] A person wearing a VRHMD 1 experiences a VR space in a real space as shown in Figure 2. During the VR experience, the wearer may move in various directions, such as forward, backward, left, right, or diagonally, as shown by arrow 8, depending on the content of the VR.

[0017] However, as shown in Figure 2, the real space in which the wearer is located may contain various objects, such as a chair 4, a desk (5, 12), a computer 6, a telephone 11, a person 20, an animal 30, a door 15, a window 7, and a wall 3. For this reason, when the wearer moves or moves their hands during the VR experience, they must avoid these objects. An example of a safe activity range in which they can move and act safely without coming into contact with these objects is shown by the dotted line 10 in Figure 2. Note that during the VR experience, a VR space image is displayed on the display 130, and these objects cannot be recognized.

[0018] Next, an example of the hardware configuration of a VRHMD will be described with reference to Fig. 3. As shown in Fig. 3, VRHMD 1 includes a control circuit 104, a sensor unit 105, a communication processing unit 106, a video processing unit 107, and an audio processing unit 108, all of which (104 to 108) are connected via a data bus 103 for exchanging various data. VRHMD 1 also includes a battery 109 as a power source.

[0019] As an example, control circuit 104 can be configured using main processor 2, RAM (Random access memory) 141, ROM (Read only memory) 142, and flash memory 143 that stores initial setting information, etc., and is configured to include a control unit and a storage unit. Main processor 2 uses programs and data stored in ROM 142 and flash memory 143 and output data from each unit (105-108) to control the operation of VRHMD 1 and various predetermined processes related to the present invention.

[0020] The sensor unit 105 can be configured using, for example, a GPS receiving sensor 151 that can be used to acquire location information, a geomagnetic sensor 152, a distance sensor 153 that can detect the distance to an object, an acceleration sensor 154, a gyro sensor 155, and a temperature sensor 156, and can be used to grasp data such as the state of the wearer and the position, size, and temperature of surrounding objects. However, the sensors listed here are only examples, and as long as they can execute predetermined processing, the listed sensors may be omitted as appropriate, or other types of sensors may be included.

[0021] The image processing unit 107 is used to generate and display images, and can be configured, for example, using a camera 200, a VR space image generation unit 195 (virtual space image generation unit in FIG. 3), an image superimposition processing unit 196, and a display 130. The VR space image generation unit 195 is a component used to generate images in the VR space. The image superimposition processing unit 196 is a component used to superimpose images on the VR space.

[0022] The audio processing unit 108 can be configured using, for example, a microphone 181, a codec 182 that processes audio signals, and a speaker 183. The microphone 181 is provided as appropriate, and, for example, may be provided so that the wearer's voice is input. The microphone 181 may also be provided so that external voice is input when the device is being worn. The speaker 183 may, for example, be provided so that it is close to the wearer's ear when the device is being worn.

[0023] For example, the communication processing unit 106 can be configured using a wireless LAN interface 161 and a short-range communication interface 162. The wireless LAN interface 161 is used as a communication interface for wireless LAN communication, and the short-range communication interface 162 is used as a communication interface for short-range communication. Note that, for example, Bluetooth (registered trademark) can be used as the short-range communication interface.

[0024] Camera 200 captures 360° images around the wearer. An example of the configuration and operation of camera 200 will now be described with reference to Figures 4-5, 6A, and 6B.

[0025] As shown in FIG. 4, camera 200 is equipped with two image capturing units (201, 202) that can capture images from two locations, one in front of and one behind the wearer. Here, image capturing units (201, 202) are configured to allow external light to enter, and as an example, may be configured with an opening formed therein for the light to enter. As shown in FIG. 5, camera 200 includes a front lens 210 with a wide viewing angle for capturing images of the front, a rear lens 220 with a wide viewing angle for capturing images of the rear, image capturing elements (211, 221) corresponding to each lens (210, 220), signal processing units (212, 222) for performing signal processing, and a 360° video creation unit 230 that generates a 360° image of the surroundings from the captured images of the front and rear. Here, when the viewing angles of the front and rear lenses are narrow and blind spots exist in the image capturing, making it impossible to obtain a 360° image of the surroundings, images are captured by the following method, for example.

[0026] That is, as indicated by arrow 8 in FIG. 2 described above, it is assumed that the wearer of VRHMD 1 will move in various directions and turn his or her head to look around. Here, for example, if the view angle that can be captured by front lens 210 and rear lens 220 of camera 200 is between dotted line 607 and dotted line 608 or between dotted line 606 and dotted line 609 in FIG. 6A, the objects that can be captured are people (700, 704) and animals (702, 703), and desk 701 will not be captured. If the wearer moves his or her head in this state, as shown in FIG. 6B, the objects that will be captured are people (700, 704) and desk 701, and animals (702, 703) will not be captured. Here, by combining these captured images, a 360-degree video is acquired.

[0027] Next, the setting of the safe activity range will be explained. As described above with reference to FIG. 2, the real space in which the wearer is present may include a chair 4, a desk (5, 12), a computer 6, a telephone 11, a person 20, an animal 30, a door 15, a window 7, a wall 3, and so on. To safely move around or perform actions such as moving one's hands during a VR experience, the wearer must avoid these objects. In the example of FIG. 2, the safe activity range in which the wearer can move around or perform actions such as moving one's hands without coming into contact with these objects is, for example, the area indicated by the dotted line 10 (i.e., the space on the wearer's side, with the dotted line 10 as the boundary). Therefore, before starting the VR space experience, the range corresponding to the dotted line 10 is set.

[0028] In this embodiment, as shown in FIG. 7 , the VRHMD 1 plays a role similar to the dotted line 10 in FIG. 2 , superimposing and displaying the boundary of a safe activity area within which objects can be avoided on a real-space image captured by the camera 200. Specifically, the VRHMD 1 (1) uses the control circuit 104, sensor unit 105, image processing unit 107, and other components shown in FIG. 3 to detect the position, size, and distance from the wearer of objects such as a chair 4, a desk 5, and a person 20 present in the real space. That is, the position, size, and distance from the wearer of objects are detected in a 360-degree image of the real space captured and created by the camera 200. Next, (2) the VRHMD 1 automatically sets a boundary 100 within which the wearer can avoid the object based on the detection results. Finally, (3) the VRHMD 1 superimposes the boundary 100 on the image of the real space captured by the camera 200 and displays it on the display 130. This display allows the wearer to confirm the boundary 100 of the safe activity area before starting the VR space experience, allowing the wearer to immerse themselves in the VR space with peace of mind.

[0029] In the above description, the camera 200 captures a 360° surrounding area and uses the image, but the boundary 100 may be set on an image that does not capture a 360° surrounding area, such as an image captured by a camera that captures only the area in front of the wearer. In this case, objects around the wearer may be detected each time during the VR space experience, and the boundary 100 that allows the object to be avoided may be automatically set each time. For example, when using a VRHMD with a camera that captures the front, the boundary 100 that allows the object to be avoided may be set each time the head moves and the real space image in front changes.

[0030] Next, an example of an operational flow for setting the boundary 100 of the safe activity space will be described with reference to Fig. 8. Fig. 8 is a flowchart for describing an example of an operational flow in the initial setting of the VRHMD.

[0031] First, the user wears the VRHMD 1 on their head. Then, the VRHMD 1 starts initial settings for experiencing a VR space (S1). Note that this process may start automatically after wearing the VRHMD 1, or may start in response to a command input from the user using an appropriate input device.

[0032] Next, the user sets the type of object they want to recognize while experiencing the VR space, such as a person, an animal, or a ringing telephone (S2). This setting also allows the user to set the number of objects. For example, if the user is experiencing the VR space with a large number of people and there are more people than the set number, the setting may be set to not recognize people. It is also possible to use appropriate face recognition technology to set the system to recognize only specific people. The set information is stored in the storage unit.

[0033] Next, the surroundings of the wearer are photographed by camera 200, and a 360° video is created (S3). VRHMD 1 also identifies objects (obstacles) from the created video, and uses data acquired by sensor unit 105 and the like to detect the position, distance from the wearer, size, etc. of the obstacle, and stores data on the identified obstacle, its position, distance from the wearer, size, etc. (S4).

[0034] The VRHMD 1 acquires relative position information to the object based on the positions and distances of objects such as the chair 4, desk 5, person 20, and wall 3 that exist in the real space acquired in S4 (S5).The VRHMD 1 then automatically sets a boundary 100 that can avoid contact with the object (obstacle) based on the data acquired in S4 and S5 (S6).

[0035] The VRHMD 1 displays the set boundary 100 on the display 130, superimposed on an image of real space captured by the camera 200. The wearer then looks at the image output on the display 130 to check whether the boundary 100 is appropriate (S7).

[0036] If the check result in S7 is OK, the VRHMD 1 stores the position information of the boundary 100 and creates a VR space image (S8). The created VR space image is then displayed on the display 130 as shown in FIG. 9. On the other hand, if the check result in S7 is NG, the process returns to S6, and the VRHMD 1 resets the boundary 100. Note that the check result may, for example, be input by the wearer via an appropriate input device. Furthermore, the VRHMD 1 may perform processing to consider the check result to be OK or NG after a predetermined time has passed.

[0037] After the VR space image is created, the initial settings are completed (S9).

[0038] Next, an example of a display method according to the present invention will be described. One of the objectives of the present invention is to enable the wearer to grasp the situation around the wearer, particularly the situation outside the boundary 100 of the safe activity area, only when necessary, while maintaining the sense of immersion as much as possible during the experience of the VR space.

[0039] 10A and 10B show an example of a display mode according to the first embodiment of the present invention. Figures 10A and 10B show an example in which a person 20 or an animal 30 existing outside the boundary 100 of the safe operating area described in Figure 7 is identified, and the person 20 or the animal 30 is superimposed on the VR space image shown in Figure 9 and displayed on the display 130.

[0040] Camera 200 captures 360° surrounding images as shown in Figures 6A and 6B during the VR space experience. If VRHMD 1 identifies an object set in initial setting S2 from the captured surrounding image that exists outside boundary 100, such as chair 4, desk (5, 12), computer 6, telephone 11, person 20, animal 30, door 15, window 7, or wall 3, during the VR experience, it displays the captured image of person 20, animal 30, etc. superimposed on the VR space image. Note that if an object that outputs sound, such as a ringing telephone, is set, microphone 181 that inputs sound from outside may be used to identify the object.

[0041] FIG. 10A shows a display when a person and an animal are set in the initial setting S2. Both the person 20 and the animal 30 exist outside the boundary 100, but because they are set in S2, these objects are displayed superimposed in the VR space. On the other hand, FIG. 10B shows a display when only a person is set in the initial setting S2. Both the person 20 and the animal 30 exist outside the boundary 100, but because only the person 20 is set in S2, only the person 20 is displayed superimposed in the VR space, and the animal 30 is not displayed. In this way, it is possible to display only the objects that the wearer initially set as objects that he or she wants to understand the situation.

[0042] If a new object that was not present at the time of initial setup appears inside the boundary 100 in the surrounding image, it will hinder safe movement and operation. For this reason, VRHMD 1 displays a captured image of the object superimposed on the VR space image, regardless of the type of the new object.

[0043] As described above, when the VRHMD 1 identifies that an object set outside the boundary 100 has newly appeared, it displays a captured image of the object superimposed on the VR space image. This allows the wearer to grasp external situations that the wearer may want to be aware of. On the other hand, when an object is identified as not set, it is not displayed unless it interferes with safe operation, so the sense of immersion in the VR space is not impaired. In this way, according to this embodiment, a VRHMD is provided that can display an image with an appropriate balance between understanding the surrounding situation and the sense of immersion, which are in a trade-off relationship.

[0044] Next, an operation flowchart according to the first embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart for explaining an example of processing during operation of the VRHMD.

[0045] When the experience of the VR space starts (S10), the VRHMD 1 generates a VR space image by the VR space image generation unit 195 of the video processing unit 107 (S11). Here, the VR space image is generated by a generation method similar to the generation method in S8 described above.

[0046] When the VRHMD 1 is in use, the camera 200 captures images of the wearer's surroundings and creates a 360-degree surrounding image (S12). The VRHMD 1 then detects objects from the created 360-degree surrounding image (S13). The VRHMD 1 identifies the position of the detected object using sensor functions such as the sensor unit 105 as appropriate, and compares this with the data detected in S4 described above. If a new object is detected, the object is stored, and if the object is moving, the direction of the object is detected (S14). The direction in which the object is moving can be detected, for example, using captured images (for example, by determining the object's moving direction using images captured at short time intervals).

[0047] The VRHMD1 identifies whether the object detected in S14 and the moving object are located outside or inside the boundary 100 (S15). It is assumed that the wearer is located inside the boundary 100.

[0048] If it is determined in S15 that the object exists outside the boundary 100, the VRHMD 1 determines the type of object. For example, the VRHMD 1 determines that the object is a person, an animal, a desk, a chair, etc. (S16). As one example, the VRHMD 1 can determine the type of object using a known image matching technique. Alternatively, the VRHMD 1 may determine the type of object by using an appropriate matching technique based on the input sound from the object.

[0049] It is determined whether the type determined in S16 matches the type previously set in S2 of Fig. 8. For example, if people and animals are set in S2, the VRHMD 1 extracts people and animals (S17).

[0050] If the object set in S2 is identified in S17 (YES) or if a moving object is detected in S14 (YES), the VRHMD 1 acquires (extracts) images of those objects (S18). The VRHMD 1 then superimposes the images acquired (extracted) in S18 on the VR space image (S20). The VRHMD 1 also displays the images superimposed in S20 on the display 130 (S21). After displaying in S21, the process returns to S11. On the other hand, if no object is extracted in S2 in S17 (NO), the VRHMD 1 displays the VR space image generated in S11 as is on the display 130 (S21). If an object is found inside the boundary 100 in S15 and that object is a new object or a moving object detected in S14, the VRHMD 1 extracts those images (S19). Then, the VRHMD 1 superimposes the image acquired by extraction in S19 on the VR space image (S20). The VRHMD 1 also displays the image superimposed in S20 on the display 130 (S21). Note that when the processing of S19 is performed, there is a high possibility that the wearer will come into contact with an object, so in this embodiment, the VRHMD 1 performs processing such as superimposing the image of S19 on the center part of the VR space image, or displaying the image of S19 and stopping the output of the VR space image.

[0051] As described above, when it is recognized that an object set outside the boundary 100 has newly appeared, a captured image of the object (more specifically, an image in which a portion of the object has been cut out from an image captured by the camera 200, or an image in which the outline of the object has been extracted) is superimposed on the VR space image and displayed. This makes it possible for the wearer to grasp external situations that they may want to be aware of. On the other hand, when it is recognized as an object that has not been set, it is not displayed unless it interferes with safe operation, so that the sense of immersion in the VR space is not impaired. In this way, according to this embodiment, a VRHMD is provided that can display an image with an appropriate balance between understanding the surrounding situation and the sense of immersion, which are in a trade-off relationship.

[0052] Second Embodiment Next, a second embodiment will be described with reference to FIGS. 12 and 13. Functions similar to those in other embodiments are assigned the same reference numerals, and descriptions thereof may be omitted. In the second embodiment, if an object to be grasped, set in S2 of FIG. 8, exists outside the boundary 100 while experiencing the VR space, the VRHMD 1 replaces the object with a virtual object. Then, the VRHMD 1 superimposes the replaced virtual object on the top, bottom, left, and right edges of the VR space image in accordance with its actual location, and displays it on the display 130.

[0053] In the second embodiment, camera 200 first creates a 360-degree surrounding image, and VRHMD 1 identifies objects from the 360-degree image. Of the identified objects, VRHMD 1 detects an object that exists outside boundary 100 and matches the object to be grasped (e.g., person 20 or animal 30) set in S2 of Fig. 8, and identifies whether the object is located in front of, behind, to the right, to the left, or diagonally from the wearer. Then, VRHMD 1 replaces the detected object with a virtual object and displays it in accordance with the direction of the object relative to the wearer's location.

[0054] An example of virtual object display in this embodiment will be described with reference to Fig. 12. As shown in Fig. 12, VRHMD1 displays virtual objects of objects superimposed on dotted-line frames 111, 112, 113, and 114 at the edges of the VR space image. In this way, by displaying the objects as virtual objects and at the edges of the VR space image, the wearer can grasp the surrounding situation without losing the sense of immersion in the VR space.

[0055] Here, Fig. 12 shows the display of the VRHMD 1 in the situation shown in Fig. 2. In this example, a person 20 is present in front of the wearer, so a virtual object of the person 20 present in front is displayed within an upper dotted frame 111. Also, an animal 30 is present to the right of the wearer, so a virtual object of the animal 30 present on the right is displayed within a right dotted frame 113. In this way, the virtual object is displayed according to the direction in which it exists relative to the position of the wearer.

[0056] Next, an operational flowchart of the second embodiment will be described with reference to Fig. 13. Note that the same functions as those in other embodiments are given the same reference numerals, and the description thereof may be omitted.

[0057] First, when the experience of the VR space begins (S10), the VRHMD 1 generates a VR space image using the VR space image generation unit 195 of the image processing unit 107 (S11). Then, the camera 200 captures the surroundings of the wearer and creates a 360° surrounding image (S12), and the VRHMD 1 detects objects from the created 360° surrounding image (S13).

[0058] The VRHMD 1 determines the position of the detected object using the sensor functions of the sensor unit 105, etc. as appropriate, and also determines whether the object is located in front, behind, to the right, to the left, or diagonally from the wearer. The VRHMD 1 also compares the detected data with the data detected in S2 of FIG. 8, and if a new object is detected, it stores the object, and if the object is moving, it detects the direction of movement (S14).

[0059] Note that the direction of an object may be determined by the following method, for example. The VRHMD 1 determines and processes an object located in the horizontal center of the captured image as an object located in the same direction as the camera 200 (for example, forward or backward), and determines and processes an object located on the left or right edge of the captured image as an object located laterally (for example, left or right). Then, the VRHMD 1 determines and processes an object located in the middle of the two in the captured image as an object located diagonally.

[0060] The VRHMD 1 identifies whether the object detected in S14 and the moving object are located outside or inside the boundary 100 (S15). Note that here, the wearer is located inside the boundary 100. If the object is located outside the boundary 100 in S15, the VRHMD 1 determines the type of object (S16).

[0061] The VRHMD 1 determines whether the type determined in S16 matches the type previously set in S2 of FIG. 8. For example, if a person and an animal are set in S2, the VRHMD 1 extracts the person and the animal (S17). If an object set in S2 is extracted in S17 (YES), or if a new object is detected in S14 (YES), the VRHMD 1 replaces those objects with virtual objects. Here, the VRHMD 1 may replace a person with a human-shaped object, or an animal with an animal-shaped object (S31). Note that the replacement method is not limited to the method described above. The object may have any shape that allows the object to be identified.

[0062] 12, the VRHMD 1 superimposes the image replaced with the virtual object in S31 on the portion of the VR space image enclosed by the dotted line (111-114) in accordance with the direction of the wearer detected in S14 (S32). Then, the VRHMD 1 displays the image superimposed in S32 on the display 130 (S21). After the display in S21, the process returns to S11.

[0063] On the other hand, if the object set in S2 is not extracted (does not match) in S17 (NO), the VRHMD 1 displays the VR space image generated in S11 as is on the display 130 (S21). Also, if an object is found inside the boundary 100 in S15 and that object is a new object or a moving object detected in S14, the VRHMD 1 extracts those images (S19). Note that because there is a high possibility that the wearer will come into contact with the object in the image of S19, in this embodiment, the VRHMD 1 performs processing such as superimposing the image of S19 on a portion of the VR space image that is not enclosed by the dotted frame 111 (for example, the center portion), or displaying real space together with the image of S19 instead of the VR space image to interrupt immersion in the VR space.

[0064] As described above, when an object set outside the boundary 100 is identified as newly appearing, the object is displayed as a virtual object superimposed on the VR space image. This allows the wearer to grasp external situations that the wearer may want to be aware of. On the other hand, when an object is identified as not set, the object is not displayed unless it interferes with safe operation, so the sense of immersion in the VR space is not impaired. In this way, according to this embodiment, a VRHMD is provided that can display an appropriate balance between understanding the surrounding situation and the sense of immersion, which are in a trade-off relationship.

[0065] Third Embodiment Next, a third embodiment will be described with reference to Figures 14 and 15. Functions similar to those in other embodiments are given the same reference numerals, and descriptions thereof may be omitted. In the third embodiment, the VRHMD 1 detects an object that is emitting a sound, such as a telephone, and displays that a sound is being generated.

[0066] As already explained, VRHMD 1 can determine the presence and location of an object such as a telephone by using the 360° surrounding image captured by camera 200. On the other hand, sound detection processing is required to detect whether the determined object is emitting sound (for example, a telephone ringing or a person speaking).

[0067] FIG. 14 shows an example of the hardware configuration of the voice detection processing unit 300 in this embodiment. The voice detection processing unit 300 includes the microphone 181 of the voice processing unit 108 shown in FIG. 3 and a codec 182 (voice processing device). The microphone 181 is composed of a left microphone 301, a left microphone amplifier 311 (microphone amplifier 311), a right microphone 302, and a right microphone amplifier 321 (microphone amplifier 321). The codec 182 is composed of a left signal processing unit 312, a right signal processing unit 322, and a 360° sound image creation unit 330. The signal processing units (312, 322) perform signal processing on the sounds collected by the two left and right microphones (301, 302) to generate digital signals. The 360° sound image creation unit 330 creates a sound image and generates data for determining the direction of sound generation and the type of sound (e.g., a telephone ringtone or a human voice).

[0068] An example of the operation flow of the third embodiment will be described using Fig. 15. Fig. 15 is a flowchart for explaining an example of processing during operation of the VRHMD. In the third embodiment, the VRHMD 1 judges sounds to prevent misidentification between a mannequin and a person, or a stuffed toy and an animal. Note that functions similar to those in other embodiments will be assigned the same reference numerals, and explanations may be omitted.

[0069] When the user starts experiencing the VR space (S10), the VRHMD 1 generates a VR space image (S11). The camera 200 captures the wearer's surroundings, and the VRHMD 1 creates a 360-degree surrounding image (S12). The VRHMD 1 detects objects from the created 360-degree surrounding image (S13).

[0070] The VRHMD1 measures the temperature of the object detected in S13 using temperature sensor 156 and compares it with the data detected in S4 of Fig. 8 to distinguish between a mannequin and a person with a body temperature, and between a stuffed animal and an animal with a body temperature (S41). Note that if the VRHMD is not equipped with a temperature sensor, S41 is skipped.

[0071] The VRHMD1 identifies the position of the object detected in S13 and determines whether the object is in front, behind, to the right, to the left, or diagonally from the wearer. If a new object is detected as a result of comparison with the data detected in S4 of Fig. 8, the VRHMD1 stores the object, and if the object is moving, detects the direction of movement (S14).

[0072] The VRHMD 1 detects the location where the sound is being generated based on the data from the sound detection processing unit 300, and compares the output data of S14 with the data detected in S4 of FIG. 8 to identify the object that is generating the sound. In addition, when an emergency bell is ringing, the location where the sound is being generated may be recognized as simply a wall. In this case, the VRHMD 1 determines that no object corresponding to the sound source can be found (S42). Note that the image from the camera 200 may also be used to identify the object that is generating the sound.

[0073] The VRHMD1 identifies (S43) whether the output data of S42 relates to an object existing outside the boundary 100 or an object existing inside the boundary 100. Here, the wearer is inside the boundary 100.

[0074] If an object or sound is detected outside the boundary 100 in S43, the VRHMD1 determines the type of object or sound. For example, the VRHMD1 determines whether the object or sound is a ringing telephone, a person calling, a chime announcing a visitor, an emergency bell, or the like (S44).

[0075] The VRHMD1 determines whether the type determined in S24 matches the type previously set in S2 of Fig. 8. The VRHMD1 extracts, for example, a telephone set ringing with the ringtone set in S2, the voice of a person calling, or a chime announcing a visitor (S17).

[0076] If the object set in S2 is extracted in S17, or if a moving object is detected in S14 (YES), the VRHMD1 replaces those objects and sounds with virtual objects. For example, if a telephone is ringing, the VRHMD1 can replace it with an object in the shape of the telephone being called. Similarly, if it is a person making a call, the VRHMD1 can replace it with an object in the shape of a person that makes it clear that the person is making the call; if it is a chime notifying a visitor, the VRHMD1 can replace it with an object in the shape of a doorbell that makes it clear that the chime is ringing; and if it is an emergency bell ringing, the VRHMD1 can replace it with a virtual object of an emergency bell (S45).

[0077] The VRHMD 1 superimposes the image replaced with the virtual object in S45 onto the portion of the dotted frame 111 of the VR space image, in accordance with the direction of the wearer detected in S14. If it is determined in S44 that no corresponding object can be found, such as when an emergency bell is ringing, the VRHMD 1 may switch from the VR space image to a real space image. Furthermore, the VRHMD 1 may superimpose a virtual object of an emergency bell on the real space image to warn of danger (S32). Note that the operations from S32 onwards are the same as those in the operation flowchart of FIG. 13 described above, and therefore description thereof will be omitted.

[0078] As described above, when a sound that should notify the wearer is generated, even if no new object appears, a virtual object representing the generated sound is superimposed on the VR space image and displayed. This allows the wearer to grasp the external situation that the wearer may want to be aware of. On the other hand, if the sound is identified as being of low importance, it is not displayed, so the sense of immersion in the VR space is not impaired. In this way, according to this embodiment, a VRHMD is provided that can display a display that strikes an appropriate balance between the ability to grasp the surrounding situation and the sense of immersion, which are in a trade-off relationship.

[0079] <Fourth embodiment> Next, a fourth embodiment will be described with reference to FIGS. 16 and 17. Functions similar to those in other embodiments are designated by the same reference numerals, and their description may be omitted. FIG. 16 shows that a boundary 1000 is set outside the boundary 100 of the safe operating range. In FIG. 16, the area inside the boundary 100 is defined as a first area, the area between the boundary 100 and the boundary 1000 is defined as a second area, and the area outside the boundary 1000 is defined as a third area. The wearer of the VRHMD 1 faces in the direction of the arrow 70. FIG. 16 also shows an example in which a person 299, an animal 399, and a ringing telephone 199 exist in the second area, and a person 1200 and an animal 1300 exist in the third area. When setting the boundaries in S6 of the flowchart shown in FIG. 8, two boundaries, boundary 100 and boundary 1000, are set. That is, in this embodiment, the VRHMD 1 sets a boundary 100 that targets a distance (first distance) at which an object is displayed regardless of the type condition, and a boundary 1000 at a distance (second distance) at which an object is displayed only when the type condition is met. Note that this explanation is just an example, and it goes without saying that there is no limit to the number of boundaries that can be set.

[0080] In the fourth embodiment, when an object to be grasped, which was set in S2 of Fig. 8, is present in the set first area, second area, and third area 3, the VRHMD 1 determines whether to display the object present in each area and displays it superimposed on the VR space image. Fig. 17 shows an example of display on the VRHMD 1 when an object is present as shown in Fig. 16.

[0081] 17, similar to FIG. 12, while experiencing the VR space, VRHMD 1 replaces the object to be grasped, set in S2 of FIG. 8, with a virtual object and displays it superimposed on the upper, lower, left, and right edges of the VR space image, indicated by dotted-line frames 111, 112, 113, and 114, according to the object's location. As illustrated in FIG. 17, a person 299 present in the background is displayed superimposed on the lower dotted-line frame 114. Similarly, an animal 399 present on the left is displayed superimposed on the left dotted-line frame 112, a person 1200 present on the right and a ringing telephone 199 are displayed superimposed on the right dotted-line frame 113, and an animal 1300 present in the front is displayed superimposed on the upper dotted-line frame 111. Furthermore, the size of the objects of person 299 and animal 1300 present in the second area is made larger than the size of the objects of person 1200 and animal 1300 present in the third area, allowing the wearer to recognize the area in which the object is present.

[0082] In this process, in S16 and S44 of the above-described operational flowchart, the VRHMD 1 determines whether the object or sound is present in the second area or the third area. Then, in S32, the VRHMD 1 changes the size of the superimposed object depending on the area in which it is present.

[0083] Note that the setting for identifying objects in the second area may be limited to objects that are emitting sound. By setting it in this way, when experiencing a VR space in a large space such as a gymnasium, the space that the wearer wants to identify can be limited to a certain area (for example, a few meters) around the wearer. Another feature is that in areas beyond a certain range, for example, outside a gymnasium, only emergency bells and emergency announcements in the event of an emergency such as a fire can be identified.

[0084] As described above, multiple areas separated by boundaries are set, and virtual objects corresponding to the areas in which the objects exist are superimposed on the VR space image and displayed. This makes it possible to recognize objects according to their distance from the wearer. On the other hand, if safe operation is not impaired, the virtual objects are not displayed, and if the distance is far, they can be displayed inconspicuously, so the sense of immersion in the VR space is not impaired. In this way, according to this embodiment, a VRHMD is provided that can display an object with an appropriate balance between awareness of the surrounding situation and a sense of immersion, which are in a trade-off relationship.

[0085] Fifth Embodiment Next, a fifth embodiment will be described with reference to Figures 18 to 20. Functions similar to those in the other embodiments are given the same reference numerals, and descriptions thereof may be omitted. In the fifth embodiment, an example will be described in which processing is performed using data acquired through communication.

[0086] There are many devices with short-range communication interfaces (wireless communication devices), such as smartphones. This short-range communication interface uses radio waves and is intended for use over short distances of up to about 10 meters. This short-range communication interface periodically transmits ID information, and because it uses radio waves, it can be found even if it is located behind a wall or in other places that cannot be seen with the naked eye.

[0087] Therefore, for example, as shown in Fig. 18, VRHMD 1 can find smartphone 110 with a short-range communication interface that is located outside door 15. Here, Fig. 18 shows a situation in which a person carrying smartphone 110 with a short-range communication interface is located outside door 15. Then, as shown in Fig. 19, VRHMD 1 can display on display 130 by superimposing smartphone object 110 on dotted frame 112 in the lower left of the VR space image, to indicate that a person carrying this smartphone 110 is located around the wearer who is experiencing the VR space.

[0088] An example of the operation flow of the fifth embodiment will be described using Fig. 20. Fig. 20 is a flowchart for explaining an example of processing during operation of the VRHMD. Note that functions similar to those in other embodiments will be assigned the same reference numerals, and explanations may be omitted.

[0089] When the experience of the VR space starts (S10), the VRHMD 1 generates a VR space image (S11).

[0090] The short-range communication interface periodically transmits ID information. Therefore, the VRHMD 1 detects the short-range communication interface by acquiring radio waves from the short-range communication interface (S51). The VRHMD 1 also detects ID information from the acquired radio waves (S52).

[0091] Additionally, as one example, the VRHMD 1 detects (estimates) the distance of a device equipped with a short-range communication interface from the strength of the acquired radio waves (S53). Note that the VRHMD 1 may also detect (estimate) the distance of a device equipped with a short-range communication interface from the delay time in communication. Furthermore, as one example, position detection is also possible if a method capable of detecting direction, such as UWB (Ultra Wide Band), is used.

[0092] The VRHMD 1 determines whether the position of the device detected in S53 is outside or inside the boundary 100 (S15). If the device is determined to be inside the boundary 100 in S15, the process proceeds to S21. Note that the VRHMD 1 may not only determine the position based on distance, but may also detect whether the device is approaching or moving away and take that information into account when making the determination. For example, if the device is moving away even if it is outside the boundary 100 in terms of distance, it may be determined that there is little need to notify the wearer, and the process may proceed to S21.

[0093] If the device is found to be outside the boundary 100 in S15, it is determined whether the detected ID information matches the device that the user wants to track and that was set in S2 of Fig. 8 (S17). Note that, for the setting in S2, the user may select and register a device that the user wants to track and that the user wants to track, for example, from a list of devices with short-range communication interfaces that have been detected in the past.

[0094] If the device is identified in S17 (Yes), the VRHMD 1 replaces the identified device with a virtual object (S54). Then, as shown in Fig. 20, the VRHMD 1 superimposes the object of S54 on the portion enclosed by the dotted line frame 112 in the lower left of the VR space image (S32).

[0095] In this embodiment, an example has been described in which the object of S54 is superimposed on the portion of the dotted frame 112, but the display format is not limited to this example, and for example, the position of the displayed dotted frame can be changed as appropriate. Furthermore, if the orientation of the device can be identified, a display may be made that is aligned with the orientation of the wearer, as described above in FIG. 12. Furthermore, the devices may be classified by type, and a display may be made that is organized by device type.

[0096] As described above, a short-range communication interface is used to detect target devices and display virtual objects superimposed on a VR space image. This makes it possible to recognize target devices even in places where cameras cannot capture images. On the other hand, devices that are not registered as targets for detection are not displayed, so the sense of immersion in the VR space is not impaired. In this way, according to this embodiment, a VRHMD is provided that can display a display that strikes an appropriate balance between awareness of the surrounding situation and the sense of immersion, which are in a trade-off relationship.

[0097] Sixth Embodiment Next, a sixth embodiment will be described with reference to Fig. 21. Functions similar to those in other embodiments will be assigned the same reference numerals, and descriptions thereof will be omitted. In the sixth embodiment, an example of a VRHMD using a smartphone will be described.

[0098] 21, VRHMD 1 may be VR goggles 90 equipped with a smartphone 110. VRHMD 1 may perform similar processing using camera 200, distance sensor 153, temperature sensor 156 on the back side of smartphone 110, and display 130 on the front side of smartphone 110.

[0099] Here, the VR goggles 90 have an appropriate configuration to which the smartphone 110 can be attached. As an example, the VR goggles 90 may be smartphone goggles into which the smartphone 110 is attached by the user inserting the smartphone 110. The VR goggles 90 may also be smartphone goggles into which the smartphone 110 is attached by the user inserting the smartphone 110. Here, "smartphone" is an abbreviation for smartphone.

[0100] According to the above description, there is provided a VRHMD that recognizes the type of object from an image captured by camera 200, extracts objects that match type conditions and distance conditions, superimposes an image showing the extracted object on a VR space image, and displays it on display 130. Also, as an example, there is provided a display method for a head-mounted display that includes a storage step (S2) of storing type conditions and distance conditions for the object to be displayed, an image generation step (S11) of generating an image depicting a virtual space, an imaging step (S12) of imaging the real space around the head-mounted display, a distance detection step (S14) of detecting the distance to an object existing in the real space, a recognition step (S16) of recognizing the type of object from the captured image, extraction steps (S17, S18) of extracting objects that match the type conditions and distance conditions from the recognized objects, and a superimposed display step (S20, S21) of superimposing and displaying an image showing the extracted object on an image of the virtual space.

[0101] In this way, surrounding conditions such as people, equipment, and sounds can be detected even outside the VRHMD wearer's safe activity range, and it can be determined whether it is appropriate to inform the wearer of the conditions. If it is determined that the conditions should be made known, the detected conditions are displayed on the display by superimposing a captured image of the detected object, a virtual object representing the object, or an object indicating the direction of the object on the VR space image. This allows the wearer to grasp external conditions that they may want to be aware of. On the other hand, if an object is identified as not set, it is not displayed unless it interferes with safe operation, so the sense of immersion in the VR space is not impaired. Therefore, according to the present invention, it is possible to display a display that strikes an appropriate balance between the trade-off between understanding the surrounding conditions and the sense of immersion.

[0102] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0103] It is sufficient to be able to execute a predetermined process, and for example, the programs used in each process example may be independent programs, or multiple programs may constitute a single application program. Also, the order in which each process is performed may be changed.

[0104] Some or all of the functions of the present invention described above may be implemented in hardware, for example, by designing them as integrated circuits. They may also be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program that implements each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Some or all of the functions may also be implemented by a server. The server may be, for example, a local server, a cloud server, an edge server, or an online service, as long as it can cooperate with other components via communications to execute the functions. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0105] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0106] The positions of the cameras in the VRHMD 1 are not limited to the example described above, and the number and structure of the cameras 200 are not limited to the example described above and may be changed as appropriate.

[0107] An appropriate camera capable of communicating with the VRHMD 1 may be installed in the environment in which the VRHMD 1 is used, and the VRHMD 1 may perform processing based on captured images acquired from the camera via communication. In other words, a system including a camera and the VRHMD 1 may be provided.

[0108] This system may also use one camera to operate multiple VRHMDs 1. Therefore, for example, it is possible to easily operate the system by installing one or a small number of cameras so that the entire environment can be viewed.

[0109] Here, VRHMD 1 uses the image acquired by the camera to determine whether the object is the one set in S2. If it is determined that the object is the one set in S2, VRHMD 1 can superimpose and display the image of the object captured by the camera. Note that the object in the image acquired by the camera or the virtual object replacing the object may be superimposed at an appropriate predetermined position (for example, an edge of display 130) or a predetermined position, for example. Furthermore, when multiple cameras are installed and the image of an object is superimposed, for example, the object or virtual object captured from any one of the cameras may be superimposed.

[0110] In S2, non-superimposed objects are set, and the storage unit may store information indicating the types of objects not to be displayed. Then, VRHMD1 may perform processing to not display objects identified from this information. By setting non-superimposed objects in this way, the wearer can be immersed in the VR space without being aware of the objects. For example, by hiding a home appliance such as a robot vacuum cleaner, the wearer can be immersed in the VR space without being aware of the appliance, even when the appliance is in use.

[0111] For example, the VRHMD 1 may acquire data from the sensor unit 105 depending on the situation and perform processing. The VRHMD 1 may detect tilt using the acceleration sensor 154 or the gyro sensor 155, for example, and perform processing that corrects for the influence of the tilt.

[0112] To display information about the battery 109 (such as the current amount of electricity), the battery 109 may be connected to the data bus 103. The VRHMD 1 may then display the information about the battery 109 on the display 130. [Explanation of symbols]

[0113] 1:VR HMD 104: Control circuit 105: Sensor unit 106: Communication processing unit 107: Video processing unit 108: Audio processing unit 130: Display 200: Camera

Claims

1. A head-mounted display for a virtual space, a display for displaying an image; A camera that captures real space, a distance detection unit that detects a distance to an object existing in real space; an image generating unit that generates an image to be displayed on the display; a storage unit that stores a type condition and a distance condition of an object to be displayed; a control unit, the storage unit stores, as conditions, a first distance at which an object is displayed regardless of the type condition, and a second distance greater than the first distance at which an object that meets the type condition is displayed; The control unit Recognizing the type of object from the image captured by the camera; extracting an object that satisfies the first distance condition regardless of the type condition; extracting an object that meets the type condition and the second distance condition; a video image showing the extracted object is superimposed on the image of the virtual space and displayed on the display; A head-mounted display characterized by:

2. The head-mounted display according to claim 1, The image showing the extracted object is an image obtained by cutting out a portion of the object from the image captured by the camera, or an image obtained by extracting the outline of the object from the image captured by the camera. A head-mounted display characterized by:

3. The head-mounted display according to claim 1, the image showing the extracted object is a virtual object showing a type of the object; A head-mounted display characterized by:

4. The head-mounted display according to claim 1, If the object that satisfies the second distance condition is a moving object, an image showing the moving object is superimposed on the image in the virtual space and displayed on the display, regardless of the type condition. A head-mounted display characterized by:

5. The head-mounted display according to claim 1, The distance detection unit a microphone that collects surrounding sounds; and a sound processing device that creates data on surrounding sound images based on the collected sounds and that is used to identify the type of sound source and identify its location, The control unit Recognizing the type of sound source from the data; Extract objects that meet the type and distance conditions, a video image showing the extracted object is superimposed on the image of the virtual space and displayed on the display; A head-mounted display characterized by:

6. The head-mounted display according to claim 1, The distance detection unit a wireless communication interface; The storage unit storing the identification number information of the wireless communication device as a condition for the type of object to be displayed; The control unit estimating a distance to the wireless communication device from the received radio wave intensity or communication delay time of the wireless communication interface; extracting, as an object, a wireless communication device that matches a type condition and a distance condition based on the identification number information from the wireless communication devices connected via the wireless communication interface; a virtual object image showing the extracted object is superimposed on the image of the virtual space and displayed on the display; A head-mounted display characterized by:

7. The head-mounted display according to claim 1, The storage unit storing information indicating the type of object not to be displayed; The control unit not displaying the object identified from the information; A head-mounted display characterized by:

8. A head-mounted display system including a camera for capturing images of a real space and a head-mounted display for a virtual space, The head-mounted display includes: a display for displaying an image; a distance detection unit that detects a distance to an object existing in real space; an image generating unit that generates an image to be displayed on the display; a storage unit that stores a type condition and a distance condition of an object to be displayed; a control unit, the storage unit stores, as conditions, a first distance at which an object is displayed regardless of the type condition, and a second distance greater than the first distance at which an object that meets the type condition is displayed; The control unit Recognizing the type of object from the image captured by the camera; extracting an object that satisfies the first distance condition regardless of the type condition; extracting an object that meets the type condition and the second distance condition; a video image showing the extracted object is superimposed on the image of the virtual space and displayed on the display; A head-mounted display system.

9. 9. The head-mounted display system according to claim 8, The image showing the extracted object is an image obtained by cutting out a portion of the object from the image captured by the camera, or an image obtained by extracting the outline of the object from the image captured by the camera. A head-mounted display system.

10. 9. The head-mounted display system according to claim 8, the image showing the extracted object is a virtual object showing a type of the object; A head-mounted display system.

11. 9. The head-mounted display system according to claim 8, If the object that satisfies the second distance condition is a moving object, an image showing the moving object is superimposed on the image in the virtual space and displayed on the display, regardless of the type condition. A head-mounted display system.

12. 9. The head-mounted display system according to claim 8, The distance detection unit a microphone that collects surrounding sounds; and a sound processing device that creates data on surrounding sound images based on the collected sounds and that is used to identify the type of sound source and identify its location, The control unit Recognizing the type of sound source from the data; Extract objects that meet the type and distance conditions, a video image showing the extracted object is superimposed on the image of the virtual space and displayed on the display; A head-mounted display system.

13. 9. The head-mounted display system according to claim 8, The distance detection unit a wireless communication interface; The storage unit storing the identification number information of the wireless communication device as a condition for the type of object to be displayed; The control unit estimating a distance to the wireless communication device from the received radio wave intensity or communication delay time of the wireless communication interface; extracting, as an object, a wireless communication device that matches a type condition and a distance condition based on the identification number information from the wireless communication devices connected via the wireless communication interface; a virtual object image showing the extracted object is superimposed on the image of the virtual space and displayed on the display; A head-mounted display system.

14. 9. The head-mounted display system according to claim 8, The storage unit storing information indicating the type of object not to be displayed; The control unit not displaying the object identified from the information; A head-mounted display system.

Images