Head-mounted display, head-mounted display system, and method of displaying with a head-mounted display.

Abstract

This invention provides a VR HMD and a system equipped with a VR HMD that can appropriately understand the surrounding environment even while experiencing a VR space. This is achieved by determining whether the surrounding environment is something the wearer wants to be aware of, even if it is outside the wearer's safe activity range, and displaying the information on the VR HMD's display based on the determination, even if it is outside the safe activity range. [Solution] The head-mounted display comprises a display, a camera, a distance detection unit, an image generation unit, a storage unit, and a control unit. The display shows images. The camera captures images of the real world. The distance detection unit detects the distance to objects in the real world. The image generation unit generates images to be displayed on the display. The storage unit stores the type and distance conditions of the objects to be displayed. The control unit extracts objects that match the type and distance conditions from the images captured by the camera, superimposes them onto an image in a virtual space, and displays them on the display.

Description

Technical Field

[0001] The present invention relates to a head-mounted display (HMD) for virtual reality (VR), a system using the head-mounted display, and a display method in the head-mounted display. Hereinafter, the head-mounted display for virtual reality may be referred to as VRHMD.

Background Art

[0002] Virtual space (hereinafter sometimes referred to as VR space) is used in various fields such as games, education, and tourism. A VRHMD is used to experience this VR space. As an example, a VRHMD is a device that is worn on the head and displays a virtual space image on a goggle-shaped display. This device is, for example, equipped with a plurality of sensors such as a camera, a sensor for measuring the distance to an object, a position measurement sensor, a CPU for performing image processing, a battery, and the like. When wearing this VRHMD and experiencing the VR space, depending on the content, it is conceivable that the wearer can freely move around in the VR space. However, the actual space where the wearer is located has various objects (such as obstacles) such as walls and desks, and the place where the wearer can move around is limited. Therefore, for safety, when the wearer of the VRHMD approaches the boundary of the activity area where these obstacles can be avoided, that is, the limit of movement, such as superimposing and displaying the boundary on the display of the VRHMD, is performed to make the wearer recognize it.

[0003] Here, when the obstacle is fixed, it is useful to display on the display that the wearer of the VRHMD has approached the boundary of the above-mentioned safe activity range. However, it is also conceivable that people, animals such as dogs, and objects such as balls enter this safe activity range. Regarding such a viewpoint, when these people, animals, etc. enter the safe activity range, a technique of superimposing and displaying the invading people, animals, etc. on the display of the VRHMD is known.

Prior Art Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2013-257716 [Patent Document 2] Japanese Patent Publication No. 2015-143976 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] When wearing a VR HMD and experiencing a VR space, the immersive experience often leads users to want to understand their surroundings, particularly what is happening outside their safe activity zone. Some of the reasons for this include the following: I don't want anyone to see me while I'm immersed in the VR space. Someone appeared in the actual space where the VR HMD wearer was. There is something that needs to be communicated to the wearer for some reason, but the wearer is immersed in the game and it is not possible to speak to them. • There's a phone call coming in. The doorbell is ringing, indicating a visitor has arrived. These are just a few examples of the various things that can be mentioned.

[0006] In conventional cases, if a person or other object enters the safe activity range of the VR HMD wearer, that person or object can be displayed by superimposing it on the VR HMD's display. However, if a person appears outside the safe activity range, the situation cannot be ascertained.

[0007] Therefore, the present invention aims to provide a VR HMD and a system equipped with a VR HMD that enable the wearer to appropriately understand their surroundings even while experiencing a VR space by determining whether the surrounding conditions are what the wearer wants to understand, even if they are outside the wearer's safe activity range, and displaying the result on the VR HMD's display even if it is outside the safe activity range. The present invention also aims to provide a display method related to this display. [Means for solving the problem]

[0008] According to a first aspect of the present invention, the following head-mounted display is provided. That is, the head-mounted display is a head-mounted display for virtual space. The head-mounted display comprises a display, a camera, a distance detection unit, an image generation unit, a storage unit, and a control unit. The display displays an image. The camera captures images of the real space. The distance detection unit detects the distance to objects present in the real space. The image generation unit generates an image to be displayed on the display. The storage unit stores the type conditions and distance conditions of the objects to be displayed. The control unit then recognizes the type of object from the image captured by the camera, extracts objects that match the type conditions and distance conditions, superimposes an image showing the extracted object onto an image in the virtual space, and displays it on the display.

[0009] According to a second aspect of the present invention, the following head-mounted display system is provided. That is, the head-mounted display system comprises a camera for capturing images of real space and a head-mounted display for virtual space. The head-mounted display comprises a display for displaying images, a distance detection unit for detecting the distance to objects present in real space, an image generation unit for generating images to be displayed on the display, a storage unit for storing the type conditions and distance conditions of the objects to be displayed, and a control unit. The control unit recognizes the type of object from the image captured by the camera, extracts objects that match the type conditions and distance conditions, superimposes images showing the extracted objects onto the image of the virtual space, and displays them on the display.

[0010] According to a third aspect of the present invention, the following display method for a head-mounted display is provided. This display method is performed using a head-mounted display for a virtual space. This method comprises a storage step of storing type conditions and distance conditions for objects to be displayed; an image generation step of generating an image of a virtual space; an image capture step of capturing the real space around the head-mounted display; a distance detection step of detecting the distance to objects existing in the real space; a recognition step of recognizing the type of object from the captured image; an extraction step of extracting objects that match the type conditions and distance conditions from the recognized objects; and an overlay display step of superimposing an image of the extracted object onto an image of the virtual space. [Effects of the Invention]

[0011] According to the present invention, even outside the wearer's safe activity range, the system provides a VR HMD and a system equipped with the VR HMD that can appropriately grasp the surrounding situation even while experiencing a VR space by determining whether the surrounding situation is one that the wearer wants to be aware of, and displaying the result on the VR HMD's display even outside the safe activity range according to the determination result. Furthermore, a display method related to this display is also provided. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of VRHMD. [Figure 2] This is a diagram used to explain the actual real-world space in which a VRHMD wearer is located. [Figure 3] This figure shows an example of a VRHMD hardware configuration. [Figure 4] This is a diagram used to explain an example of a camera configuration. [Figure 5] This is a diagram used to explain an example of a camera configuration. [Figure 6A] This is a diagram illustrating one example of a method for acquiring images of the surrounding environment. [Figure 6B] This is a diagram illustrating one example of a method for acquiring images of the surrounding environment. [Figure 7] FIG. is a diagram for explaining an example of the display of the boundary of the safe activity range. [Figure 8] A flowchart for explaining an example of an operation flow in the initial setting of the VRHMD. [Figure 9] FIG. is a diagram showing an example of a VR space image displayed on a display. [Figure 10A] FIG. is a diagram showing an example of a VR space image in which an object is superimposed and displayed. [Figure 10B] FIG. is a diagram showing an example of a VR space image in which an object is superimposed and displayed. [Figure 11] A flowchart for explaining an example of processing during the operation of the VRHMD according to the first embodiment. [Figure 12] FIG. is a diagram showing an example of a VR space image in which a virtual object representing an object is superimposed and displayed. [Figure 13] A flowchart for explaining an example of processing during the operation of the VRHMD according to the second embodiment. [Figure 14] FIG. is a diagram showing an example of the hardware configuration of the voice detection processing unit according to the third embodiment. [Figure 15] A flowchart for explaining an example of processing during the operation of the VRHMD according to the third embodiment. [Figure 16] FIG. is a diagram showing an example of boundary setting according to the fourth embodiment. [Figure 17] FIG. is a diagram showing an example of a VR space image in which a virtual object representing an object is superimposed and displayed. [Figure 18] FIG. is a diagram used for explaining an example of a method for detecting an object existing outside the visual field according to the fifth embodiment. [Figure 19] FIG. is a diagram showing an example of a VR space image in which a virtual object representing an object is superimposed and displayed. [Figure 20] A flowchart for explaining an example of processing during the operation of the VRHMD according to the fifth embodiment. [Figure 21]This is a diagram illustrating an example of a sixth embodiment using a smartphone. [Modes for carrying out the invention]

[0013] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Throughout the drawings, similar components are denoted by the same reference numerals, and redundant explanations may be omitted. According to the embodiments, an HMD (Head Mounted Display) is provided that allows for appropriate perception of the surrounding situation even outside the safety activity range. As one example, this can contribute to the United Nations' Sustainable Development Goal (SDG) 9 (Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation).

[0014] <First Embodiment> The first embodiment will be described with reference to Figures 1-11. First, an overview of the VR HMD will be described with reference to Figures 1-3. Figure 1 is an example of a VR HMD relating to one embodiment of the present invention, and shows the VR HMD in a worn state and the display inside the VR HMD. Figure 2 is a diagram used to explain the actual real space in which the wearer of the VR HMD is located.

[0015] As shown in Figure 1, the VR HMD 1 is equipped with a camera 200 and other components, and is worn on the user's head. The camera 200 captures images of the real world surrounding the wearer. Inside the VR HMD 1, there is a display 130, which displays images of the created VR space and images of the real world captured by the camera 200.

[0016] The wearer of the VRHMD1 experiences the VR space in the real world as shown in Figure 2. During the VR experience, depending on the VR content, the wearer may move in various directions, such as forward, backward, left, right, and diagonally, as indicated by arrow 8.

[0017] However, the real space in which the wearer is located may contain various objects, such as chairs (4), desks (5, 12), computers (6), telephones (11), people (20), animals (30), doors (15), windows (7), and walls (3), as shown in Figure 2. Therefore, when the wearer moves or moves their hands during a VR experience, they need to avoid these objects. An example of a safe activity range where it is possible to move and act safely without coming into contact with these objects is shown by the dotted line 10 in Figure 2. Note that during a VR experience, the display 130 shows an image of the VR space, and these objects cannot be recognized.

[0018] Next, an example of the hardware configuration of a VR HMD will be described with reference to Figure 3. As shown in Figure 3, the VR HMD 1 comprises a control circuit 104, a sensor unit 105, a communication processing unit 106, a video processing unit 107, and an audio processing unit 108. These (104-108) are connected via a data bus 103 for exchanging data. The VR HMD 1 also includes a battery 109 that provides power.

[0019] The control circuit 104 can be configured, for example, using a main processor 2, RAM (Random access memory) 141, ROM (Read-only memory) 142, and flash memory 143 for storing initial setting information, and includes a control unit and a memory unit. The main processor 2 uses the programs and data stored in the ROM 142 and flash memory 143, as well as the output data of each unit (105-108), to control the operation of the VRHMD1 and various predetermined processes related to the present invention.

[0020] The sensor unit 105 can be configured, for example, with a GPS receiving sensor 151, a geomagnetic sensor 152, a distance sensor 153, an acceleration sensor 154, a gyro sensor 155, and a temperature sensor 156, which can be used to acquire location information. This can be used to understand the wearer's condition and data such as the position, size, and temperature of surrounding objects. However, the sensors listed here are just examples, and as long as they can perform predetermined processing, the listed sensors may be omitted as appropriate, or other types of sensors may be included.

[0021] The video processing unit 107 is used to generate and display video, and can be configured, for example, with a camera 200, a VR space image generation unit 195 (virtual space image generation unit in Figure 3), a video superposition processing unit 196, and a display 130. The VR space image generation unit 195 is configured to generate images in VR space. The video superposition processing unit 196 is configured to superimpose video onto VR space.

[0022] The audio processing unit 108 can be configured, for example, using a microphone 181, a codec 182 for processing audio signals, and a speaker 183. The microphone 181 may be provided as appropriate, for example, to receive the wearer's voice. Alternatively, the microphone 181 may be provided to receive external audio when the device is being worn. The speaker 183 may be provided, for example, to be positioned close to the wearer's ear when the device is being worn.

[0023] The communication processing unit 106 can be configured, for example, using a wireless LAN interface 161 and a short-range communication interface 162. The wireless LAN interface 161 is used as a communication interface for wireless LAN communication, and the short-range communication interface 162 is used as a communication interface for short-range communication. For example, Bluetooth® can be used as the short-range communication interface.

[0024] Camera 200 captures images of the wearer's surroundings in a 360° range. An example of the configuration and operation of camera 200 will be explained using Figures 4-5, 6A, and 6B.

[0025] As shown in Figure 4, the camera 200 is equipped with two image acquisition units (201, 202) that allow images to be acquired from two locations, in front of and behind the wearer. Here, the image acquisition units (201, 202) are configured to receive light from the outside, and as an example, they can be configured to have an opening for receiving light. As shown in Figure 5, the camera 200 includes a wide-angle front lens 210 for capturing the front view, a wide-angle rear lens 220 for capturing the rear view, image sensors (211, 221) corresponding to each lens (210, 220), signal processing units (212, 222) for signal processing, and a 360° video creation unit 230 that generates a 360° surrounding image from the captured images of the front and rear. Here, if the field of view of the front and rear lenses is narrow and there is a blind spot in the shooting, and a 360° surrounding image cannot be obtained, as an example, an image is acquired by the method described below.

[0026] In other words, as indicated by arrow 8 in Figure 2 above, it is assumed that the wearer of the VR HMD 1 will move in various directions and turn their head to look around. Here, for example, if the field of view that can be captured by the front lens 210 and rear lens 220 of the camera 200 is between the dotted lines 607 and 608 and between the dotted lines 606 and 609 in Figure 6A, then the objects that can be captured will be people (700, 704) and animals (702, 703), and the desk 701 will not be captured. If the wearer moves their head in this state, as shown in Figure 6B, the objects that will be captured will be people (700, 704) and the desk 701, and the animals (702, 703) will not be captured. Here, a 360° image is obtained by combining these captured images.

[0027] Next, we will explain how to set the safe activity area. As shown above using Figure 2, the real space in which the wearer is located may contain objects such as chairs (4), desks (5, 12), computers (6), telephones (11), people (20), animals (30), doors (15), windows (7), and walls (3). In order for the wearer to move or perform actions such as moving their hands safely during the VR experience, they must avoid these objects. In the example in Figure 2, the safe activity area in which it is possible to move or perform actions such as moving one's hands without coming into contact with these objects is, for example, the area indicated by the dotted line 10 (i.e., the space on the wearer's side with the dotted line 10 as the boundary). Therefore, before starting the VR experience, the area corresponding to this dotted line 10 is set.

[0028] In this embodiment, as shown in Figure 7, the VRHMD1 overlays and displays a boundary of a safe activity range on the real-world image captured by the camera 200, which has a similar role to the dotted line 10 in Figure 2 and allows the user to avoid objects. Specifically, the VRHMD1 (1) uses the control circuit 104, sensor unit 105, and video processing unit 107 shown in Figure 3 to detect the position, size, and distance from the wearer of objects such as the chair 4, desk 5, and person 20 that exist in the real world. That is, the position, size, and distance from the wearer of objects in the 360° surrounding real-world image captured and created by the camera 200 are detected. Next, (2) the VRHMD1 automatically sets a boundary 100 on which the wearer can avoid objects based on the detected results. Finally, (3) the VRHMD1 overlays the boundary 100 on the real-world image captured by the camera 200 and displays it on the display 130. This display allows the wearer to confirm the boundary 100 of the safe activity range before starting the VR experience, enabling them to immerse themselves in the VR space with peace of mind.

[0029] In the above description, the camera 200 captures a 360° view and uses that image. However, the boundary 100 may also be set to an image that is not a 360° view, such as an image from a camera that only captures what is in front of the wearer. In that case, objects around the wearer may be detected each time during the VR space experience, and the boundary 100 that allows the wearer to avoid objects may be automatically set each time. For example, if a VR HMD is used with a camera that captures what is in front, the boundary 100 that allows the wearer to avoid objects may be set each time the head moves and the image of the real space in front changes.

[0030] Next, we will describe an example of the operation flow for setting the safety activity area boundary 100, referring to Figure 8. Figure 8 is a flowchart illustrating an example of the operation flow in the initial setup of a VR HMD.

[0031] First, the user puts on the VRHMD1. Then, the VRHMD1 begins the initial setup for experiencing the VR space (S1). This process may start automatically after the VRHMD1 is put on, or it may be started by user input using an appropriate input device.

[0032] Next, the user sets the types of objects they want to identify while experiencing the VR space, such as people, animals, or a ringing telephone (S2). The number of objects can also be set; for example, if multiple people are experiencing the VR space and the number of people exceeds the set limit, the system may choose not to identify people. It is also possible to utilize appropriate facial recognition technology to identify only specific individuals. The set information is stored in the memory unit.

[0033] Next, the wearer's surroundings are captured by camera 200, and a 360° video is created (S3). The VRHMD1 then identifies objects (obstacles) from the created video and uses the data acquired by the sensor unit 105, etc., to detect the position, distance from the wearer, size, etc. of the obstacles, and stores the data of the identified obstacles, their position, distance from the wearer, size, etc. (S4).

[0034] VRHMD1 acquires relative positional information with respect to objects based on the positions and distances of objects in the real space, such as chair 4, desk 5, person 20, and wall 3, obtained in S4 (S5). Then, based on the data acquired in S4 and S5, VRHMD1 automatically sets a boundary 100 that can avoid contact with objects (obstacles) (S6).

[0035] The VRHMD1 overlays the set boundary 100 onto the image of the real space captured by the camera 200 and displays it on the display 130. The wearer then looks at the image output to the display 130 to confirm whether the boundary 100 is appropriate (S7).

[0036] If the confirmation result in S7 is OK, the VRHMD1 stores the position information of the boundary 100 and creates a VR space image (S8). The created VR space image is then displayed on the display 130 as shown in Figure 9. On the other hand, if the confirmation result in S7 is NG, the process returns to S6, and the VRHMD1 resets the boundary 100. The confirmation result may, for example, be input by the wearer via an appropriate input device. The VRHMD1 may also perform a process to determine whether the confirmation result is OK or NG after a predetermined time has elapsed.

[0037] Then, after the VR space image is created, the initial setup is completed (S9).

[0038] Next, an example of a display method relating to the present invention will be described. One of the objectives of the present invention is to enable the wearer to understand the surrounding environment, particularly the environment outside the boundary 100 of the safe activity area, only when necessary, without compromising the sense of immersion during the VR space experience.

[0039] Figures 10A and 10B show an example of a display mode according to the first embodiment of the present invention. Figures 10A and 10B show an example in which a person 20 or animal 30 located outside the boundary 100 of the safe operating range described in Figure 7 is identified, and the person 20 or animal 30 is superimposed on the VR space image shown in Figure 9 and displayed on the display 130.

[0040] Camera 200 captures 360° surrounding images as shown in Figures 6A and 6B even during the VR space experience. If VRHMD1 identifies an object set in the initial S2 from the captured surrounding images, such as chair 4, desk (5,12), computer 6, telephone 11, person 20, animal 30, door 15, window 7, or wall 3 that is outside the boundary 100, it will superimpose the captured image of person 20, animal 30, etc. onto the VR space image and display it. If an object that outputs sound, such as a telephone ringing, is set, a microphone 181 or similar device that inputs external sound may be used to identify the object.

[0041] Figure 10A shows the display when both a person and an animal are set in the initial S2 configuration. Although both person 20 and animal 30 exist outside the boundary 100, they are superimposed in the VR space because they are set in S2. On the other hand, Figure 10B shows the display when only a person is set in the initial S2 configuration. Although both person 20 and animal 30 exist outside the boundary 100, only person 20 is set in S2, so only person 20 is superimposed in the VR space and animal 30 is not displayed. In this way, only the objects that the wearer has initially set to understand the situation can be displayed.

[0042] Furthermore, if a new object appears inside boundary 100 in the surrounding image that was not present during the initial setup, it may interfere with safe movement and operation. For this reason, VRHMD1 displays the captured image of the new object superimposed on the VR space image, regardless of the type of object.

[0043] As explained above, when the VRHMD1 identifies that an object set outside the boundary 100 has newly appeared, it superimposes the image of that object onto the VR space image and displays it. This makes it possible for the wearer to grasp external situations that they may want to be aware of. On the other hand, if it identifies an object that has not been set, it will not be displayed unless it interferes with safe operation, so as not to impair the sense of immersion in the VR space. Thus, according to this embodiment, a VRHMD is provided that can display information with an appropriate balance between awareness of the surrounding situation and a sense of immersion, which are in a trade-off relationship.

[0044] Next, we will explain the operation flowchart according to the first embodiment using Figure 11. Figure 11 is a flowchart illustrating an example of processing during the operation of a VR HMD.

[0045] When the VR space experience begins (S10), the VR HMD1 generates a VR space image using the VR space image generation unit 195 of the video processing unit 107 (S11). Here, the VR space image is generated using the same generation method as described in S8 above.

[0046] When using the VRHMD1, the camera 200 captures images of the wearer's surroundings and creates a 360° ambient image (S12). The VRHMD1 then detects objects from the created 360° ambient image (S13). The VRHMD1 uses the sensor functions of the sensor unit 105 and other sensors as appropriate to determine the position of the detected object and compares it with the data detected in S4 above. If a new object is detected, it is stored in memory, and if the object is moving, its direction is detected (S14). The direction of movement of an object can be detected, for example, using captured images (for example, by determining the direction of movement of the object using images taken at short time intervals).

[0047] VRHMD1 identifies whether the object detected in S14 and the moving object are located outside or inside boundary 100 (S15). It is assumed that the wearer is located inside boundary 100.

[0048] If in S15 the VRHMD1 identifies that an object is located outside the boundary 100, the VRHMD1 determines the type of object. The VRHMD1 determines, for example, that it is a person, an animal, a desk, a chair, etc. (S16). The VRHMD1 can determine the type of object using, for example, a known image matching technique. Alternatively, the VRHMD1 may determine the type of object by using an appropriate matching technique based on the sound from the input object.

[0049] In S16, it is determined whether the type determined in S2 matches the type set in advance in Figure 8. For example, if "person" and "animal" are set in S2, VRHMD1 extracts "person" and "animal" (S17).

[0050] If an object set in S2 is identified in S17 (YES), or if a moving object is detected in S14 (YES), VRHMD1 acquires (extracts) images of those objects (S18). Then, VRHMD1 superimposes the images acquired (extracted) in S18 onto the VR space image (S20). VRHMD1 also displays the superimposed image on the display 130 in S20 (S21). After the display in S21, the process returns to S11. On the other hand, if no object is extracted in S2 in S17 (NO), VRHMD1 displays the VR space image generated in S11 directly on the display 130 (S21). Also, if an object exists inside the boundary 100 in S15, and that object is a new object or a moving object detected in S14, VRHMD1 extracts images of those objects (S19). Then, VRHMD1 superimposes the image obtained by extraction in S19 onto the VR space image (S20). VRHMD1 also displays the superimposed image on the display 130 in S20 (S21). Note that if the process in S19 is performed, there is a high probability that the wearer will come into contact with an object. Therefore, in this embodiment, VRHMD1 either superimposes the image from S19 onto the central part of the VR space image, or displays the image from S19 and stops outputting the VR space image.

[0051] As explained above, when an object set outside the boundary 100 is identified as having newly appeared, the captured image of that object (specifically, an image extracted from the video captured by camera 200, or an image with the object's outline extracted) is superimposed on the VR space image and displayed. This makes it possible for the wearer to grasp external situations that they may want to be aware of. On the other hand, if an object that has not been set is identified, it is not displayed unless it interferes with safe operation, so as not to impair the sense of immersion in the VR space. Thus, according to this embodiment, a VR HMD is provided that can display information with an appropriate balance between awareness of the surrounding environment and a sense of immersion, which are in a trade-off relationship.

[0052] <Second Embodiment> Next, a second embodiment will be described with reference to Figures 12-13. Functions similar to those in other embodiments are denoted by the same reference numerals, and their descriptions may be omitted. In the second embodiment, if the object to be grasped, set in S2 of Figure 8, is located outside the boundary 100 while experiencing the VR space, the VRHMD1 replaces that object with a virtual object. The VRHMD1 then superimposes the replaced virtual object onto the top, bottom, left, and right edges of the VR space image, aligning it with its actual location, and displays it on the display 130.

[0053] In the second embodiment, first, the camera 200 creates a 360° surrounding image, and the VR HMD 1 identifies objects from the 360° image. The VR HMD 1 detects objects that are located outside the boundary 100 and match the object to be identified in S2 of Figure 8 (for example, a person 20 or an animal 30), and identifies whether the object is located in front of, behind, to the right, to the left, or diagonally to the wearer. The VR HMD 1 then replaces the detected object with a virtual object and displays it in a direction aligned with the wearer's position.

[0054] Referring to Figure 12, an example of how virtual objects are displayed in this embodiment will be described. As shown in Figure 12, the VRHMD1 displays virtual objects of objects superimposed on the dotted line frames 111, 112, 113, and 114 at the edges of the VR space image. By displaying objects as virtual objects at the edges of the VR space image in this way, the wearer can grasp their surroundings without compromising their sense of immersion in the VR space.

[0055] Here, Figure 12 shows the display of the VR HMD1 in the situation shown in Figure 2. In this example, since person 20 is in front of the wearer, the virtual object of person 20 in front is displayed within the upper dotted frame 111. Also, since animal 30 is to the right of the wearer, the virtual object of animal 30 on the right is displayed within the right dotted frame 113. In this way, virtual objects are displayed according to their orientation relative to the wearer's position.

[0056] Next, the operation flowchart of the second embodiment will be explained using Figure 13. Note that functions similar to those in other embodiments may be denoted by the same reference numerals, and their descriptions may be omitted.

[0057] First, when the VR space experience begins (S10), the VR HMD 1 generates a VR space image using the VR space image generation unit 195 of the video processing unit 107 (S11). Then, the camera 200 captures the area around the wearer and creates a 360° surrounding image (S12), and the VR HMD 1 detects objects from the created 360° surrounding image (S13).

[0058] The VRHMD1 uses the appropriate sensor functions, such as the sensor unit 105, to determine the position of the detected object, and also determines whether it is in front of, behind, to the right, to the left, or diagonally to the wearer. Furthermore, the VRHMD1 compares this with the data detected in S2 of Figure 8, and if a new object is detected, it stores the object, and if the object is moving, it detects the direction of its movement. (S14)

[0059] The orientation of an object may be determined, for example, by the following method: VRHMD1 processes objects located in the left-right center of the captured image as being in the same direction as the camera 200 (for example, forward or backward), and processes objects located at the left or right edges of the captured image as being in the lateral direction (for example, left or right). Then, VRHMD1 processes objects located in between these directions as being in the diagonal direction.

[0060] VRHMD1 identifies whether the object detected in S14 and the moving object are located outside or inside boundary 100 (S15). Note that the wearer is located inside boundary 100. If the object is located outside boundary 100 in S15, VRHMD1 determines the type of object (S16).

[0061] VRHMD1 determines whether the type determined in S16 matches the type previously set in S2 in Figure 8. For example, if "person" and "animal" are set in S2, it extracts "person" and "animal" (S17). If an object set in S2 is extracted in S17 (YES), or if a new object is detected in S14 (YES), VRHMD1 replaces those objects with virtual objects. Here, VRHMD1 can replace, for example, a person with a human-shaped object, or an animal with an animal-shaped object (S31). Note that the replacement method is not limited to the method described above. Also, the object can be any shape that allows for object identification.

[0062] As shown in Figure 12, the VRHMD1 superimposes the image replaced with a virtual object in S31 onto the dotted line frame (111-114) of the VR space image, aligning it with the direction to the wearer detected in S14 (S32). Then, the VRHMD1 displays the superimposed image on the display 130 in S32 (S21). After the display in S21, the process returns to S11.

[0063] On the other hand, if the object set in S2 is not extracted (does not match) in S17 (NO), the VRHMD1 displays the VR space image generated in S11 on the display 130 as is (S21). Also, if an object is located inside the boundary 100 in S15 and that object is a new object or moving object detected in S14, the VRHMD1 extracts images of those objects (S19). Since there is a high possibility that the wearer will come into contact with the object in the image of S19, in this embodiment, the VRHMD1 performs a process to interrupt immersion in the VR space by superimposing the image of S19 onto a part of the VR space image that is not the dotted line frame 111 (for example, the central part), or by displaying the real space along with the image of S19 instead of the VR space image.

[0064] As explained above, when an object set outside boundary 100 is identified as having newly appeared, that object is superimposed on the VR space image as a virtual object and displayed. This makes it possible for the wearer to grasp external situations that they may want to be aware of. On the other hand, if an object that has not been set is identified, it is not displayed unless it interferes with safe operation, so the sense of immersion in the VR space is not impaired. Thus, according to this embodiment, a VR HMD is provided that can display information with an appropriate balance between awareness of the surrounding environment and a sense of immersion, which are in a trade-off relationship.

[0065] <Third Embodiment> Next, a third embodiment will be described with reference to Figures 14 and 15. Functions similar to those in other embodiments are denoted by the same reference numerals, and their descriptions may be omitted. In the third embodiment, the VRHMD1 detects an object that is emitting sound, such as a telephone, and indicates that sound is being emitted.

[0066] As already explained, the VRHMD1 can use the 360° surrounding image captured by camera 200 to determine the presence and location of objects such as telephones. On the other hand, in order to detect whether the detected objects are emitting sound (for example, telephone ringtones or human voices), sound detection processing is required.

[0067] Figure 14 shows an example of the hardware configuration of the voice detection processing unit 300 in this embodiment. The voice detection processing unit 300 includes the microphone 181 and codec 182 (voice processing unit) of the voice processing unit 108 shown in Figure 3. The microphone 181 consists of a left microphone 301, a left microphone amplifier 311 (microphone amplifier 311), a right microphone 302, and a right microphone amplifier 321 (microphone amplifier 321). The codec 182 consists of a left signal processing unit 312, a right signal processing unit 322, and a 360° sound image creation unit 330. The signal processing units (312, 322) perform signal processing on the sound collected by the two left and right microphones (301, 302) to generate a digital signal. The 360° sound image creation unit 330 creates a sound image and generates data to determine the direction of sound generation and the type of sound (e.g., telephone ringtone, human voice).

[0068] An example of the operation flow of the third embodiment will be explained using Figure 15. Figure 15 is a flowchart illustrating an example of processing during the operation of the VR HMD. In the third embodiment, the VR HMD 1 performs sound detection to prevent misidentification between mannequins and people, and between stuffed animals and animals. Functions similar to those in other embodiments are denoted by the same reference numerals and their descriptions may be omitted.

[0069] When the VR space experience begins (S10), the VR HMD 1 generates a VR space image (S11). Camera 200 captures the wearer's surroundings, and the VR HMD 1 creates a 360° surrounding image (S12). The VR HMD 1 detects objects from the created 360° surrounding image (S13).

[0070] VRHMD1 measures the temperature of the object detected in S13 using the temperature sensor 156 and compares it with the data detected in S4 in Figure 8 to distinguish between a mannequin and a person with body temperature, and between a stuffed animal and an animal with body temperature (S41). Note that if the VRHMD is not equipped with a temperature sensor, S41 is skipped.

[0071] VRHMD1 identifies the position of the object detected in S13 and determines whether it is in front of, behind, to the right, to the left, or diagonally to the wearer. Furthermore, if a new object is detected as a result of comparing it with the data detected in S4 in Figure 8, VRHMD1 stores the object and, if the object is moving, detects the direction of its movement (S14).

[0072] The VRHMD1 detects the location of the sound source based on the data from the sound detection processing unit 300, compares the output data from S14 with the data detected in S4 in Figure 8, and identifies the object that is emitting the sound. In cases such as when an emergency bell is ringing, the location of the sound source may be recognized as simply a wall. In this case, the VRHMD1 determines that there is no object corresponding to the sound source (S42). The video from the camera 200 may also be used to identify the object emitting the sound.

[0073] VRHMD1 identifies whether the output data from S42 relates to an object located outside boundary 100 or an object located inside boundary 100 (S43). In this case, the wearer is located inside boundary 100.

[0074] In S43, VRHMD1 determines the type of object and sound if the presence of an object or the generation of a sound is outside the boundary 100. VRHMD1 determines, for example, that it is a ringing telephone, the voice of a person calling, a doorbell announcing a visitor, or an emergency bell (S44).

[0075] VRHMD1 determines whether the type determined in S24 matches the type pre-set in S2 in Figure 8. For example, VRHMD1 extracts telephones with the ringtone set in S2, the voice of the person making the call, and chimes that announce visitors (S17).

[0076] In S17, if an object set in S2 is extracted, and in S14, if a moving object is detected (YES), VRHMD1 replaces those objects and sounds with virtual objects. For example, if a telephone is ringing, VRHMD1 should replace it with an object shaped like a ringing telephone. Similarly, if a person is making a call, it can be replaced with an object shaped like a person making a call; if a doorbell is ringing, it can be replaced with an object shaped like a doorbell; and if an emergency bell is ringing, it can be replaced with a virtual object of an emergency bell (S45).

[0077] In S45, the VRHMD1 superimposes the image replaced with a virtual object onto the dotted frame 111 portion of the VR space image, aligning it with the direction of the wearer detected in S14. If in S44 it is determined that no corresponding object is found, such as when an emergency bell is sounding, the VRHMD1 may switch from the VR space image to the real space image. Furthermore, it may superimpose a virtual object of the emergency bell onto the real space image to indicate danger (S32). Note that the operations from S32 onward are the same as the operation flowchart in Figure 13 described above, so the explanation is omitted.

[0078] As explained above, even if no new objects appear, if a sound that should alert the wearer is generated, a virtual object representing the generated sound is superimposed on the VR space image and displayed. This makes it possible for the wearer to grasp external situations that they may want to be aware of. On the other hand, if the sound is identified as of low importance, it is not displayed, so the sense of immersion in the VR space is not impaired. Thus, this embodiment provides a VR HMD that can display information with an appropriate balance between awareness of the surrounding environment and immersion, which are in a trade-off relationship.

[0079] <Fourth Embodiment> Next, the fourth embodiment will be described with reference to Figures 16 and 17. Functions similar to those in other embodiments are denoted by the same reference numerals, and their descriptions may be omitted. Figure 16 shows that an additional boundary 1000 is provided outside the boundary 100 of the safe operating range. In Figure 16, the area inside boundary 100 is designated as the first area, the area between boundary 100 and boundary 1000 is designated as the second area, and the area outside boundary 1000 is designated as the third area, with the wearer of the VR HMD 1 facing in the direction of arrow 70. Figure 16 also shows an example where a person 299, an animal 399, and a ringing telephone 199 are present in the second area, and a person 1200 and an animal 1300 are present in the third area. Then, when setting the boundaries in S6 of the flowchart shown in Figure 8, two boundaries, boundary 100 and boundary 1000, are set. In other words, in this embodiment, the VRHMD1 sets a boundary 100 for the distance at which objects are displayed regardless of type conditions (first distance), and a boundary 1000 for the distance at which objects are displayed only when the type conditions are met (second distance). It goes without saying that this explanation is just one example, and there is no limit to the number of boundaries that can be set.

[0080] In the fourth embodiment, the VRHMD1 determines whether to display the objects present in the first, second, and third areas 3, as defined in S2 of Figure 8, and overlays them onto the VR space image. Figure 17 shows an example of how the VRHMD1 displays when objects are present, as shown in Figure 16.

[0081] In Figure 17, similar to Figure 12, the VR HMD1, while experiencing the VR space, replaces the object to be identified (set in S2 of Figure 8) with a virtual object and displays it superimposed on the dotted lines 111, 112, 113, and 114 at the top, bottom, left, and right edges of the VR space image, according to its actual location. As illustrated in Figure 17, the person 299 located behind is displayed with its object superimposed on the lower dotted line 114. Similarly, the animal 399 located to the left is displayed with its object superimposed on the left dotted line 112, the person 1200 and the ringing telephone 199 located to the right are displayed with their objects superimposed on the right dotted line 113, and the animal 1300 located in front is displayed with its object superimposed on the upper dotted line 111. Furthermore, by making the size of the person 299 and animal 1300 objects in the second area larger than the size of the person 1200 and animal 1300 objects in the third area and superimposing them, the wearer can recognize the area where the objects are located.

[0082] In this process, VRHMD1 determines in S16 and S44 of the aforementioned operation flowchart whether the object or sound is located in the second area or the third area. Then, in S32, VRHMD1 changes the size of the superimposed object according to the area in which it is located.

[0083] Furthermore, the setting for detecting objects in the second area may be limited to objects that emit sound. By setting it this way, when experiencing VR space in a large space such as a gymnasium, the space that the wearer wants to perceive can be limited to a certain area around the wearer (for example, a few meters). In addition, it has the advantage of being able to perceive only emergency bells and emergency announcements in areas beyond that certain range, such as outside the gymnasium in the event of an emergency such as a fire.

[0084] As explained above, multiple areas are defined by boundaries, and virtual objects corresponding to the areas where objects exist are superimposed and displayed on the VR space image. This makes it possible to recognize objects according to their distance from the wearer. On the other hand, if there is no impediment to safe operation, the display is not performed, and if the distance is far, the virtual objects can be displayed inconspicuously, so as not to impair the sense of immersion in the VR space. Thus, according to this embodiment, a VR HMD is provided that can perform display with an appropriate balance between the trade-off relationship between awareness of the surrounding environment and a sense of immersion.

[0085] <Fifth Embodiment> Next, the fifth embodiment will be described with reference to Figures 18-20. Functions similar to those in other embodiments are denoted by the same reference numerals, and their descriptions may be omitted. In the fifth embodiment, an example of processing using data obtained from communication will be described.

[0086] Many devices have short-range communication interfaces (wireless communication devices), such as smartphones. These interfaces use radio waves and are designed for short-range use, with a communication range of up to about 10 meters. These interfaces periodically transmit ID information and, because they use radio waves, can be detected even if they are in places that cannot be seen, such as behind a wall.

[0087] Therefore, for example, as shown in Figure 18, the VRHMD1 can detect a smartphone 110 with a near-field communication interface that is located outside the door 15. Here, Figure 18 shows a situation in which a person possessing a smartphone 110 with a near-field communication interface is located outside the door 15. Then, as shown in Figure 19, the VRHMD1 can display on the display 130 that the person possessing this smartphone 110 is located around the wearer experiencing the VR space by superimposing the smartphone object 110 onto the dotted line frame 112 in the lower left of the VR space image.

[0088] An example of the operation flow of the fifth embodiment will be explained using Figure 20. Figure 20 is a flowchart illustrating an example of processing during the operation of a VR HMD. Note that functions similar to those in other embodiments are denoted by the same reference numerals, and their descriptions may be omitted.

[0089] When the VR space experience begins (S10), the VR HMD1 generates a VR space image (S11).

[0090] The short-range communication interface periodically transmits ID information. Therefore, VRHMD1 detects the short-range communication interface by acquiring radio waves from it (S51). VRHMD1 also detects ID information from the acquired radio waves (S52).

[0091] Furthermore, as an example, VRHMD1 can detect (estimate) the distance of a device equipped with a short-range communication interface from the strength of the acquired radio waves (S53). Alternatively, VRHMD1 may also detect (estimate) the distance of a device equipped with a short-range communication interface from the delay time in communication. Additionally, as an example, position detection is possible by using a method capable of detecting direction, such as UWB (Ultra Wide Band).

[0092] VRHMD1 determines in S53 whether the device's position is outside or inside boundary 100 (S15). If the device is inside boundary 100 in S15, the process proceeds to S21. Note that VRHMD1 may not only determine based on distance, but also detect whether the device is approaching or moving away and take that information into account when making its determination. For example, even if the device is outside boundary 100 in terms of distance, if it is moving away, it may be determined that there is little need to notify the wearer and the process proceeds to S21.

[0093] If the device is located outside boundary 100 in S15, it is determined whether the detected ID information matches the device to be monitored, as set in S2 in Figure 8 (S17). For the settings in S2, the user can, for example, select and register devices to be monitored for proximity from a list of previously detected devices equipped with a short-range communication interface.

[0094] If a device is identified in S17 (Yes), VRHMD1 replaces the identified device with a virtual object (S54). Then, as shown in Figure 20, VRHMD1 superimposes the object from S54 onto the dotted frame 112 in the lower left of the VR space image (S32).

[0095] In this embodiment, an example was described in which the object of S54 is superimposed on the portion of the dotted line frame 112. However, the display method is not limited to this example, and for example, the position of the dotted line frame to be displayed can be changed as appropriate. Also, if the direction of the device can be determined, the display may be oriented toward the wearer, as described in Figure 12 above. Furthermore, displays may be classified by type of device and grouped by type of device.

[0096] As explained above, the system uses a short-range communication interface to detect target devices and displays virtual objects superimposed on the VR space image. This makes it possible to recognize target devices even in locations that cannot be captured by a camera. On the other hand, devices that are not registered as targets are not displayed, so the sense of immersion in the VR space is not impaired. Thus, this embodiment provides a VR HMD that can display information with an appropriate balance between the trade-off relationship between understanding the surrounding environment and immersion.

[0097] <Sixth Embodiment> Next, the sixth embodiment will be described with reference to Figure 21. Functions similar to those in other embodiments will be denoted by the same reference numerals, and their descriptions may be omitted. The sixth embodiment will describe an example of a VR HMD using a smartphone.

[0098] As shown in Figure 21, the VRHMD1 may be a VR goggle 90 with a smartphone 110 attached. The VRHMD1 may also perform similar processing using the camera 200 on the back of the smartphone 110, the distance sensor 153, the temperature sensor 156, and the display 130 on the front of the smartphone 110.

[0099] Here, the VR goggles 90 are configured in a suitable manner to which a smartphone 110 can be attached. For example, the VR goggles 90 may be smartphone goggles into which the user inserts the smartphone 110 to attach it. Alternatively, the VR goggles 90 may be smartphone goggles into which the user inserts the smartphone 110 to attach it. Here, "smartphone" is an abbreviation for smartphone.

[0100] As described above, a VR HMD is provided that recognizes the type of object from an image captured by camera 200, extracts objects that match the type and distance conditions, superimposes images of the extracted objects onto a VR space image, and displays them on display 130. As an example, a display method for a head-mounted display is also provided, comprising a storage step (S2) for storing the type and distance conditions of the objects to be displayed, an image generation step (S11) for generating an image of a virtual space, an image capture step (S12) for capturing the real space around the head-mounted display, a distance detection step (S14) for detecting the distance to objects existing in the real space, a recognition step (S16) for recognizing the type of object from the captured image, an extraction step (S17, S18) for extracting objects that match the type and distance conditions from the recognized objects, and an overlay display step (S20, S21) for superimposing images of the extracted objects onto a virtual space image.

[0101] In this way, even outside the safe activity range of the VR HMD wearer, surrounding conditions such as people, equipment, and sounds can be detected, and it can be determined whether it is desirable to inform the wearer of these conditions. If it is determined that notification is necessary, the detected conditions are displayed on the display by superimposing the detected situation onto the VR space image, such as a captured image of the detected object, a virtual object representing the object, or an object indicating the direction of the object's presence. This makes it possible for the wearer to grasp external conditions that they may want to be aware of. On the other hand, if an object that has not been set is identified, it will not be displayed unless it interferes with safe operation, so the sense of immersion in the VR space is not impaired. Therefore, according to the present invention, it is possible to perform a display that strikes an appropriate balance between awareness of the surrounding conditions and immersion, which are in a trade-off relationship.

[0102] While embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above embodiments, and various modifications are conceivable. For example, the embodiments described above are described in detail for the purpose of explaining the present invention in an easy-to-understand manner, and are not necessarily limited to those comprising all the described configurations. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. In addition, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0103] It is sufficient that the program can perform the specified processing. For example, the programs used in each processing example may be independent programs, or multiple programs may constitute a single application program. Furthermore, the order in which each processing step is performed may be changed.

[0104] The functions of the present invention described above may be implemented in hardware, in whole or in part, for example, by designing them as an integrated circuit. Alternatively, they may be implemented in software by a microprocessor unit, CPU, etc., interpreting and executing an operating program that realizes each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used in combination. In addition, some or all of each function may be implemented on a server. The server only needs to be able to execute functions in cooperation with other components via communication, and its form is not limited, for example, a local server, cloud server, edge server, network service, etc. Information such as programs, tables, files that realize each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or it may be stored on a device on a communication network.

[0105] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0106] In VRHMD1, the camera positions are not limited to the examples described above. Furthermore, the number and structure of the 200 cameras are not limited to the examples described above and may be changed as appropriate.

[0107] In the environment where the VRHMD1 is used, a suitable camera capable of communicating with the VRHMD1 may be installed, and the VRHMD1 may perform processing based on the images captured and acquired via communication from the camera. In other words, a system comprising a camera and the VRHMD1 may be provided.

[0108] Furthermore, this system may use (operate) multiple VR HMD1s with a single camera. Therefore, it is possible to easily operate the system by, for example, installing one or a few cameras to provide an overview of the entire environment.

[0109] Here, VRHMD1 uses the image acquired by the camera to determine whether it is the object set in S2. If it determines that it is the object set in S2, VRHMD1 can overlay and display the image of the object captured by the camera. The object in the image acquired by the camera, or the virtual object that replaces the object, may be overlaid at a predetermined appropriate position (for example, the edge of the display 130) or a specific position. Furthermore, when multiple cameras are installed and images of objects are overlaid, the object or virtual object acquired from any one of the cameras may be overlaid.

[0110] In S2, objects that are not to be superimposed are set, and the memory unit may store information indicating the type of object that is not to be displayed. The VRHMD1 may then perform a process of not displaying the object identified from this information. In this way, by setting objects that are not to be superimposed, the wearer can immerse themselves in the VR space without being aware of those objects. For example, by hiding home appliances such as robotic vacuum cleaners, the wearer can immerse themselves in the VR space without being aware of those appliances, even when those appliances are being used.

[0111] For example, VRHMD1 may acquire data from the sensor unit 105 and process it depending on the situation. VRHMD1 may, for example, detect tilt using the accelerometer 154 or gyroscope 155 and perform processing that corrects for the effect of tilt.

[0112] The battery 109 may be connected to the data bus 103 to display information about the battery (for example, the current charge level). The VRHMD1 may then display the information from the battery 109 on the display 130. [Explanation of symbols]

[0113] 1: VR HMD 104: Control Circuit 105: Sensor section 106: Communication Processing Unit 107: Video Processing Section 108: Audio Processing Unit 130: Display 200: Camera

Claims

1. A head-mounted display for virtual space, A display that shows images, A camera that photographs real space, A distance detection unit that detects the distance to an object in real space, A video generation unit that generates images to be displayed on the aforementioned display, A storage unit that stores the type and distance conditions of the object to be displayed, It comprises a control unit and, The control unit, The type of object is recognized from the image captured by the aforementioned camera. Extract objects that match the type and distance conditions. The image representing the extracted object is superimposed onto the image in the virtual space and displayed on the display. A head-mounted display characterized by the following features.

2. A head-mounted display according to claim 1, The video showing the extracted object is either a video cropped from the video captured by the camera, or a video in which the outline of the object is extracted from the video captured by the camera. A head-mounted display characterized by the following features.

3. A head-mounted display according to claim 1, The image showing the extracted object is a virtual object indicating the type of the object. A head-mounted display characterized by the following features.

4. A head-mounted display according to claim 1, The aforementioned storage unit is Regardless of the aforementioned type conditions, a first distance for displaying an object and a second distance for displaying an object that meets the aforementioned type conditions are stored as conditions. The control unit, Objects that meet the type and distance conditions are extracted, and, Regardless of the type, extract objects that satisfy the first distance condition described above. A head-mounted display characterized by the following features.

5. A head-mounted display according to claim 1, The distance detection unit, It includes a microphone for collecting ambient sounds and an audio processing device that creates ambient sound image data used to identify the type and location of sound sources based on the collected sounds. The control unit, From the aforementioned data, the type of sound source is recognized. Extract objects that match the type and distance conditions. The image representing the extracted object is superimposed onto the image in the virtual space and displayed on the display. A head-mounted display characterized by the following features.

6. A head-mounted display according to claim 1, The distance detection unit, Includes wireless communication interface, The aforementioned storage unit is The system stores the identification number information of wireless communication devices as a condition for the type of object to be displayed. The control unit, The distance to the wireless communication device is estimated from the received radio wave strength or communication delay time of the wireless communication interface. From the wireless communication devices connected via the wireless interface, wireless communication devices that match the type and distance conditions based on the identification number information are extracted as objects. A virtual object image representing the extracted object is superimposed onto the image in the virtual space and displayed on the display. A head-mounted display characterized by the following features.

7. A head-mounted display according to claim 1, The aforementioned storage unit is It stores information indicating the type of object that will not be displayed. The control unit, The object identified from the aforementioned information will not be displayed. A head-mounted display characterized by the following features.

8. A head-mounted display system comprising a camera for capturing images of real space and a head-mounted display for virtual space, The aforementioned head-mounted display is A display that shows images, A distance detection unit that detects the distance to an object in real space, A video generation unit that generates images to be displayed on the aforementioned display, A storage unit that stores the type and distance conditions of the object to be displayed, It comprises a control unit and, The control unit, The type of object is recognized from the image captured by the aforementioned camera. Extract objects that match the type and distance conditions. The image representing the extracted object is superimposed onto the image in the virtual space and displayed on the display. A head-mounted display system characterized by the following features.

9. A head-mounted display system according to claim 8, The video showing the extracted object is either a video cropped from the video captured by the camera, or a video in which the outline of the object is extracted from the video captured by the camera. A head-mounted display system characterized by the following features.

10. A head-mounted display system according to claim 8, The image showing the extracted object is a virtual object indicating the type of the object. A head-mounted display system characterized by the following features.

11. A head-mounted display system according to claim 8, The aforementioned storage unit is Regardless of the aforementioned type conditions, a first distance for displaying an object and a second distance for displaying an object that meets the aforementioned type conditions are stored as conditions. The control unit, Objects that meet the type and distance conditions are extracted, and, Regardless of the type, extract objects that satisfy the first distance condition described above. A head-mounted display system characterized by the following features.

12. A head-mounted display system according to claim 8, The distance detection unit, It includes a microphone for collecting ambient sounds and an audio processing device that creates ambient sound image data used to identify the type and location of sound sources based on the collected sounds. The control unit, From the aforementioned data, the type of sound source is recognized. Extract objects that match the type and distance conditions. The image representing the extracted object is superimposed onto the image in the virtual space and displayed on the display. A head-mounted display system characterized by the following features.

13. A head-mounted display system according to claim 8, The distance detection unit, Includes wireless communication interface, The aforementioned storage unit is The system stores the identification number information of wireless communication devices as a condition for the type of object to be displayed. The control unit, The distance to the wireless communication device is estimated from the received radio wave strength or communication delay time of the wireless communication interface. From the wireless communication devices connected via the wireless interface, wireless communication devices that match the type and distance conditions based on the identification number information are extracted as objects. A virtual object image representing the extracted object is superimposed onto the image in the virtual space and displayed on the display. A head-mounted display system characterized by the following features.

14. A head-mounted display system according to claim 8, The aforementioned storage unit is It stores information indicating the type of object that will not be displayed. The control unit, The object identified from the aforementioned information will not be displayed. A head-mounted display system characterized by the following features.

15. A method for displaying information using a head-mounted display for virtual space, A storage step for storing the type and distance conditions of the object to be displayed, A video generation step that generates an image that renders a virtual space, A shooting step that captures the real space around the head-mounted display, A distance detection step for detecting the distance to an object existing in the real space, A recognition step that recognizes the type of object from the captured image, An extraction step in which objects matching type conditions and distance conditions are extracted from the recognized objects, The system includes a superimposed display step, which involves superimposing an image showing the extracted object onto an image in a virtual space. A display method for a head-mounted display characterized by the following features.

Images