Device for displaying a virtual room and camera images

Abstract

Device (10) comprising: - a control device (12) for generating graphic data (G) representing a virtual space (28), - a display unit (14) to be worn on the head (20) by a user (18), which is designed to shield the eyes of the user (18) from light from an environment (24) and to display the virtual space (28) into a field of vision (V) of the user (18) by displaying the graphic data (G), - a localization device (16) designed to detect the spatial orientation (P) of the display unit (14), wherein the display unit (14) has a camera (34) designed to generate image data (C) of the environment (24), which represents the real environment in which the user is located while using the display unit, and the control device (12) is designed to output the image data (C) to the display unit (14) depending on the detected spatial orientation (P), so that the display unit (14), in the case of a certain orientation of the display unit, displays the environment (24), i.e., the real environment in which the user is located, at least in a partial area (46) of the field of view (V) by displaying the image data (C), characterized in that a) the control device (12) is designed to display the image data (C) by means of the display unit (14) during a translational movement of the display unit (14) caused by a movement of the user (18), and / or b) the control device (12) is designed to display the image data (C) by means of the display unit (14) when a change in perspective from which the virtual space (28) is represented by the graphic data (G) occurs, independent of any body movement of the user (18).

Description

[0001] The invention relates to a device by which a virtual space can be displayed to a user. For this purpose, the user wears a display unit on their head that shields their eyes from light and projects the virtual space into their field of vision. The invention also includes a corresponding method for projecting the virtual space into the user's field of vision.

[0002] Virtual reality (VR) refers to the representation of a virtual space using a head-mounted display (HMD), where the user's perspective within the virtual space is adjusted based on their head movements. This allows the user to "look around" within the virtual space.

[0003] In virtual reality, users may encounter the problem that their spatial orientation is impaired due to the lack of direct visual contact with objects in the real environment. In particular, many first-time VR users find it very unusual and uncomfortable not to be able to see their own body and limbs when looking down at themselves.

[0004] Virtual reality should be distinguished from augmented reality (AR), in which the user can still view their surroundings through glasses. Only additional graphical content is superimposed onto their field of vision. The user can therefore orient themselves using objects in the real environment and also view their own limbs. Data glasses for displaying augmented reality are known, for example, from US 2013 / 0050432A1, US 2013 / 0328925A1, and US 2013 / 0093788A1.

[0005] For the stereoscopic representation of graphic data that represent a virtual space, an autostereoscopic image reproduction device for generating a floating real stereo image is known from DE 10 2007 006 038 B3.

[0006] US patent 2010 / 0159434A1 describes a mixed simulation system for interaction with both virtual images and real objects. The graphic data representing virtual objects is displayed on a tablet PC, which the user can hold in front of a real object to get the impression that the real object behaves according to a result of the simulation.

[0007] US patent 2008 / 0030429A1 describes a virtual reality system in which a user can see their own hand in virtual space. This is achieved by filming the user's hand, removing all video data representing the user's real-world surroundings from the resulting video image, and then superimposing this video data of the user's hand into the virtual space. The system can track the hand's position to determine when it is within the camera's field of view.

[0008] US patent 2014 / 0204002A1 describes a virtual reality system in which a user can operate virtual interfaces with their hand. For this purpose, the user's real hand is filmed, and the video image of the real hand is superimposed onto the virtual environment. To obtain only image data of the hand within a camera image of the user's surroundings, the video image is segmented. After segmentation, only the image of the hand remains.

[0009] US Patent 2010 / 0 079 356 A1 describes a head-mounted display unit for displaying a virtual reality environment, which is capable of projecting the user's real environment into the user's field of vision when required.

[0010] The invention is based on the objective of supporting a user of a virtual reality system in orienting themselves in the real environment.

[0011] The problem is solved by the subject matter of the independent patent claims. Advantageous further developments of the invention result from the features of the dependent patent claims.

[0012] According to the invention, a device is provided comprising a control unit for generating graphic data representing a virtual space, and a display unit worn on the user's head. This display unit is designed to shield the user's eyes from light and to project the virtual space into the user's field of vision by displaying the graphic data. Furthermore, the device includes a localization device, known in the prior art, for detecting the spatial orientation of the display unit. This allows the perspective from which the virtual space is displayed to the user via the graphic data to depend on the current spatial orientation of the display unit. This enables the user to look around in the virtual space as described, although this is not essential to the invention.

[0013] According to the invention, the display unit, i.e., the head-mounted display, includes a camera designed to generate image data of the environment, i.e., the real-world environment in which the user is located while using the display unit. The control unit is designed to output the image data to the display unit based on the spatial orientation detected by the localization device, so that the display unit projects the environment into at least a portion of the user's field of vision by displaying the image data. In other words, the user sees the real-world image from the camera when the display unit is in a certain orientation, for example, when the user's head is tilted at a specific angle. This allows, for example, the VR user to see their own body when looking downwards.

[0014] Accordingly, the method provided according to the invention involves generating the graphic data representing the virtual space and displaying it in front of the user's light-tightly shielded eyes by means of the display unit, so that the virtual space is superimposed into the user's field of vision. The spatial orientation of the display unit is detected by the localization device. A camera provided on the display unit generates image data of the surroundings, and this image data is output or displayed depending on the detected spatial orientation of the display unit. In other words, the camera transmits its image data to a control unit, which outputs the image data to the display unit for display when the display unit is in a specific spatial orientation. The display unit then displays this image data, thereby superimposing the surroundings into at least a portion of the user's field of vision.

[0015] The invention offers the advantage that, by assuming a specific body posture which causes the display unit to assume a predetermined spatial orientation, the user can have image data from the real environment superimposed into their field of vision instead of virtual space. This allows them to orient themselves in the real environment.

[0016] A further advantage arises from the fact that, according to a first aspect of the invention, the control unit is designed to display the image data during a translational movement of the display unit, i.e., when the display unit, for example, the smart glasses, is moved horizontally through the environment. This translational movement can be caused, for example, by the user walking. In other words, it is detected when the user walks around in their environment or generally moves in such a way that the display unit undergoes a translational movement. By displaying the image data in this way, i.e., by projecting the environment into the user's field of vision, the user is prevented from walking blindly through the environment. This prevents the user from tripping.Preferably, it may be provided that the image data is only displayed when the translational movement exceeds a predetermined threshold in terms of movement distance and / or movement speed.

[0017] Another advantage arises from the fact that, according to an additional or alternative second aspect of the invention, the control device is designed to display the image data via the display unit when a change in perspective within the virtual space occurs independently of any user movement. Normally, the user changes the perspective by turning or tilting their head, or by walking. However, it may also be desirable for the virtual space to be displayed to the user alternately from different perspectives, e.g., from two different directions, with the perspective simply switching independently of the user's movement. This allows the user to assume a new position in the virtual space without having to leave their actual location, for example, sitting in a chair.In such a sudden or gliding change of perspective, which is not caused by the user's own body movement, the environment is now, according to the training, displayed in the user's field of vision during the perspective change, i.e., the image data is shown. This has the advantage that the user does not lose their balance and does not experience nausea.

[0018] According to the invention, the display unit is preferably a pair of smart glasses or a smart helmet designed for the stereoscopic display of graphic and / or image data. If the camera image data is to be displayed stereoscopically, a stereo camera or a time-of-flight camera is preferably used, the latter being provided, for example, by a PMD (Photonic Mixing Device). Stereoscopic display of the virtual space and / or the environment improves the user's orientation when viewing the graphic or image data.

[0019] Preferably, the control unit that controls the display unit or smart glasses is designed to display the image data in the correct position. This means that when displaying the image data, the environment depicted or represented is shown from a viewing direction that corresponds to the user's own viewing direction. In other words, the user is not shown just any image section in their field of vision—for example, not an image of the ceiling if the user is looking in a different direction, such as at the floor. Instead, the user sees the section of the environment that corresponds to their viewing direction, i.e., the objects they are currently looking at.For example, if the user tilts their head downwards and holds their hand in front of their face, the camera image of their hand is also superimposed into their field of vision. This enhanced functionality offers the advantage that the user can better coordinate their own movements in relation to their real-world surroundings.

[0020] Preferably, the control unit is designed to display the image data via the display unit when the user's head is tilted at a predetermined angle. This allows the user to selectively switch between the virtual space and the real environment by tilting their head. The image data is displayed depending on the orientation of the display unit, i.e., not always. It is particularly preferred that the head tilt is one that allows the user to look at their own body. This avoids the previously described, disconcerting sensation of a VR user not being able to see their own body in the virtual space.

[0021] Displaying image data, i.e., overlaying the environment into the user's field of vision, can impair the immersion the user experiences when viewing the virtual space. Immersion is the sensory impression of the user's self-perception within the virtual world. It is achieved through the correlation between the user's own movement and the perspective displayed in the virtual space. Overlaying the real environment reduces immersion. Therefore, a beneficial improvement involves designing the control system to overlay the image data onto the graphics data using alpha blending. The user then sees the environment not completely, but only in outline. This can be sufficient to provide the user with the desired orientation in the real environment without completely destroying immersion.

[0022] The use of the device for displaying a virtual lecture has proven to be a particularly suitable application. A corresponding embodiment of the invention therefore provides that the control unit is configured to display a virtual lecture using graphic data, in which, for example, an avatar (i.e., a virtual person) presents lecture content, such as a presentation, to the user in a virtual lecture hall. The device enables the user to follow the virtual lecture with a normal forward gaze. However, if the user looks down, they see their arms and hands and, for example, a sheet of paper on which they are taking notes.

[0023] The invention also includes further developments of the method according to the invention, which have features that have already been described in connection with the further developments of the device according to the invention. For this reason, a description of the corresponding further development of the method according to the invention is omitted here.

[0024] An embodiment of the invention is described below. The single figure (Fig.) shows an embodiment of the device according to the invention.

[0025] The embodiment described below is a preferred embodiment of the invention. In this embodiment, however, the described components each represent individual features of the invention that must be considered independently of one another. Each of these features further develops the invention independently and can therefore be considered part of the invention individually or in a combination other than that shown. Furthermore, the described embodiment can also be supplemented by other features of the invention already described.

[0026] The figure shows a device 10 with a control unit 12, a display unit 14, and a localization device 16. The control unit 12 can be, for example, a processor unit, such as a computer or an embedded system, or a digital signal processor (DSP). The control unit 12 controls the display unit 14. The display unit 14 is worn on the head 20 of a user 18. The display unit 14 can comprise a screen assembly 22, which is held in front of the user 18's eyes, for example, by a frame or visor. The user 18 thus does not see their surroundings 24, but rather the screen content displayed by the screen assembly 22 in front of their eyes. The screen assembly 22 can be configured as a stereoscopic screen pair, i.e., it can have two screens, each of which can display the image content for one eye of the user 18.Instead of screens, projectors can also be provided for the retina. The display unit 14 can be configured, for example, as data glasses or a data helmet.

[0027] The localization unit 16 can be configured to determine a spatial orientation P of the display unit 14 within the environment 24. The spatial orientation P can include a tilt or orientation of the display unit 14 in space within the environment 24 and / or a position of the display unit 14 within the environment 24. The localization device 16 can be fixed in the environment 24 and / or at least partially integrated into the display unit 14. The localization device 16 can, for example, include a camera that films the display unit 14, and / or include a gyroscopic sensor or accelerometer, and / or be based on a localization method using triangulation and / or trilateration.

[0028] The control unit 12 enables the user to have a virtual space 28 displayed in the user's field of vision V by means of the display unit 14, based on graphic data G, which can be generated, for example, by a VR simulator 26 in a known manner depending on the position data P. For example, it can be provided that the user 18 sees a product 30 displayed in the virtual space, such as a motor vehicle that the user intends to buy.

[0029] The user can change his perspective on the product 30 in the virtual space 28 in a manner known per se by turning and moving his head 20, by moving the display unit 14 and the localization device 16 detects a change in position of the display unit 14 and signals this by means of the position data P and this can be taken into account by the simulator 26 by adjusting the graphic data G in a manner known per se.

[0030] The display unit 14 obscures the user's 18 eyes from the surroundings 24. However, with the device 10, the user can still orient themselves within the surroundings 24 at any time. For example, it may be provided that the user 18 can tilt their head 20 32, for example towards the ground or towards their own body, thereby displaying the surroundings 24 in their field of vision V.

[0031] For this purpose, the device 10 can include a camera 34, which, as part of the display unit 14, can also be worn on the user's head 20. The camera 34 can, for example, be a video camera and be designed to produce stereoscopic or monoscopic images. A detection area 36 of the camera 24 can be aligned with the user's straight-ahead viewing direction 38. The camera 34 can generate camera data C, which can be transmitted to the control unit 12.

[0032] In the example shown in the figure, the user 18 wants to look at his hand 40. To do this, he tilts his head 20 downwards by means of the tilting movement 32. The control device 12 can have an analysis device 42 which determines, based on the position data P, whether the user 18 is directing his field of view V, for example, below a horizontal boundary 44. The analysis device 42 can, for example, be a program module of the control device 12. Due to the tilting movement 32, the perspective on the product 30, or more generally in the virtual space 28, is shifted in the graphic data G in a corresponding counter-movement 32', so that the field of view V also includes an area beyond the boundary 44. In this sub-area 46, the analysis device 42 displays the real environment 24 instead of the virtual space 28 by means of the analysis device 42, i.e.In total, the control unit 12 outputs, sends or transmits camera data C to the display unit 14.

[0033] The display unit 14 shows the camera data C on the screen 22 in the corresponding sub-area 46. The sub-area 46 is selected such that a Fig. The hand 40 is in the correct position with respect to the straight-ahead viewing direction 38, just as the hand 40 is in relation to the straight-ahead viewing direction 38. In other words, the user sees his hand 40 in the same place in the field of vision V as he would see it if he were not wearing the display unit 14.

[0034] The user can selectively switch between the virtual space 28 and the real environment 24 by simply performing a predetermined tilting movement 32 with their head 20. The real environment 24 appears to the user as a filmed reality. By adjusting the tilt angle of the image area to the real image, the user 18 sees objects in their real environment 24 in the correct position, i.e., in the same position in their field of vision V as they would see them if they were not wearing the display unit 14. Thus, the virtual and real worlds can be meaningfully linked using simple means.

[0035] Overall, the example shows how the invention allows certain real-image areas to be displayed depending on the orientation of VR glasses.

Claims

[1] Device (10) comprising: - a control device (12) for generating graphic data (G) representing a virtual space (28), - a display unit (14) to be worn on the head (20) by a user (18), which is designed to shield the eyes of the user (18) from light from an environment (24) and to display the virtual space (28) into a field of vision (V) of the user (18) by displaying the graphic data (G), - a localization device (16) designed to detect a spatial orientation (P) of the display unit (14), wherein the display unit (14) has a camera (34) designed to generate image data (C) of the environment (24), which represents the real environment in which the user is located while using the display unit, and the control device (12) is designed to output the image data (C) to the display unit (14) depending on the detected spatial orientation (P), so that the display unit (14), in the case of a certain orientation of the display unit, displays the environment (24), i.e. the real environment in which the user is located, at least in a partial area (46) of the field of view (V) by displaying the image data (C), characterized by , that a) the control device (12) is designed to display the image data (C) by means of the display unit (14) during a translational movement of the display unit (14) caused by a movement of the user (18), and / or b) the control device (12) is designed to display the image data (C) by means of the display unit (14) when a change in perspective from which the virtual space (28) is represented by the graphic data (G) occurs, independent of any body movement of the user (18). [2] Device (10) according to claim 1, wherein the display unit (14) is designed as data glasses (14) designed for stereoscopic display of the graphic data (G) and / or the image data (C). [3] Device (10) according to one of the preceding claims, wherein the control device (12) is configured to display the image data (C) in the correct position by means of the display unit (14), so that the environment (24) is shown from a viewing direction (38) that corresponds to the user's own viewing direction (38) into the environment (24). [4] Device (10) according to one of the preceding claims, wherein the control device (12) is configured to display the image data (C) by means of the display unit (14) at a predetermined head tilt (32) of the user (18). [5] Device (10) according to claim 4, wherein the head tilt (32) is such that the user (18) looks at his own body (40). [6] Device (10) according to one of the preceding claims, wherein the control device (12) is configured to superimpose the image data (C) onto the graphics data (G) by means of alpha blending. [7] Device (10) according to one of the preceding claims, wherein the control device (12) is designed to display a virtual lecture using the graphic data (G). [8] Method for displaying a virtual space (28) into a user's field of view (V) (18), comprising the steps: - Generating the graphic data (G) representing the virtual space (28) by a control device (12), - light-tight shielding of a user's eyes (18) from an environment (24) and displaying the virtual space (28) into a field of vision (V) of the user (18) by displaying the graphic data (G) by means of a display unit (14) worn by the user (18) on the head (20); - Determining the spatial position (P) of the display unit (14) by a localization device (16), - Generating image data (C) of the environment (24), which represents the real environment in which the user is located while using the display unit, by means of a camera (34) of the display unit (14), - Output of the image data (C) for display to the display unit (14) depending on the detected spatial orientation (P) by the control unit (12) at a specific spatial orientation of the display unit and - Displaying the image data (C) by the display unit (14) and thereby showing the environment (24), i.e. the real environment in which the user is located, at least in a partial area (46) of the field of view (V), characterized by , that a) the control unit (12) displays the image data (C) by means of the display unit (14) during a translational movement of the display unit (14) caused by a movement of the user (18), and / or b) the control unit (12) displays the image data (C) by means of the display unit (14) when a change in perspective from which the virtual space (28) is represented by the graphic data (G) occurs, independent of any body movement of the user (18).

Images