Method for operating a virtual reality system and virtual reality system

Abstract

Method for operating a virtual reality system (10), comprising the steps: - detecting a spatial position of a head (25) of a first person (26) wearing virtual reality glasses (12) and headphones (20); - displaying at least one virtual object (34) within a virtual environment (32) from a virtual viewing direction (38) by means of the virtual reality glasses (12), wherein the virtual viewing direction (38) is predetermined as a function of the detected spatial position of the head (25); - Playing back an acoustic recording by means of the headphones (20); characterized by the steps: - detecting speech sound (44) from at least one second person (40) by means of a microphone device (18) and converting the detected speech sound (44) into a speech signal; - Reproducing the speech signal by means of the headphones (20), wherein a left and a right loudspeaker (22, 24) of the headphones (20) are operated as a function of the detected spatial position of the head (25) in such a way that the speech signal is reproduced by the loudspeakers (22, 24) in the same way as the speech sound (44) would reach the first person (26) without the headphones (20) being worn; - wherein the microphone device (18) comprises an artificial head (42) equipped with a binaural recording device, which is positioned between the first person (26) and the second person (40), in particular on a connecting line between the first and the second person (26, 40), wherein the speech sound (44) is recorded by means of the binaural recording device in the form of a binaural recording method.

Description

[0001] The invention relates to a method for operating a virtual reality system and to a virtual reality system.

[0002] A virtual reality system within the meaning of the invention is a system by means of which a virtual reality can be displayed. The virtual reality system comprises, in particular, so-called virtual reality glasses, which are a specific form of a so-called head-mounted display, i.e. a visual output device worn on the head. They present images on a screen close to the eyes or project them directly onto the retina. Virtual reality glasses also have sensors for detecting head movements. This allows the display of the calculated graphics to be adapted to the movements of the wearer. Due to the physical proximity, the displayed image areas of head-mounted displays appear considerably larger than those of free-standing screens and, in extreme cases, even cover the user's entire field of vision.Since the display follows all head movements of the wearer due to the head position, the wearer gets the feeling of moving directly in an image landscape generated by a computer.

[0003] Using such virtual reality glasses, a virtual reality can be presented, whereby virtual reality is usually referred to as the representation and simultaneous perception of reality in its physical properties in a real-time computer-generated, interactive virtual environment.

[0004] Such a virtual reality system can be used, for example, in the sales of motor vehicles, to virtually display a vehicle using virtual reality glasses. A particular challenge in such an application is that the wearer of the virtual reality glasses should, on the one hand, be able to experience the highest possible quality sound via appropriate headphones, while, on the other hand, being able to understand statements from a salesperson and / or other accompanying persons and follow their conversations.

[0005] The wearer of the virtual reality headset can, for example, move around a virtual object, such as a motor vehicle, within a displayed virtual environment. A particular challenge in this context is to play the spoken utterances of people in the wearer's surroundings through the headset in such a way that the wearer is not confused.

[0006] The publication "Virtual and Augmented Reality" (ISBN 978-3-642-28903-3) describes a method in which the representation of a virtual environment is adapted depending on a person's detected head position. It also describes how the audio information contained in a virtual environment can be reproduced via headphones. To enable spatial perception of the sound in the virtual environment, sound can be reproduced via two channels, with the sound being played across the channels with a slight time offset and a level difference.

[0007] US 2008 / 0144794 A1 describes a method for operating a teleconferencing system. Several people each wear headphones and microphones. A virtual arrangement is specified for the people relative to each other. Speech captured by the respective microphones is reproduced via the headphones in such a way that the respective people hear the speech of the other people as it corresponds to the specified virtual arrangement.

[0008] US 2006 / 0029243 A1 discloses a method for sound reproduction. The arrangement of a virtual audio source relative to a virtual position of a person within a virtual environment is taken into account during sound reproduction.

[0009] The publication "3-D Sound for Virtual Reality and Multimedia" (http: / / ntrs.nasa.gov / search.jsp?R=20010044352) describes the audio design in virtual environments. Among other things, it also describes active noise compensation using anti-noise.

[0010] It is therefore the object of the present invention to provide a method for operating a virtual reality system and a virtual reality system by means of which it is possible for a wearer of virtual reality glasses to be provided with, in particular, linguistic utterances of one or more persons in an improved manner.

[0011] This object is achieved by a method for operating a virtual reality system and by a virtual reality system having the features of the independent patent claims. Advantageous embodiments with expedient and non-trivial refinements of the invention are specified in the dependent claims.

[0012] The method according to the invention for operating a virtual reality system comprises the following steps: - detecting a spatial position of a head of a first person wearing virtual reality glasses and headphones; - Displaying at least one virtual object within a virtual environment from a virtual viewing direction using the virtual reality glasses, wherein the virtual viewing direction is specified depending on the detected spatial position of the head; - Playing an acoustic recording using the headphones; - detecting speech sound from at least one second person by means of a microphone device and converting the detected speech sound into a speech signal; - Reproduction of the speech signal via the headphones, whereby a left and a right loudspeaker of the headphones are operated depending on the detected spatial position of the head in such a way that the speech signal is reproduced by the loudspeakers in the same way as the speech sound would reach the first person (ears, ear canal) without the headphones being worn.

[0013] On the one hand, the method according to the invention enables a wearer of virtual reality glasses to receive a particularly realistic representation of a virtual object within a virtual environment, since they can easily change their viewing angle of the displayed virtual object by changing the spatial position of their head. Preferably, it is also possible for the wearer of the virtual reality glasses to move within the displayed virtual environment. In other words, this means that they can change their virtual position within the virtual environment, so that a respective perspective on the virtual object can be changed. In addition, an acoustic recording is played back via the headphones, so that, for example, the highest possible quality sound is played, which can further improve or enhance the virtual reality experience.

[0014] Essential to the invention is that the speech signal is reproduced via the headphones in such a way that a left and a right loudspeaker of the headphones are operated depending on the detected spatial position of the head of the wearer of the virtual reality glasses. The speech signal is reproduced by the loudspeakers in the same way that the speech sound would reach the wearer of the virtual reality glasses without the headphones being worn. Therefore, if the wearer of the virtual reality glasses changes their virtual position within the virtual environment, the acoustically detectable position of the second person does not change for the wearer of the virtual reality glasses.

[0015] In other words, the headphones are operated in such a way that, regardless of the virtual positioning within the virtual environment, the same directional localization can always be ensured by the inventive reproduction of the speech signal, namely in the same way as the user would hear the second person without headphones on. In addition to the second person, who could be a salesperson, for example, a third person may also be present. The speech sound from the third person can also be captured by the microphone device and converted into a corresponding speech signal.The speech signal of the third person is also reproduced by means of the headphones in such a way that a left and a right loudspeaker of the headphones are operated depending on the detected spatial position of the head of the wearer of the virtual reality glasses in such a way that the speech signal of the third person is also reproduced by the loudspeakers in the same way as the speech sound would reach the wearer of the virtual reality glasses without the headphones being worn.

[0016] The wearer of the virtual reality glasses therefore always has an essentially fixed directional localization with regard to linguistic utterances of people in the environment of the wearer of the virtual reality glasses, so that he or she maintains a kind of acoustic orientation and anchoring to reality, even if the virtual environment is presented in a particularly realistic manner.

[0017] Furthermore, the invention provides for the speech sound to be captured using a binaural recording method, in particular using a binaural artificial head recording. In the simplest case, two microphones are used, pointing sideways away from each other and spaced approximately 17 cm to 22 cm apart, preferably 17.5 cm. This spacing and placement approximately represent the position of the ear canals of an average person. A sound-absorbing or sound-reflecting separating body, such as a soccer ball or a metal plate, is placed between the microphones to approximately simulate a head. This type of sound recording allows the headphones to create a particularly natural auditory impression with particularly precise directional localization.Binaural recordings, which replace the natural ear signals suppressed by headphone playback, are the best way to realistically reproduce a spatial auditory impression.

[0018] According to the invention, the microphone device comprises an artificial head equipped with a binaural recording device, which is positioned between the first person and the second person, in particular on a connecting line between the first and second persons, wherein the speech sound is captured by means of the binaural recording device. The artificial head is a head simulation, wherein the recording device comprises, for example, two condenser studio microphones with an omnidirectional characteristic, which are inserted into an artificial ear canal of the artificial head. In this case, so-called head-related transfer functions are simulated.

[0019] An advantageous embodiment of the invention provides that, during the reproduction of the speech signal, a time difference between the left and right loudspeakers of the headset is adjusted depending on the detected spatial position of the first person's head. This allows the speech signal to be reproduced by the loudspeakers in a particularly realistic manner, just as the speech sound would reach the first person without the headphones being worn.

[0020] A further advantageous embodiment of the invention provides that, when reproducing speech sound, a level difference between the left and right speakers of the headset is adjusted depending on the detected spatial position of the first person's head. This also makes it possible for the speakers to reproduce the speech signal in a particularly realistic manner, just as the speech sound would reach the first person without the headset being worn.

[0021] In a further advantageous embodiment of the invention, the relative position and / or location of the artificial head to the head of the first person, in particular also to the head of the second person, is detected and taken into account during the reproduction of the speech signal. The corresponding location and position information is preferably used to control the reproduction of the speech signal in such a way that the reproduction via the headphones reproduces a particularly realistic and spatial auditory impression, thus enabling particularly precise directional localization for the first person.

[0022] It can also be provided that the microphone device comprises a microphone worn by the other person, by means of which the speech sound is captured. Because the microphone device is worn by the other person, the speech sound of the other person is primarily captured, with other ambient noises being captured to a lesser extent by the microphone device.

[0023] Furthermore, it is possible for the relative position and / or location of the second person's head to the first person's head to be detected and taken into account when playing the speech signal. In other words, the relative positioning of the two people to each other and the respective alignment of their heads to each other are taken into account, so that the speech signal can be output in such a way that a particularly good, realistic spatial auditory impression is achieved for the second person when played through the headphones.

[0024] According to a further advantageous embodiment of the invention, further ambient sound is detected by the microphone device, whereby this ambient sound is filtered out and not reproduced via the headphones if it is lower than the detected speech sound of the second person by a predetermined volume level. This allows conversations beyond a certain distance to be virtually blocked and not transmitted via the headphones, which is particularly helpful in a semi-public situation in a salesroom.

[0025] In a further advantageous embodiment of the invention, additional ambient noise is detected by the microphone device, with this ambient noise, with the exception of the speech sound of the second person, being attenuated by active noise compensation generated by the headphones. In other words, a type of anti-noise is generated, by means of which the remaining ambient noise, except for the speech sound of the second person, is attenuated or eliminated.

[0026] The virtual reality system according to the invention comprises - virtual reality glasses designed to display at least one virtual object within a virtual environment; - a detection device which is designed to detect a spatial position of a head of a first person wearing the virtual reality glasses; - a control device which is designed to determine a virtual viewing direction depending on the detected spatial position of the head of the first person and to control the virtual reality glasses in such a way that they display the virtual object within the virtual environment from the virtual viewing direction; - a microphone device which is designed to capture speech sound from at least a second person and to convert it into a speech signal, wherein the microphone device comprises an artificial head equipped with a binaural recording device which is positioned between the first person and the second person, in particular on a connecting line between the first and the second person, and the binaural recording device is designed to capture the speech sound in the form of a binaural recording method; - a headset with a left and right speaker, which is designed to reproduce an acoustic recording and the speech signal; - wherein the control device is designed to control the headphones in such a way that the left and right loudspeakers of the headphones are operated as a function of the detected spatial position of the head in such a way that the speech signal is reproduced by the loudspeakers in such a way that the speech sound would reach the first person without the headphones being worn.

[0027] The advantageous embodiments of the method according to the invention are to be regarded as advantageous embodiments of the virtual reality system according to the invention, wherein the virtual reality system in particular has means for carrying out the method steps.

[0028] Further advantages, features, and details of the invention will become apparent from the following description of preferred embodiments and from the drawings. The features and combinations of features mentioned above in the description, as well as the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures, can be used not only in the respective combinations specified, but also in other combinations or on their own, without departing from the scope of the invention.

[0029] Embodiments of the invention are explained in more detail below with reference to schematic drawings. They show: Fig. 1 a schematic representation of a virtual reality system for displaying a virtual object within a virtual environment; Fig. 2 a perspective view of a partially shown sales room, with a person wearing virtual reality glasses of the virtual reality system; Fig. 3 a representation of a virtual environment in which a virtual object in the form of a motor vehicle is shown in a side view; Fig. 4 a schematic plan view of a possible embodiment of the Fig. 2, in which, in addition to the person wearing the virtual reality glasses, another person and an artificial head arranged between them are shown; and Fig. 5 a schematic plan view of an alternative embodiment of the sales room, again showing the wearer of the virtual reality glasses and, in this case, only the person sitting opposite, who is wearing a microphone.

[0030] In the figures, identical or functionally identical elements are provided with the same reference symbols.

[0031] A virtual reality system 10 for displaying a virtual environment is shown in a schematic representation in Fig. 1. The virtual reality system 10 comprises virtual reality glasses configured to display at least one virtual object within a virtual environment. The virtual reality glasses 12 comprise a detection device 14 configured to detect the spatial position of a head of a person wearing the virtual reality glasses 12.

[0032] The virtual reality system 10 further comprises a control device 16 which is designed to determine a virtual viewing direction depending on the detected spatial position of the head of the wearer of the virtual reality glasses 12 and to control the virtual reality glasses 12 in such a way that they display the currently displayed virtual object within the virtual environment from the virtual viewing direction.

[0033] Furthermore, the virtual reality system 10 has a microphone device 18, which is designed to capture speech sound from at least a second person and convert it into a speech signal. Finally, the virtual reality system 10 also has a headset 20 with a left and right loudspeaker 22, 24, which is designed to reproduce an acoustic recording and the speech signal. The control device 16 is designed to control the headset 20 such that the left and right loudspeakers 22, 24 of the headset 20 are operated depending on the detected spatial position of the head of the wearer of the virtual reality glasses 12 such that the speech signal is reproduced by the loudspeakers 22, 24 in the same way as the speech sound of the other person would reach the wearer of the virtual reality glasses 12 without the headset 20 being worn.

[0034] In Fig. 2 shows an unspecified salesroom in a car dealership. In this case, a first person 26 is wearing the virtual reality glasses 12 of the virtual reality system 10. The virtual reality glasses 12 are coupled to the control device 16 arranged under a table 28, which may be, for example, a conventional PC. The virtual reality system 10 further comprises a remote control 30, by means of which the user 26 can control the display of the virtual reality glasses 12. By means of the axes x 1 , y 1 and z 1 is a head-fixed coordinate system of the first person 26.

[0035] In Fig. 3 shows a virtual environment 32, within which a virtual object in the form of a motor vehicle 34 is depicted. The dashed circle 36 indicates the current virtual position of the first person 26 within the virtual environment 32. The arrow 38 indicates the current virtual viewing direction, starting from the virtual position 36. The virtual viewing direction 38 corresponds to the current spatial position, i.e. the orientation, of the first person 26 wearing the virtual reality glasses 12. If, for example, the person turns their head to the left, they no longer look at the motor vehicle 34, as shown here, but rather at an area further to the left within the virtual environment 32. The same applies to a swivel movement of the head of the person 26 up and down.Furthermore, the person 26 can move within the virtual environment 32 by actuating, for example, the remote control 30, for example, by virtually walking around the vehicle 34. With the axes x. 2 , y 2 and z 2 is the coordinate system within the virtual environment 32.

[0036] In addition to the purely visual representation of the virtual environment 32, a recording accompanying the virtual representation is played via the headphones 20. For example, the recording can be purely music or also corresponding functional sounds of the virtual motor vehicle 34, such as exhaust noises, sounds from the stereo system of the motor vehicle 34, and the like. These virtual sounds can, for example, also be changed depending on the virtual position of the person 26 within the virtual environment 32, so that a type of virtual spatial auditory impression is enabled within the displayed virtual environment 32 by playing them via the headphones 20.

[0037] In Fig. Figure 4 shows a schematic top view of a possible arrangement of the first person 26 relative to a second person 40, for example, a salesperson in a car dealership. An artificial head 42 is arranged between the first person 26 and the second person 40 on the table 28 of the salesroom. In the present case, the microphone device 18 is formed by respective microphones, not designated in more detail, arranged on the outer sides of the artificial head 42. Speech sound 44 emitted by the second person 40 is captured by the microphone device 18. In the present case, the speech sound 44 is thus captured using a binaural recording method, more precisely using a binaural artificial head recording. The relative location and / or position of the artificial head 42 to the head 25 of the first person 26 and also to the head 46 of the second person 40 is captured and taken into account when reproducing the speech signal via the headphones 20.

[0038] The speech signal is reproduced via headphones 20, with the left and right loudspeakers 22, 24 of headphones 20 being operated depending on the detected spatial position of the head 25 of the first person 26 and the additionally detected position and orientation information of the head 25 relative to the artificial head 42 and the head 46 of the second person 40 such that the speech signal is reproduced by the loudspeakers 22, 24 in the same way as the speech sound would reach the first person, more precisely, to their ears or into their auditory canals, without the headphones 20 being worn. For example, when reproducing the speech signal, a delay difference and / or level difference between the left and right loudspeakers 22, 24 of headphones 20 is adjusted depending on the spatial position and orientation information.

[0039] With the axes x 3 , y 3 and z 3is a head-fixed coordinate system with respect to the second person 40. By means of the axes x 4 , y 4 and z 4 is a coordinate system that is stationary with respect to the artificial head 42. The respective relative positions with respect to the stationary coordinate systems of the head 46 of the second person, the artificial head 42, and the head 25 of the first person 26 can thus be recorded and evaluated with respect to their relative positions and positioning. In addition, the volume setting by means of which the speech signal converted from the recorded speech sound 44 is fed via the headphones 20 is adjusted taking into account the respective distances A 1, A 2 and A 3 between the respective heads 25, 42, 46.

[0040] Thus, if the first person 26 moves around within the virtual environment 32 using the content displayed on the virtual reality glasses 12, the captured speech sound 44 is always played through the headphones 20 using the converted speech signal in such a way that the perceived position of the second person 40 relative to the first person 26 does not change. In other words, the directional localization for the first person 26 wearing the virtual reality glasses 12 always remains constant with respect to the second person 40, at least as long as the second person 40 does not move.

[0041] Further ambient sound can also be detected, for example, by means of the microphone device 18, wherein this ambient sound is filtered out and not reproduced via the headphones 20 if it is lower by a predetermined volume level than the detected speech sound of the second person 40. This allows conversations beyond a certain distance to be virtually blocked and not transmitted to the first person 26 via the headphones 20, which is particularly helpful in a semi-public situation in a car dealership.

[0042] Alternatively or additionally, it is also possible for the headphones 20 to be so-called active noise-canceling headphones. Either the headphones 20 themselves have corresponding microphones for detecting ambient noise, or the sound information picked up by the microphone device 18 is attenuated, with the exception of the speech sound 44 of the second person 40, by means of active noise compensation, i.e., by means of anti-noise, generated by the headphones 20. This can also ensure that, above all, only the speech sound 44 reaches the ears of the first person 26.

[0043] In Fig. Figure 5 shows an alternative arrangement between the first and second person. Here, the artificial head 42 is no longer located between the first and second person 26, 40. Instead, the second person 40 wears a microphone 48 belonging to the microphone device 18 directly in front of their mouth, by means of which the speech sound 44 is recorded. This has the advantage that ambient noise is hardly recorded, or to a significantly lesser extent, than in the Fig.4. Here, too, the relative position and / or location of the head 46 of the second person 40 to the head 25 of the first person 26 is detected and taken into account when reproducing the converted speech signal. The speech signal is again reproduced via the headphones 20, with the left and right loudspeakers 22, 24 of the headphones 20 being operated depending on the detected location and position information such that the speech signal is reproduced by the loudspeakers 22, 24 in the same way as the speech sound 44 would have reached the first person 26 without the headphones 20 being worn.In this case, for example, the time difference and / or the level difference between the left and right loudspeakers 22, 24 can also be adjusted accordingly to enable the most realistic reproduction of the detected speech sound 44 and the associated particularly precise and realistic directional localization of the speech sound 44 and thus of the second person 40. In a similar manner, it is possible to perform active noise compensation to attenuate or mask out further ambient noise as much as possible. Or, just as well, detected ambient noise can be filtered out and not reproduced via the headphones 20 if it is lower than the detected speech sound 44 of the second person 40 by a predetermined volume level.The latter should be particularly easy since the microphone 48 is worn directly in front of the mouth of the second person 40, so that the speech sound 44 emitted by the second person 40 should reach the microphone 48 much more strongly than the remaining ambient sound.

Claims

[1] Method for operating a virtual reality system (10), comprising the steps: - detecting a spatial position of a head (25) of a first person (26) wearing virtual reality glasses (12) and headphones (20); - displaying at least one virtual object (34) within a virtual environment (32) from a virtual viewing direction (38) by means of the virtual reality glasses (12), wherein the virtual viewing direction (38) is predetermined as a function of the detected spatial position of the head (25); - Playing back an acoustic recording by means of the headphones (20); characterized by the steps: - detecting speech sound (44) from at least one second person (40) by means of a microphone device (18) and converting the detected speech sound (44) into a speech signal; - Reproducing the speech signal by means of the headphones (20), wherein a left and a right loudspeaker (22, 24) of the headphones (20) are operated as a function of the detected spatial position of the head (25) in such a way that the speech signal is reproduced by the loudspeakers (22, 24) in the same way as the speech sound (44) would reach the first person (26) without the headphones (20) being worn; - wherein the microphone device (18) comprises an artificial head (42) equipped with a binaural recording device, which is positioned between the first person (26) and the second person (40), in particular on a connecting line between the first and the second person (26, 40), wherein the speech sound (44) is recorded by means of the binaural recording device in the form of a binaural recording method. [2] Method according to claim 1, characterized bythat when reproducing the speech signal, a time difference between the left and right loudspeakers (22, 24) of the headphones (20) is adjusted depending on the detected spatial position of the head (25) of the first person (26). [3] Method according to claim 1 or 2, characterized by that when reproducing the speech signal, a level difference between the left and right loudspeakers (22, 24) of the headphones (20) is adjusted depending on the detected spatial position of the head (25) of the first person (26). [4] Method according to one of the preceding claims, characterized by that the relative position and / or location of the artificial head (42) to the head (25) of the first person (26), in particular also to the head (46) of the second person (40), is detected and taken into account when reproducing the speech signal. [5] Method according to one of the preceding claims, characterized bythat further ambient sound is detected by means of the microphone device (18), wherein this ambient sound is filtered out and not reproduced by means of the headphones (20) if it is lower by a predetermined volume level than the detected speech sound (44) of the second person (40). [6] Method according to one of the preceding claims, characterized by that further ambient sound is detected by means of the microphone device (18), wherein this ambient sound, with the exception of the speech sound (44) of the second person (40), is attenuated by means of an active noise compensation generated by the headphones (20). [7] Virtual reality system (10), with - virtual reality glasses (12) designed to display at least one virtual object (34) within a virtual environment (32); - a detection device (14) which is designed to detect a spatial position of a head (25) of a first person (26) wearing the virtual reality glasses (12); - a control device (16) which is designed to determine a virtual viewing direction (38) as a function of the detected spatial position of the head (25) of the first person (26) and to control the virtual reality glasses (12) in such a way that they display the virtual object (34) within the virtual environment (32) from the virtual viewing direction (38); characterized by - a microphone device (18) which is designed to capture speech sound (44) from at least a second person (40) and to convert it into a speech signal, wherein the microphone device (18) comprises an artificial head (42) equipped with a binaural recording device, which is positioned between the first person (26) and the second person (40), in particular on a connecting line between the first and the second person (26, 40), and the binaural recording device is designed to capture the speech sound (44) in the form of a binaural recording method; - a headset (20) with a left and right loudspeaker (22, 24) which is designed to reproduce an acoustic recording and the speech signal; - wherein the control device (16) is designed to control the headphones (20) in such a way that the left and right loudspeakers (22, 24) of the headphones (20) are operated as a function of the detected spatial position of the head (25) in such a way that the speech signal is reproduced by the loudspeakers (22, 24) in such a way that the speech sound (44) would reach the first person (26) without the headphones (20) being worn.

Images