Recording virtual and real objects in mixed reality devices
The virtual image generation system addresses the challenge of recording and playing back relevant sounds in VR/AR systems by using an object selection device to isolate and track objects, ensuring only desired audio and video data is captured and played back, improving the user experience.
Patent Information
- Application Number
- JP2023076101
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2017-02-28
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2038-02-27
AI Technical Summary
Existing VR/AR systems face challenges in recording experiences with unwanted or unexpected sounds due to noisy environments, making it difficult to isolate and record sounds from virtual or real objects of interest.
A virtual image generation system that includes an object selection device to persistently select objects within the field of view, generate video and audio data from selected objects, and store it in memory, while tracking the object's location to manage data storage based on visibility, using microphones and cameras for input and sensors for tracking.
Enables efficient recording and playback of audio and video data from selected virtual or real objects, ensuring that only relevant sounds are captured and played back, enhancing the user experience by reducing noise and improving clarity.
Smart Images

Figure 0007578755000001 
Figure 0007578755000002 
Figure 0007578755000003
Abstract
Description
[Technical field]
[0001] The present invention relates generally to virtual reality and augmented reality systems. [Background technology]
[0002] Modern computing and display technologies have facilitated the development of mixed reality systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or parts thereof are presented to the user in a manner that appears or can be perceived as real. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user (i.e., transparent to other actual real-world visual inputs). Thus, AR scenarios involve the presentation of digital or virtual image information with transparency to other actual real-world visual inputs.
[0003] For example, referring to Figure 1, an augmented reality scene 4 is depicted in which a user of the AR technology sees a real-world park-like setting 6 featuring people, trees, buildings in the background, and a concrete platform 8. In addition to these items, the end user of the AR technology also perceives as "seeing" a robotic figure 10 standing on the real-world platform 8, and a flying cartoon-like avatar character 12 that appears to be an anthropomorphic bumblebee, although these elements 10, 12 do not exist in the real world. In conclusion, the human visual perception system is highly complex, making it difficult to produce VR or AR technology that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements.
[0004] VR and AR systems typically employ head-mounted displays (or helmet-mounted displays, or smart glasses) that are at least loosely coupled to the user's head and thus move as the end user's head moves. When the end user's head movement is detected by the display system, the displayed data can be updated to account for changes in head pose (i.e., the orientation and / or location of the user's head). Head-mounted displays that enable AR (i.e., simultaneous viewing of virtual and real objects) can have several different types of configurations. In one such configuration, often referred to as a "video see-through" display, a camera captures elements of a real scene, a computing system superimposes virtual elements on the captured real scene, and an opaque display presents a synthetic image to the eyes. Another configuration, often referred to as an "optical see-through" display, allows the end user to see light from real objects in the environment through transparent (or semi-transparent) elements in the display system and directly view them. A transparent element, often referred to as a "combiner," superimposes the light from the display across the end-user's view of the real world.
[0005] Often, users of a VR / AR system may wish to share their experiences with others by recording and storing the experiences on the VR / AR system for later publication online (e.g., when playing a game, video conferencing, or watching a movie). However, typically due to a noisy environment, there may be numerous sources of noise and other undesirable or unexpected sounds that may be present in the recording or that distract attention from the rest of the experience. Such undesirable / unexpected sounds may come from real objects, e.g., from children playing near the VR / AR system, or from virtual objects, e.g., from a virtual television playing in the context of the VR / AR system.
[0006] Thus, there remains a need to provide a simple and efficient means for recording sounds from only those virtual or real objects in which a user is interested. Summary of the Invention [Means for solving the problem]
[0007] According to a first aspect of the present invention, a virtual image generation system for use by an end user comprises a memory, a display subsystem, and an object selection device configured to receive input from an end user and persistently select at least one object (e.g., a real object and / or a virtual object) in response to the end user input. In one embodiment, the display subsystem has a field of view, and the object selection device is configured to persistently select an object within the field of view. In this case, the object selection device may be configured to move a three-dimensional cursor within the field of view of the display subsystem to select an object in response to receiving the end user input. In another embodiment, the end user input comprises one or more voice commands, and the object selection device comprises one or more microphones configured to sense the voice commands. In yet another embodiment, the end user input comprises one or more hand gestures, in which case the object selection device may comprise one or more cameras configured to sense the hand gestures.
[0008] If multiple objects are selected, the object selection device may be configured for individually selecting and / or globally selecting the objects in response to an end user input. If globally selected, the object selection device may be configured for globally selecting all objects within an angular range of the field of view (which may be less than the entire angular range of the field of view, or may be the entire angular range of the field of view) in response to an end user input. In one embodiment, the object selection device is further configured for receiving another input from the end user in response to another end user input to persistently deselect the previously selected object.
[0009] The virtual image generation system further comprises a control subsystem configured to generate video data originating from at least one selected object, render a plurality of image frames within the three-dimensional scene from the video data, and communicate the image frames to the display subsystem. In one embodiment, the display subsystem is configured for being positioned in front of an eye of an end user. In another embodiment, the display subsystem includes a projection subsystem and a partially transparent display surface. In this case, the projection subsystem may be configured to cause the image frames to be projected onto the partially transparent display surface, which may be configured for being positioned within a field of view between the eye of the end user and the surrounding environment. The virtual image generation system may further comprise a frame structure configured for being worn by an end user and carrying at least a portion of the display subsystem.
[0010] The control subsystem is further configured for generating audio data originating from the selected object and storing the audio data in memory. The virtual image generation system may further comprise a plurality of speakers, in which case the control subsystem may further be configured for communicating the generated audio data to the speakers. In an optional embodiment, the control subsystem is further configured for storing video data in memory synchronous with the audio data. In yet another embodiment, the virtual image generation system further comprises at least one sensor configured for tracking a location of the selected object relative to a field of view of the display subsystem. In this case, the control subsystem may be configured for ceasing to store the audio data in memory when the tracked location of the selected object moves outside of the field of view of the display subsystem, or alternatively, for continuing to store the audio data in memory when the tracked location of the selected object moves outside of the field of view of the display subsystem.
[0011] If the selected object includes a real object, the virtual image generation system may further include a microphone assembly configured to generate an audio output, in which case the control subsystem may be further configured to modify a direction of the audio output to preferentially sense sounds originating from the selected real object. The audio data may be derived from the modified audio output. The virtual image generation system may further include one or more cameras configured to capture video data originating from the selected real object, in which case the control subsystem may be further configured to store in memory the video data synchronized with the audio data. The control subsystem may be configured to convert the captured video data into virtual content data for the selected real object and store the virtual content in memory.
[0012] If the selected object includes a virtual object, the virtual image generation system may further comprise a database configured to store content data corresponding to sounds for the plurality of virtual objects, in which case the control subsystem may further be configured to obtain content data corresponding to the selected virtual object from the database, and the audio data stored in the memory may include the obtained content data. The control subsystem may further be configured to generate metadata corresponding to the selected virtual object (e.g., position, orientation, and volume data for the selected virtual object), in which case the audio data stored in the memory may include the obtained content data and the generated metadata. In one embodiment, the virtual image generation system may further comprise one or more sensors configured to track a head pose of an end user, in which case the database may be configured to store absolute metadata for the plurality of virtual objects, and the control subsystem may further be configured to obtain absolute metadata corresponding to the selected virtual object and generate the metadata by localizing the absolute metadata to the end user based on the tracked head pose of the end user.
[0013] The virtual image generation system may further comprise at least one speaker, in which case the control subsystem may be further configured for retrieving stored audio data from the memory, deriving audio from the read audio data, and communicating the audio to the speaker. The audio data stored in the memory may include content data and metadata, in which case the control subsystem may be further configured for retrieving the stored content data and metadata from the memory, rendering spatialized audio based on the read content data and metadata, and communicating the rendered spatialized audio to the speaker.
[0014] According to a second aspect of the present invention, a method of operating a virtual image generation system by an end user is provided. The method includes persistently selecting at least one object (e.g., real object and / or virtual object). In one method, the step of selecting the object includes moving a three-dimensional cursor in the end user's field of view and selecting the object with the three-dimensional cursor. In another method, the step of selecting the object includes issuing one or more voice commands. In yet another method, the step of selecting at least one object includes performing one or more hand gestures. If multiple objects are selected, the step of selecting the multiple objects may include individually selecting the objects and / or globally selecting the objects. If selected globally, the objects may be selected by defining an angular range of the end user's field of view (which may be less than the entire angular range of the field of view or may be the entire angular range of the field of view) and selecting all of the objects within the defined angular range of the end user's field of view. The optional method may further include persistently deselecting a previously selected object.
[0015] The method further includes generating video data originating from the selected object, rendering a plurality of image frames in a three-dimensional scene from the generated video data, displaying the image frames to an end user, generating audio data originating from at least one selected object, and storing the audio data originating from at least one selected object in memory. One method may further include converting the audio data originating from the selected object into sound for perception by the end user. The method may optionally include storing the video data in memory synchronized with the audio data. Yet another method may further include tracking a location of the selected object relative to a field of view of the end user. In this case, the method may further include ceasing to store the audio data in memory when the tracked location of the selected object moves out of the field of view of the end user, or alternatively continuing to store the audio data in memory when the tracked location of the selected object moves out of the field of view of the end user.
[0016] If the selected object includes a real object, the method may further include preferentially sensing sounds originating from the selected real object compared to sounds originating from other real objects, in which case the audio data may be derived from the preferentially sensed sounds. The method may further include capturing video data originating from the selected real object and storing the video data in memory synchronized with the audio data. The captured video data may be converted into virtual content data for storage in the memory.
[0017] If the selected object includes a virtual object, the method may further include storing content data corresponding to sounds for the plurality of virtual objects and obtaining the content data corresponding to the selected virtual object, in which case the audio data stored in the memory may include the obtained content data. The method may further include generating metadata corresponding to the selected virtual object (e.g., position, orientation, and volume data for the selected virtual object), in which case the audio data stored in the memory may include the obtained content data and the generated metadata. The method may further include tracking a head pose of the end user and storing absolute metadata for the plurality of virtual objects. In this case, generating the metadata may include retrieving the absolute metadata corresponding to the selected virtual object and localizing the absolute metadata to the end user based on the tracked head pose of the end user.
[0018] The method may further include the steps of reading the stored audio data, deriving audio from the read audio data, and converting the audio into sound for perception by an end user. The stored audio data may include content data and metadata, in which case the method may further include reading the stored content data and metadata from memory, rendering spatialized audio based on the read content data and metadata, and converting the spatialized audio into sound for perception by an end user.
[0019] According to a third aspect of the present invention, a virtual image generation system for use by a playback user is provided. The virtual image generation system comprises a memory configured for storing audio and video content data originating from at least one object (e.g., a real object and / or a virtual object) in an original spatial environment, a plurality of speakers, and a display subsystem. In one embodiment, the display subsystem is configured for being positioned in front of an eye of an end user. In another embodiment, the display subsystem includes a projection subsystem and a partially transparent display surface. In this case, the projection subsystem may be configured for causing an image frame to be projected onto the partially transparent display surface, which may be configured for being positioned within a field of view between the eye of the end user and the surrounding environment. The virtual image generation system may further comprise a frame structure configured for being worn by an end user and carrying at least a portion of the display subsystem.
[0020] The virtual image generation system further includes a control subsystem configured to read audio content data and video content data from the memory, render audio and video from the read audio content data and video content data, respectively, in a new spatial environment different from the original spatial environment, and synchronously transmit the rendered audio to speakers and the generated video data to a display subsystem.
[0021] In one embodiment, the control subsystem is configured for storing the audio content data and the video content data in the memory. The virtual image generation system may further comprise an object selection device configured for receiving input from an end user and for persistently selecting an object in the original spatial environment in response to the end user input prior to storing the audio content data and the video content data in the memory.
[0022] If the object includes a real object, the virtual image generation system may further comprise a microphone assembly configured to capture audio content data from the real object in the original spatial environment. The microphone assembly may be configured to generate an audio output, in which case the control subsystem may be further configured to modify a direction of the audio output to preferentially sense sounds originating from the selected real object. The audio content data may be derived from the modified audio output. The virtual image generation system may further comprise one or more cameras configured to capture video data from the selected real object in the original spatial environment. In an optional embodiment, the control subsystem may be configured to convert the captured video data into virtual content data for the selected real object and store the virtual content data in the memory as video content data.
[0023] If the object includes a virtual object, the virtual image generation system may further include a database configured to store content data corresponding to sounds for the plurality of virtual objects, in which case the control subsystem may be further configured to obtain content data corresponding to the virtual object from the database, and the audio data stored in the memory may include the obtained content data.
[0024] In one embodiment, the control subsystem is configured to obtain absolute metadata corresponding to at least one object in the new spatial environment and render audio from the retrieved audio content data and the absolute metadata in the new spatial environment. Obtaining absolute metadata corresponding to an object in the new spatial environment may include positioning the object in the new spatial environment. In this case, the virtual image generation system may further include a user input device configured to receive input from a playback user, in which case the control subsystem may be configured to position the object in the new spatial environment in response to the input from the playback user. The virtual image generation system may further include one or more sensors configured to track a head pose of the playback user, in which case the control subsystem may be further configured to localize the absolute metadata to the playback user based on the tracked head pose of the playback user such that the rendered audio is spatialized.
[0025] According to a fourth aspect of the present invention, there is provided a method of operating a virtual image generation system by a playback user to play back audio and video of at least one object (e.g., real object and / or virtual object) previously recorded as audio content data and video content data in an original spatial environment. The method includes the step of reading the audio content data and the video content data from a memory. One method further includes the step of storing the audio content data and the video content data in the memory. In this case, the method may further include the step of persistently selecting the object in the original spatial environment prior to storing the audio content data and the video content data in the memory.
[0026] If the object includes a real object, the method may further include capturing audio content data from the real object. In this case, the method may further include preferentially sensing sounds originating from the selected real object compared to sounds originating from other real objects. The audio content data is derived from the preferentially sensed sounds. The method may further include capturing video data from the selected real object and converting the captured video data into virtual content data. If the object includes a virtual object, the method may further include storing content data corresponding to sounds for a plurality of virtual objects and obtaining content data corresponding to the virtual objects from a database. The audio content data stored in the memory may include the obtained content data.
[0027] The method further includes rendering audio and video from the read audio content data and video content data, respectively, into a new spatial environment different from the original spatial environment, converting the audio and video into sound and image frames, respectively, and synchronously transmitting the sound and image frames to a playback user. One method further includes obtaining absolute metadata corresponding to objects in the new spatial environment, where the audio is rendered into the new spatial environment from the read audio content data and the absolute metadata. The method may further include tracking a head pose of the playback user, and localizing the absolute metadata to the playback user based on the tracked head pose of the playback user, where the audio may be rendered into the new spatial environment from the read audio content data and the local metadata such that the rendered audio is spatialized. Obtaining absolute metadata corresponding to objects in the new spatial environment may include, for example, positioning the objects in the new spatial environment in response to an input from the playback user.
[0028] Additional and other objects, features, and advantages of the present invention are set forth in the detailed description, drawings, and claims. The present invention provides, for example, the following: (Item 1) 1. A virtual image generation system for use by an end user, comprising: Memory, A display subsystem; an object selection device configured to receive input from the end user and to persistently select at least one object in response to the end user input; a control subsystem configured for generating video data resulting from at least one selected object, rendering a plurality of image frames into a three-dimensional scene from the video data, communicating the image frames to the display subsystem, generating audio data resulting from the at least one selected object, and storing the audio data in the memory; A virtual image generation system comprising: (Item 2) 2. The virtual image generation system of claim 1, wherein the control subsystem is further configured to store the video data in the memory synchronized with the audio data. (Item 3) 2. The virtual image generation system of claim 1, further comprising a plurality of speakers, the control subsystem further configured to communicate the generated audio data to the speakers. (Item 4) 2. The virtual image generation system of claim 1, wherein the display subsystem has a field of view, and the object selection device is configured for persistently selecting the at least one object within the field of view. (Item 5) 5. The virtual image generation system of claim 4, wherein the object selection device is configured to move a three-dimensional cursor within a field of view of the display subsystem to select the at least one object in response to receiving the end user input. (Item 6) 2. The virtual image generation system of claim 1, wherein the end user input includes one or more voice commands, and the object selection device comprises one or more microphones configured to sense the one or more voice commands. (Item 7) 2. The virtual image generation system of claim 1, wherein the end user input includes one or more hand gestures, and the object selection device comprises one or more cameras configured to sense the one or more hand gestures. (Item 8) 2. The virtual image generation system of claim 1, wherein the at least one object includes a plurality of objects, and the object selection device is configured for individually selecting the objects in response to the end user input. (Item 9) 2. The virtual image generation system of claim 1, wherein the at least one object includes a plurality of objects, and the object selection device is configured for globally selecting the objects in response to the end user input. (Item 10) 10. The virtual image generation system of claim 9, wherein the object selection device is configured for globally selecting all objects within an angular range of the field of view in response to the end user input. (Item 11) Item 11. The virtual image generation system of item 10, wherein the angular range is less than the entire angular range of the field of view. (Item 12) 11. The virtual image generation system of claim 10, wherein the angular range is the entire angular range of the field of view. (Item 13) 2. The virtual image generation system of claim 1, wherein the object selection device is further configured to receive another input from the end user and persistently deselect at least one previously selected object in response to another end user input. (Item 14) 2. The virtual image generation system of claim 1, further comprising at least one sensor configured to track a location of the at least one selected object relative to a field of view of the display subsystem. (Item 15) Item 15. The virtual image generation system of item 14, wherein the control subsystem is configured to discontinue storing the audio data in the memory when the tracked location of the at least one selected object moves out of the field of view of the display subsystem. (Item 16) Item 15. The virtual image generation system of item 14, wherein the control subsystem is configured to continue storing the audio data in the memory when the tracked location of the at least one selected object moves out of the field of view of the display subsystem. (Item 17) 2. The virtual image generation system of claim 1, wherein the at least one selected object includes a real object. (Item 18) 18. The virtual image generation system of item 17, further comprising a microphone assembly configured to generate an audio output, the control subsystem further configured to modify a direction of the audio output and to preferentially sense sounds originating from the selected real object, and the audio data is derived from the modified audio output. (Item 19) 20. The virtual image generation system of claim 17, further comprising one or more cameras configured to capture video data originating from the selected real object, the control subsystem further configured to store in the memory the video data synchronized with the audio data. (Item 20) 20. The virtual image generation system of claim 19, wherein the control subsystem is configured to convert the captured video data into virtual content data for the selected real object and store the virtual content in the memory. (Item 21) 2. The virtual image generation system of claim 1, wherein the at least one selected object includes a virtual object. (Item 22) 22. The virtual image generation system of claim 21, further comprising a database configured to store content data corresponding to sounds for a plurality of virtual objects, the control subsystem further configured to obtain content data from the database corresponding to the selected virtual object, and the audio data stored in the memory includes the obtained content data. (Item 23) 23. The virtual image generation system of claim 22, wherein the control subsystem is further configured to generate metadata corresponding to the selected virtual object, and the audio data stored in the memory includes the obtained content data and the generated metadata. (Item 24) 24. The virtual image generation system of claim 23, wherein the metadata includes position, orientation, and volume data for the selected virtual object. (Item 25) 24. The virtual image generation system of claim 23, further comprising one or more sensors configured to track a head pose of the end user, the database configured to store absolute metadata for the plurality of virtual objects, and the control subsystem further configured to obtain absolute metadata corresponding to the selected virtual object and generate the metadata by localizing the absolute metadata to the end user based on the tracked head pose of the end user. (Item 26) 2. The virtual image generation system of claim 1, further comprising at least one speaker, the control subsystem further configured to read the stored audio data from the memory, derive audio from the read audio data, and communicate the audio to the at least one speaker. (Item 27) 27. The virtual image generation system of claim 26, wherein the audio data stored in the memory includes content data and metadata, and the control subsystem is further configured to read the stored content data and metadata from the memory, render spatialized audio based on the read content data and metadata, and transmit the rendered spatialized audio to the at least one speaker. (Item 28) 2. The virtual image generation system of claim 1, wherein the display subsystem is configured to be positioned directly in front of the end user's eye. (Item 29) 30. The virtual image generation system of item 28, wherein the display subsystem includes a projection subsystem and a partially transparent display surface, the projection subsystem configured to project the image frame onto the partially transparent display surface, and the partially transparent display surface configured to be positioned within a field of view between the end user's eyes and a surrounding environment. (Item 30) 30. The virtual image generation system of claim 28, further comprising a frame structure configured to be worn by the end user, the frame structure supporting at least a portion of the display subsystem. (Item 31) 1. A method for operating a virtual image generation system by an end user, comprising: persistently selecting at least one object; generating video data resulting from the at least one selected object; rendering a plurality of image frames into a three-dimensional scene from the generated video data; displaying said image frames to said end user; generating audio data resulting from said at least one selected object; storing in memory audio data resulting from said at least one selected object; A method comprising: (Item 32) 32. The method of claim 31, further comprising storing the video data in the memory synchronized with the audio data. (Item 33) 32. The method of claim 31, further comprising converting audio data resulting from the at least one selected object into sound for perception by the end user. (Item 34) Item 32. The method of item 31, wherein the at least one object is selected within a field of view of the end user. (Item 35) Item 35. The method of item 34, wherein selecting the at least one object includes moving a three-dimensional cursor within the end user's field of view and selecting the at least one object with the three-dimensional cursor. (Item 36) Item 32. The method of item 31, wherein selecting the at least one object includes issuing one or more voice commands. (Item 37) Item 32. The method of item 31, wherein selecting the at least one object includes performing one or more hand gestures. (Item 38) Item 32. The method of item 31, wherein the at least one object includes a plurality of objects, and selecting the plurality of objects includes individually selecting the objects. (Item 39) 32. The method of claim 31, wherein the at least one object includes a plurality of objects, and selecting the plurality of objects includes globally selecting the objects. (Item 40) 40. The method of claim 39, wherein globally selecting the objects includes defining an angular range of the end user's field of view and selecting all of the objects within the defined angular range of the end user's field of view. (Item 41) Item 41. The method of item 40, wherein the defined angular range is less than the entire angular range of the end user's field of view. (Item 42) Item 41. The method of item 40, wherein the defined angular range is the entire angular range of the end user's field of view. (Item 43) Item 32. The method of item 31, further comprising persistently deselecting at least one previously selected object. (Item 44) 32. The method of claim 31, further comprising tracking a location of the at least one selected object relative to a field of view of the end user. (Item 45) 45. The method of claim 44, further comprising ceasing to store in the memory audio data originating from the at least one selected object when the tracked location of the at least one selected object moves out of the field of view of the end user. (Item 46) 45. The method of claim 44, further comprising continuing to store in the memory audio data originating from the at least one selected object when the tracked location of the at least one selected object moves out of the field of view of the end user. (Item 47) Item 32. The method of item 31, wherein the at least one selected object includes a real object. (Item 48) 48. The method of claim 47, further comprising preferentially detecting sounds originating from the selected real object compared to sounds originating from other real objects, and the audio data is derived from the preferentially detected sounds. (Item 49) capturing video data originating from the selected real object; storing said video data in said memory synchronous with said audio data; Item 48. The method of item 47, further comprising: (Item 50) 50. The method of claim 49, further comprising converting the captured video data into virtual content data and storing the virtual content data in the memory. (Item 51) Item 32. The method of item 31, wherein the at least one selected object includes a virtual object. (Item 52) storing content data corresponding to sounds for a plurality of virtual objects; obtaining content data corresponding to the selected virtual object, wherein the audio data stored in the memory includes the obtained content data; 52. The method of claim 51, further comprising: (Item 53) generating metadata corresponding to the selected virtual object, wherein audio data stored in the memory includes the obtained content data and the generated metadata. 53. The method of claim 52, further comprising: (Item 54) 54. The method of claim 53, wherein the metadata includes position, orientation, and volume data for the selected virtual object. (Item 55) tracking a head pose of the end user; storing absolute metadata for the plurality of virtual objects; Further comprising: 54. The method of claim 53, wherein generating the metadata includes retrieving absolute metadata corresponding to the selected virtual object and localizing the absolute metadata to the end user based on a tracked head pose of the end user. (Item 56) 32. The method of claim 31, further comprising reading the stored audio data, deriving audio from the read audio data, and converting the audio to sound for perception by the end user. (Item 57) The stored audio data includes content data and metadata, and the method further comprises: reading the stored content data and metadata from the memory; rendering spatialized audio based on the retrieved content data and metadata; converting said spatialized audio into sound for perception by said end user; 32. The method according to claim 31, comprising: (Item 58) 1. A virtual image generation system for use by a playback user, comprising: a memory configured to store audio content data and video content data originating from at least one object in an original spatial environment; A plurality of speakers; A display subsystem; a control subsystem configured to read the audio content data and the video content data from the memory, render audio and video from the read audio content data and the video content data, respectively, in a new spatial environment different from the original spatial environment, and synchronously deliver the rendered audio to the speakers and the generated video data to the display subsystem; A virtual image generation system comprising: (Item 59) Item 59. The virtual image generation system of item 58, wherein the control subsystem is configured to store the audio content data and video content data in the memory. (Item 60) 60. The virtual image generation system of item 59, further comprising an object selection device configured to receive input from an end user prior to storing the audio content data and video content data in the memory and to persistently select the at least one object within the original spatial environment in response to end user input. (Item 61) 60. The virtual image generation system of item 59, wherein the at least one object includes a real object. (Item 62) Item 62. The virtual image generation system of item 61, further comprising a microphone assembly configured to capture the audio content data from a real object within the original spatial environment. (Item 63) Item 63. The virtual image generation system of item 62, wherein the microphone assembly is configured to generate an audio output, the control subsystem is further configured to modify a direction of the audio output and to preferentially sense sounds originating from a selected real object, and the audio content data is derived from the modified audio output. (Item 64) Item 62. The virtual image generation system of item 61, further comprising one or more cameras configured to capture the video data from selected real objects within the original spatial environment. (Item 65) Item 65. The virtual image generation system of item 64, wherein the control subsystem is configured to convert the captured video data into virtual content data for the selected real object and store the virtual content data in the memory as the video content data. (Item 66) Item 59. The virtual image generation system of item 58, wherein the at least one object includes a virtual object. (Item 67) Item 67. The virtual image generation system of item 66, further comprising a database configured to store content data corresponding to sounds for a plurality of virtual objects, the control subsystem further configured to obtain content data corresponding to the virtual objects from the database, and the audio data stored in the memory includes the obtained content data. (Item 68) The virtual image generation system of item 58, wherein the control subsystem is configured to obtain absolute metadata corresponding to the at least one object in the new spatial environment and render audio from the read audio content data and the absolute metadata in the new spatial environment. (Item 69) Item 70. The virtual image generation system of item 68, further comprising one or more sensors configured to track a head pose of the playback user, the control subsystem further configured to localize the absolute metadata to the playback user based on the tracked head pose of the playback user such that the rendered audio is spatialized. 69. The virtual image generation system of item 68, wherein obtaining absolute metadata corresponding to at least one object in the new spatial environment includes positioning the at least one object within the new spatial environment. (Item 71) The virtual image generation system of item 70, further comprising a user input device configured to receive input from the playback user, the control subsystem configured to position the at least one object within the new spatial environment in response to input from the playback user. (Item 72) Item 59. The virtual image generation system of item 58, wherein the display subsystem is configured to be positioned directly in front of the playback user's eye. (Item 73) The virtual image generation system of item 72, wherein the display subsystem includes a projection subsystem and a partially transparent display surface, the projection subsystem configured to cause image frames to be projected onto the partially transparent display surface in response to the video data, and the partially transparent display surface configured to be positioned within a field of view between the playback user's eyes and the new spatial environment. (Item 74) Item 73. The virtual image generation system of item 72, further comprising a frame structure configured to be worn by the playback user, the frame structure carrying at least a portion of the display subsystem. (Item 75) 1. A method of operating a virtual image generation system by a playback user to play back audio and video of at least one object previously recorded as audio content data and video content data within an original spatial environment, the method comprising: reading the audio and video content data from a memory; rendering audio and video from the retrieved audio content data and video content data, respectively, into a new spatial environment different from the original spatial environment; converting said audio and video into sound and image frames, respectively; synchronously transmitting said sound and image frames to said playback user; A method comprising: (Item 76) 76. The method of claim 75, further comprising storing the audio content data and video content data in the memory. (Item 77) 77. The method of claim 76, further comprising persistently selecting the at least one object in the original spatial environment prior to storing the audio content data and video content data in the memory. (Item 78) Item 77. The method of item 76, wherein the at least one object includes a real object. (Item 79) 80. The method of claim 78, further comprising capturing the audio content data from the real object. (Item 80) 80. The method of claim 79, further comprising preferentially detecting sounds originating from a selected real object compared to sounds originating from other real objects, and the audio content data being derived from the preferentially detected sounds. (Item 81) capturing video data from the selected real object; converting the captured video data into the virtual content data; 79. The method of claim 78, further comprising: (Item 82) Item 77. The method of item 76, wherein the at least one object includes a virtual object. (Item 83) storing content data corresponding to sounds for a plurality of virtual objects and obtaining content data corresponding to the virtual objects from the database, wherein audio content data stored in the memory includes the obtained content data; 83. The method of claim 82, further comprising: (Item 84) obtaining absolute metadata corresponding to the at least one object in the new spatial environment, and the audio is rendered in the new spatial environment from the retrieved audio content data and the absolute metadata. 76. The method of claim 75, further comprising: (Item 85) tracking a head pose of the playback user; localizing the absolute metadata to the playback user based on a tracked head pose of the playback user, wherein the audio is rendered within the new spatial environment from the retrieved audio content data and the local metadata such that the rendered audio is spatialized; 85. The method of claim 84, further comprising: (Item 86) 85. The method of claim 84, wherein obtaining absolute metadata corresponding to at least one object in the new spatial environment includes positioning the at least one object within the new spatial environment. (Item 87) receiving input from the playback user, wherein the at least one object is positioned within the new spatial environment in response to the input from the playback user; 87. The method of claim 86, further comprising: [Brief description of the drawings]
[0029] The drawings illustrate the design and utility of preferred embodiments of the present invention, where like elements are referred to by common reference numerals. In order to better understand how the above and other advantages and objects of the present invention are obtained, a more particular description of the invention briefly described above will be given by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. With the understanding that these drawings depict only typical embodiments of the invention and therefore should not be considered as limiting its scope, the present invention will be described and explained with additional specificity and detail through the use of the accompanying drawings.
[0030] [Figure 1] FIG. 1 is an illustration of a three-dimensional augmented reality scene that may be displayed to an end user by a prior art augmented reality generation device.
[0031] [Diagram 2] FIG. 2 is a perspective view of an augmented reality system constructed in accordance with one embodiment of the present invention.
[0032] [Diagram 3] FIG. 3 is a block diagram of the augmented reality system of FIG.
[0033] [Figure 4] FIG. 4 is a plan view of one embodiment of a spatialized speaker system used in the augmented reality system of FIG.
[0034] [Diagram 5] FIG. 5 is a plan view illustrating one technique used by the augmented reality system of FIG. 2 to allow an end user to individually select objects.
[0035] [Figure 6] FIG. 6 is a plan view illustrating another technique used by the augmented reality system of FIG. 2 to allow an end user to individually select objects.
[0036] [Figure 7] FIG. 7 is a plan view illustrating yet another technique used by the augmented reality system of FIG. 2 to allow an end user to individually select objects.
[0037] [Figure 8] FIG. 8 is a plan view illustrating a technique used by the augmented reality system of FIG. 2 to allow an end user to globally select multiple objects.
[0038] [Figure 9] FIG. 9 is a plan view illustrating another technique used by the augmented reality system of FIG. 2 to allow an end user to globally select multiple objects.
[0039] [Figure 10a] FIG. 10a is a plan view of one technique that may be used to mount the augmented reality system of FIG.
[0040] [Figure 10b] FIG. 10b is a plan view of another technique that can be used to mount the augmented reality system of FIG.
[0041] [Figure 10c] FIG. 10c is a plan view of yet another technique that can be used to mount the augmented reality system of FIG.
[0042] [Figure 10d] FIG. 10d is a plan view of yet another technique that can be used to mount the augmented reality system of FIG.
[0043] [Figure 11] FIG. 11 is a block diagram illustrating the augmented reality system of FIG. 2 interacting with various exemplary virtual and real sounds.
[0044] [Figure 12] FIG. 12 is a block diagram illustrating one embodiment of an audio processor used within the augmented reality system of FIG.
[0045] [Figure 13] FIG. 13 is a schematic diagram of a memory for recording content data and metadata corresponding to virtual and real objects selected by the augmented reality system of FIG.
[0046] [Figure 14] FIG. 14 is a schematic of a microphone assembly and corresponding audio processing module used in the augmented reality system of FIG. 2 to preferentially receive sound from real objects.
[0047] [Figure 15a] FIG. 15a is a plan view of a directional pattern generated by the audio processor of the augmented reality system of FIG. 2 for preferentially receiving sound from two objects having a first orientation relative to an end user.
[0048] [Figure 15b] FIG. 15b is a plan view of a directional pattern generated by the audio processor of the augmented reality system of FIG. 2 for preferentially receiving sound from two objects having a second orientation relative to the end user.
[0049] [Figure 16a] FIG. 16a is a block diagram of objects distributed within the original spatial environment for an end user.
[0050] [Figure 16b] FIG. 16b is a block diagram of the objects of FIG. 17a distributed within a new spatial environment for an end user.
[0051] [Figure 17] FIG. 17 is a flow diagram illustrating one method of operating the augmented reality system of FIG. 2 to select and record audio and video of virtual and real objects.
[0052] [Figure 18] FIG. 18 is a flow diagram illustrating one method of operating the augmented reality system of FIG. 2 to play back the audio and video recorded in FIG. 17 within a new spatial environment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0053] The following description relates to a display system and method to be used within an augmented reality system. However, it should be understood that while the present invention is well suited for use in an augmented reality system, the present invention in its broadest aspects may not be so limited. For example, the present invention may be applied to a virtual reality system. Thus, although often described herein in terms of an augmented reality system, the present teachings should not be limited to such systems of such use. An augmented reality system may be operated, for example, in the context of video games, video conferencing with a combination of virtual and real people, or watching movies.
[0054] The augmented reality system described herein allows an end user to record audio data originating from at least one object (either virtual or real) that is persistently selected by the end user. Such recorded audio data can be subsequently played back by the same or a different end user. Sound originating from the recorded audio data may be played back to the same or a different end user in the real environment in which the audio data was originally recorded. In addition to recording the content of the audio data, metadata characterizing the environment in which the audio content was originally recorded and the end user's head pose may be recorded in association with such audio data such that during playback, the audio may be re-rendered and converted into spatialized sound that is auditorily experienced in the same manner as the end user auditorily experienced the spatialized sound during the original recording. Optionally, the audio may be re-rendered and converted into spatialized sound for perception by the same or a different end user in a new virtual or real environment such that the same or a different end user may have an auditory experience appropriate to the new environment. The audio data may be recorded in synchrony with video data originating from virtual and real objects in the surrounding environment.
[0055] The augmented reality system described herein may be operated to provide images of virtual objects mixed with real (or physical) objects in an end user's field of view, and to provide virtual sounds originating from virtual sources (either within or outside the field of view) mixed with real sounds originating from real (or physical) sources (either within or outside the field of view). To this end, one embodiment of an augmented reality system 100 constructed in accordance with the present invention will now be described with reference to Figures 2 and 3. The augmented reality system 100 comprises a display subsystem 102, which includes a display screen 104 and a projection subsystem (not shown) that projects images onto the display screen 104.
[0056] In the illustrated embodiment, the display screen 104 is a partially transparent display screen through which real objects in the surrounding environment can be seen by the end user 50, and on which images of virtual objects may be displayed. The augmented reality system 100 further comprises a frame structure 106, worn by the end user 50, carrying the partially transparent display screen 104 such that the display screen 104 is positioned in front of the eye 52 of the end user 50, specifically within the field of view of the end user 50 between the eye 52 of the end user 50 and the surrounding environment.
[0057] The display subsystem 102 is designed to present to the eyes 52 of the end user 50 a light-based radiation pattern that can be comfortably perceived as an augmentation to physical reality with a high level of image quality and three-dimensional perception, and is also capable of presenting two-dimensional content. The display subsystem 102 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene.
[0058] In an alternative embodiment, the augmented reality system 100 may employ one or more imagers (e.g., cameras) to capture images of the surrounding environment and convert them into video data, which can then be mixed with the video data representing the virtual objects, in which case the augmented reality system 100 may display an image representing the mixed video data to the end user 50 on an opaque display surface.
[0059] Further details describing the display subsystem are provided in U.S. Provisional Patent Application No. 14 / 212,961, entitled "Display Subsystem and Method," and U.S. Provisional Patent Application No. 14 / 331,216, entitled "Planar Waveguide Apparatus With Diffraction Element(s) and Subsystem Employing Same," which are expressly incorporated herein by reference.
[0060] The augmented reality system 100 further comprises one or more speakers 108 to allow the end user 50 to hear sounds from real objects directly, while presenting only sounds from virtual objects to the end user 50. In an alternative embodiment, the augmented reality system 100 may comprise one or more microphones (not shown) to capture real sounds originating from the surrounding environment and convert it into audio data, which can be mixed with the audio data from the virtual objects, in which case the speakers 108 may convey sounds representing the mixed audio data to the end user 50.
[0061] In either case, the speaker 108 is carried by the frame structure 106 such that the speaker 108 is positioned adjacent (in or around) the ear canal of the end user 50, for example as an ear bud or headphone. The speaker 108 may provide stereo / adjustable sound control. Although the speaker 108 is described as being positioned adjacent to the ear canal, other types of speakers that are not located adjacent to the ear canal can also be used to communicate sound to the end user 50. For example, the speaker may be placed at a distance from the ear canal, for example using bone conduction technology. 4, multiple spatialized speakers 108 (e.g., four speakers 108-1, 108-2, 108-3, and 108-4) may be centered about the head 54 of the end user 50 and configured to receive sound from the left, right, front, and back of the head 54 and directed toward the left and right ears 56 of the end user 50. Further details regarding spatialized speakers that may be used for an augmented reality system are described in U.S. Provisional Patent Application No. 62 / 369,561, entitled "Mixed Reality System with Spatialized Audio," which is expressly incorporated herein by reference.
[0062] Importantly, the augmented reality system 100 is configured to allow the end user 50 to select one, some, or all objects (either virtual or real) for recording of sounds only from these selected objects. To this end, the augmented reality system 100 further comprises an object selection device 110 configured to select, in response to an input from the end user 50, one or more real objects (i.e., real objects from which real sounds originate) and virtual objects (i.e., virtual objects from which virtual sounds originate) for recording sounds therefrom. The object selection device 110 may be designed to individually select real or virtual objects in the field of view of the end user 50 and / or to globally select a subset or all of the real or virtual objects in the field of view of the end user 50. The object selection device 110 may also be configured to deselect one or more previously selected real or virtual objects in response to an additional input from the end user 50. In this case, the object selection device 110 may be designed to deselect real or virtual objects in the same manner that they were previously selected. In either case, the concrete object is persistently selected, meaning that the concrete object remains selected until it is intentionally deselected.
[0063] In one embodiment, the display subsystem 102 may display a three-dimensional cursor within the field of view of the end user 50, which can be displaced within the field of view of the end user 50 in response to input to the object selection device 110 for purposes of selecting a concrete real or virtual object within the augmented reality scene.
[0064] For example, as shown in FIG. 5, four virtual objects (V1-V4) and two real objects (R1-R2) are located within the field of view 60 of the display screen 104. The display subsystem 102 may display a 3D cursor 62 within the field of view 60, which is illustrated as taking the form of a circle. The 3D cursor 62 may be moved over one of the objects, in this case virtual object V3, in response to input by the end user 50 to the object selection device 110, thereby associating the 3D cursor 62 with that object. The associated object may then be selected in response to additional input by the end user 50 to the object selection device 110. The associated object or even the 3D cursor 62 itself may be highlighted (e.g., a change in color or shading) to provide visual feedback that a concrete object (in this case virtual object V3) has been associated with the 3D cursor 62 and is ready for selection. Once selected, the object may remain highlighted until deselected. Of course, other objects in the augmented reality scene 4, including real objects, can be selected instead of or in addition to the virtual object V3, by placing the 3D cursor 62 over any of these other objects and selecting the object within the 3D cursor 62. Also, although the 3D cursor 62 in Figure 5 takes the form of a circle, it should be understood that the 3D cursor 62 can be any shape, including an arrow, that can be used by the end user 50 to point to a specific object. Any previously selected object in the field of view 60 can be deselected by moving the 3D cursor 62 over the previously selected object and deselecting it.
[0065] The object selection device 110 may take the form of any device that allows the end user 50 to move the 3D cursor 62 over a concrete object and subsequently select that concrete object. In one embodiment, the object selection device 110 takes the form of a conventional physical controller, such as a mouse, touchpad, joystick, directional buttons, etc., that may be physically manipulated to move the 3D cursor 62 over and "click" on concrete objects to select the concrete objects.
[0066] In another embodiment, the object selection device 110 may include a microphone and corresponding voice interpretation module that may respond to voice commands to move the 3D cursor 62 over specific objects and then select the specific object. For example, the end user 50 may issue a directional command, such as "move left" or "move right," to move the 3D cursor 62 incrementally over the specific objects and then issue a command such as "select" to select the specific object.
[0067] In yet another embodiment, the object selection device 110 may comprise one or more cameras (e.g., a forward-facing camera 112) mounted on the frame structure 106 and a corresponding processor (not shown) capable of tracking physical gestures (e.g., finger movements) by the end user 50 that correspondingly move the 3D cursor 62 across the concrete object for selection of the concrete object. For example, the end user 50 may use a finger to "drag" the 3D cursor 62 in the field of view 60 across the concrete object and then "tap" the 3D cursor 62 to select the concrete object. Alternatively, the forward-facing camera 112 may be employed to detect or infer, for example, a center of attention of the end user 50 based, at least in part, on the orientation of the head 54 of the end user 50 that correspondingly moves the 3D cursor 62 across the concrete object for selection of the concrete object. For example, an end user 50 may move their head 50 and "drag" the 3D cursor 62 in the field of view 60 over a specific object, and then quickly nod their head 50 to select the specific object.
[0068] In yet another embodiment, the object selection device 110 may comprise one or more cameras (e.g., a rearward facing camera 114 (shown in FIG. 2)) and a corresponding processor that tracks the eye 52 of the end user 50, and in particular the direction and / or distance at which the end user 50 is focused, which correspondingly moves the 3D cursor 62 across the specific object for selection of that specific object. The rearward facing camera 114 may track the angular position (the direction one or both eyes are pointing), eye blinks, and focal depth of the end user 50's eye 52 (by detecting eye convergence). For example, the end user 50 may move their eye 54 within the field of view, "drag" the 3D cursor across the specific object, and then blink and select the specific object. Such eye tracking information may be determined, for example, by projecting light onto the end user's eye and detecting a return or reflection of at least a portion of the projected light. Further details discussing eye tracking devices are provided in U.S. Provisional Patent Application No. 14 / 212,961, entitled "Display Subsystem and Method," U.S. Patent Application No. 14 / 726,429, entitled "Methods and Subsystem for Creating Focal Planes in Virtual and Augmented Reality," and U.S. Patent Application No. 14 / 205,126, entitled "Subsystem and Method for Augumented and Virtual Reality," which are expressly incorporated herein by reference.
[0069] In alternative embodiments, the object selection device 110 may combine a conventional physical controller, a microphone / voice interpretation module, and / or a camera to move and use the 3D cursor 62 to select an object. For example, a physical controller, finger gestures, or eye movements can be used to move the 3D cursor 62 over a specific object, and a voice command can be used to select the specific object.
[0070] Rather than using the 3D cursor 62 to select an object in the field of view of the end user 50, a concrete object may be selected by semantically identifying the concrete object or by selecting the object via a menu displayed to the end user 50, where the object does not need to be in the field of view of the end user 50. In this case, the object selection device 110 takes the form of a microphone and a voice interpretation module where the concrete object is semantically identified to translate a verbal command provided by the end user 50. For example, if the virtual object V3 corresponds to a drum, the end user 50 may utter "select drum" and in response the drum V3 will be selected. To facilitate the selection of an object corresponding to a verbal command, semantic information identifying all relevant objects in the field of view is preferably stored in a database such that a description of the object verbally expressed by the end user 50 can be matched to a description of the object stored in the database. While metadata, including semantic information, can be pre-associated with virtual objects in a database, real objects in the field of view may be pre-mapped and associated with semantic information in the manner described in U.S. patent application Ser. No. 14 / 704,800, entitled "Method and System for Inserting Recognized Object Data into a Virtual World," which is expressly incorporated herein by reference.
[0071] Alternatively, a concrete object may be selected without using the 3D cursor 62, simply by pointing at or "clicking" on it using finger gestures. In this case, the object selection device 110 may comprise one or more cameras (e.g., a forward-facing camera 114) and a corresponding processor that tracks finger gestures for selection of concrete objects. For example, the end user 50 may select a concrete object (in this case, virtual object V3) simply by pointing at it, as shown in FIG. 6. In another embodiment, a concrete object may be selected without using the 3D cursor 62, simply by forming a circle or partial circle using at least two fingers (e.g., index finger and thumb), as shown in FIG. 7.
[0072] Although the 3D cursor 62 has been described as being used to select only one object at a time, in an alternative or optional embodiment, the 3D cursor 62 may be used to select multiple objects at once. For example, as illustrated in FIG. 8, a line 64 may be drawn using the 3D cursor 62 around a group of objects, e.g., around real object R1 and virtual objects V3 and V4, thereby selecting these group of objects. The 3D cursor 62 may be controlled using the same means as described above, e.g., to select the objects individually. Alternatively, a line may be drawn around a group of objects without using the 3D cursor 62, e.g., by using finger gestures.
[0073] In an alternative embodiment, a group of objects within a predefined angular range of the field of view of the end user 50 may be selected, in which case the object selection device 110 may take the form of, for example, a single physical or virtual selection button that may be actuated by the end user 50 to select these objects. The angular range of the field of view may be predefined by the end user 50 or may be preprogrammed into the augmented reality system 100. For example, as shown in FIG. 9, an angular range 66 of 60 degrees (±30 degrees from the center of the field of view) is shown in the context of a 120 degree field of view 60. All objects within the angular range 64 of the field of view 60 (in this case virtual objects V1, V2, and V3) may be globally selected in response to actuation of the selection button, while all objects outside the angular range 64 of the field of view 60 (in this case real objects R1 and R2 and virtual object V4) would not be selected in response to actuation of the selection button. In one embodiment, the end user 50 may modify the angular range, for example, by dragging one or both of the edges that define the angular range toward or away from the centerline of the field of view 60 (indicated by the arrows). The end user 50 may, for example, adjust the angular range from a minimum of 0 degrees to the entire field of view (e.g., 120 degrees). Alternatively, the angular range 64 of the field of view 60 may be pre-programmed without the ability for the end user 50 to adjust it. For example, all objects within the entire field of view 60 may be selected in response to actuation of a select button.
[0074] The augmented reality system 100 further comprises one or more microphones configured to convert sounds from real objects in the surrounding environment into audio signals. In particular, the augmented reality system 100 comprises a microphone assembly 116 configured to preferentially receive sounds in a particular direction and / or at a particular distance corresponding to the direction and distance of one or more real objects selected by the end user 50 via the object selection device 110. The microphone assembly 116 comprises an array of microphone elements 118 (e.g., four microphones) mounted on the frame structure 106 as illustrated in FIG. 2 (only two are shown). Details regarding the microphone assembly 116 will be described in more detail below. The augmented reality system 100 further comprises a dedicated microphone 122 configured to convert speech of the end user 50 into audio signals, for example to receive commands or dictations from the end user 50.
[0075] The augmented reality system 100 tracks the positions and orientations of selected real objects within the known coordinate system so that, even as the positions or orientations of the selected real objects relative to the augmented reality system change, sounds originating from these real objects may be preferentially and continually sensed compared to non-selected real objects by the microphone assembly 116. The positions and locations of all virtual objects within the known coordinate system are typically "known" to (i.e., recorded in) the augmented reality system 100 and therefore generally do not need to be actively tracked.
[0076] In the illustrated embodiment, the augmented reality system 100 employs a spatialized audio system that renders and presents spatialized audio corresponding to virtual objects at known virtual locations and orientations in real and physical three-dimensional (3D) space, causing the sound to appear to the end user 50 as originating from the virtual location of the real object to affect the clarity or realism of the sound. The augmented reality system 100 tracks the position of the end user 50 to more accurately render the spatialized audio, such that the audio associated with various virtual objects appears to originate from that virtual location. Additionally, the augmented reality system 100 tracks the head pose of the end user 50 to more accurately render the spatialized audio, such that directional audio associated with various virtual objects appears to propagate in the appropriate virtual direction for the individual virtual object (e.g., from the virtual character's mouth, rather than from behind the virtual character's head). Additionally, the augmented reality system 100 takes into account other real physical and virtual objects when rendering the spatialized audio, such that the audio associated with various virtual objects appears to properly reflect from, or be occluded or obstructed by, the real physical and virtual objects.
[0077] To this end, the augmented reality system 100 further comprises a head / object tracking subsystem 120 for tracking the position and orientation of the head 54 of the end user 50 relative to the virtual three-dimensional scene, and for tracking the position and orientation of real objects relative to the head 54 of the end user 50. For example, the head / object tracking subsystem 120 may comprise one or more sensors configured to collect head pose data (position and orientation) of the end user 50, and a processor (not shown) configured to determine the head pose of the end user 50 within a known coordinate system based on the head pose data collected by the sensors 120. The sensors may include one or more of an image capture device (such as a visible and infrared light camera), an inertial measurement unit (including an accelerometer and gyroscope), a compass, a microphone, a GPS unit, or a wireless device. In the illustrated embodiment, the sensor comprises a forward-facing camera 112 (shown in FIG. 2). When mounted on the head in this manner, the forward-facing camera 120 is particularly suitable for capturing information indicative of the distance and angular position of the end user's 50 head 54 (i.e., the direction the head is pointing) relative to the environment in which the end user 50 is located. Head orientation may be detected in any direction (e.g., up / down, left, right relative to the end user's 50 frame of reference). As will be explained in more detail below, the forward-facing camera 114 is also configured to obtain video data of real objects in the surrounding environment and facilitate the video recording function of the augmented reality system 100. Cameras may also be provided to track real objects in the surrounding environment. The frame structure 106 may be designed such that cameras may be mounted on the front and back of the frame structure 106. In this manner, an array of cameras may surround the end user's 50 head 54 and cover all directions of relevant objects.
[0078] The augmented reality system 100 further comprises a 3D database 124 configured for storing virtual three-dimensional scenes, which comprise virtual objects (both content data of the virtual objects and absolute metadata associated with these virtual objects, e.g., absolute position and orientation of these virtual objects within the 3D scene) and virtual objects (both content data of the virtual objects and absolute metadata associated with these virtual objects, e.g., volume and absolute position and orientation of these virtual objects within the 3D scene, as well as the spatial acoustics surrounding each virtual object, room dimensions, wall / floor materials, etc., including any virtual or real objects in the vicinity of the virtual source).
[0079] The augmented reality system 100 further comprises a control subsystem that records video data resulting from virtual and real objects appearing within the field of view, as well as audio data resulting only from those virtual and real objects selected by the end user 50 via the object selection device 110. The augmented reality system 100 may also record metadata associated with the video and audio data so that the synchronized video and audio can be accurately re-rendered during playback.
[0080] To this end, the control subsystem comprises a video processor 126 configured to obtain video content and absolute metadata associated with the virtual objects from a 3D database 124, obtain head pose data of the end user 50 (which will be used to localize absolute metadata for the video to the head 54 of the end user 50, as will be explained in further detail below) from the head / object tracking subsystem 120, and then render video therefrom, which is communicated to the display subsystem 102 for conversion into images blended with images originating from real objects in the surrounding environment within the field of view of the end user 50. The video processor 126 is also configured to obtain video data originating from real objects in the surrounding environment from a forward-facing camera 112, which will be subsequently recorded along with the video data originating from the virtual objects, as will be explained further below.
[0081] Similarly, the audio processor 128 is configured to obtain audio content and metadata associated with the virtual objects from the 3D database 124, and head pose data of the end user 50 (which will be used to localize absolute metadata for the audio to the head 54 of the end user 50, as described in further detail below) from the head / object tracking subsystem 120, and then render the spatialized audio therefrom, which is transmitted to the speakers 108 for conversion into spatialized sound mixed with sounds originating from real objects in the surrounding environment.
[0082] The audio processor 128 is also configured to obtain from the microphone assembly 116 audio data originating solely from the selected real object in the surrounding environment, which will then be recorded along with spatialized audio data from the selected virtual object, along with any resulting metadata (e.g., position, orientation, and volume data) localized to the head 54 of the end user 50 for each virtual object, and along with global metadata (e.g., volume data set globally by the augmented reality system 100 or the end user 50), as will be further described below.
[0083] The augmented reality system 100 further comprises a memory 130, a recorder 132 configured to store the video and audio in the memory 130, and a player 134 configured to retrieve the video and audio from the memory 130 for subsequent playback to the end user 50 or other end users. The recorder 132 obtains spatialized audio data (both audio content audio data and metadata) from the audio processor 128 corresponding to the selected virtual and real objects, stores the audio data in the memory 130, and further obtains video data (both video content data and metadata) corresponding to the virtual and real objects that match the selected virtual and real objects. Although the player 134 is illustrated as being located in the same AR system 100 in which the recorder 132 and the memory 130 are located, it should be understood that the player may be located in a third party AR system or even on a smartphone or computer that plays back video and audio previously recorded by the AR system 100.
[0084] The control subsystem, which implements the functions of video processor 126, audio processor 128, recorder 132, and player 134, may take any of a wide variety of forms and may include a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), such as field PGAs (FPGAs), and / or programmable logic controllers (PLUs).
[0085] The functions of the video processor 126, the audio processor 128, the recorder 132, and the player 134 may each be performed by a single integrated device, at least some of the functions of the video processor 126, the audio processor 128, the recorder 132, and / or the player 134 may be combined into a single integrated device, or the functions of each of the video processor 126, the audio processor 128, the recorder 132, or the player 134 may be distributed among several devices. For example, the video processor 126 may include a graphics processing unit (GPU) that obtains video data of virtual objects from the three-dimensional database 124 and renders composite video frames therefrom, and a central processing unit (CPU) that obtains video frames of real objects from the forward-facing camera 112. Similarly, the audio processor 128 may include a digital signal processor (DSP) that processes audio data obtained from the microphone assembly 116 and the user microphone 122, and a CPU that processes audio data obtained from the three-dimensional database 124. The recording function of recorder 132 and the playback function of player 134 may be performed by a CPU.
[0086] Additionally, various processing components of the augmented reality system 100 may be physically contained within a distributed subsystem. For example, as illustrated in Figures 10a-10d, the augmented reality system 100 includes a local processing and data module 150 that is operatively coupled to components mounted on the head 54 of the end user 50 (e.g., the projection subsystem of the display subsystem 102, the microphone assembly 116, the speaker 104, and the cameras 114, 118), such as by wired leads or wireless connectivity 152. The local processing and data module 150 may be mounted in a variety of configurations, such as fixedly attached to a frame structure 106 (Figure 10a), fixedly attached to a helmet or hat 106a (Figure 10b), built into headphones, removably attached to a torso 58 of the end user 50 (Figure 10c), or removably attached to a waist 59 of the end user 50 in a belt-type configuration (Figure 10d). The augmented reality system 100 further includes a remote processing module 154 and a remote data repository 156 operably coupled to the local processing and data module 150, such as by wired conductors or wireless connectivity 158, 160, such that the remote modules 154, 156 are operably coupled to each other and available as resources to the local processing and data module 150.
[0087] The local processing and data module 150 may comprise a power efficient processor or controller and digital memory such as flash memory, both of which may be utilized to aid in the processing, caching, and storage of data captured from the sensors and / or obtained and / or processed using the remote processing module 1544 and / or the remote data repository 156, possibly for passing to the display subsystem 102 after processing or retrieval. The remote processing module 154 may comprise one or more relatively powerful processors or controllers configured to analyze and process the data and / or image information. The remote data repository 156 may comprise a relatively large scale digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all calculations are performed within the local processing and data module 150, allowing for fully autonomous use from any remote module.
[0088] The couplings 152, 158, 160 between the various components described above may include one or more wired interfaces or ports for providing wired or optical communication, or one or more wireless interfaces or ports via RF, microwave, IR, and the like for providing wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless, except for optical fibers used within the display subsystem 102. In still further implementations, the wired and wireless communication options may differ from those illustrated in FIGS. 10a-10d. Thus, the particular options for wired or wireless communication should not be considered limiting.
[0089] In the illustrated embodiment, the light sources and drive electronics (not shown) of the display subsystem 102, the processing components of the object tracking subsystem 120 and the object selection device 110, and the DSP of the audio processor 128 may be contained within the local processing and data module 150. The GPU of the video processor 126 and the CPUs of the video processor 126 and audio processor 128 may be contained within the remote processing module 154, although in alternative embodiments these components, or portions thereof, may be contained within the local processing and data module 150. The 3D database 124 and memory 130 may be associated with a remote data repository 156.
[0090] The audio processor 128 illustrated in Figure 3 will be described in further detail in processing and recording audio data from virtual and real objects selected by the end user 50. In an exemplary scenario illustrated in Figure 11, an end user 50 (e.g., parents) wants to record sounds from a virtual drummer V2 object and a real vocalist R2, e.g., a four-piece band including a child, a virtual guitarist V3, and a virtual bassist V4, wants to watch news or sports on a virtual television V1 without recording sounds from the virtual television, and also does not want to record sounds from a real kitchen R1 where, e.g., someone is cooking.
[0091] In the embodiment illustrated in Fig. 12, the functionality of the audio processor 128 is distributed between a CPU 180, which processes the audio originating from the virtual objects, and a DSP 182, which processes the audio originating from the real objects. The CPU 180 comprises one or more special effects modules 184 (in this case special effects modules 1-n) configured to generate spatialized audio data EFX-V1 to EFX-Vn corresponding to the respective virtual objects V1 to Vn. To this end, the special effects modules 184 generate audio content data AUD-V1 to AUD-Vn and absolute metadata MD a -V1~MD a -Vn from the 3D database 124 and head pose data from the head / object tracking subsystem 120, and calculates the absolute metadata MD based on the head pose data. a -V1~MD a -Vn is localized to the head 54 of the end user 50 and local metadata (e.g., position, orientation, and volume data) is applied to the audio content data to generate spatialized audio data for the virtual objects V1-Vn.
[0092] The CPU 180 further comprises a mixer 186 configured to mix the spatialized audio data EFX-V1 to EFX-Vn received from the individual special effects modules 184 to obtain mixed audio data EFX, and a global special effects module 188 configured to apply global metadata MD-OUT (e.g. global volume) to the mixed spatialized audio data to obtain final spatialized audio AUD-OUT EFX, which is output to the speakers 108 through multiple sound channels.
[0093] Importantly, the special effects module 184 is configured to transmit audio content data resulting from virtual objects selected by the end user 50 via the object selection device 110 and metadata (local and / or absolute) corresponding to these selected virtual objects to the recorder 132 for storage in the memory 130 (shown in FIG. 2), and the global special effects module 188 is configured to transmit global metadata MD-OUT to the recorder 132 for storage in the memory 130. In an exemplary embodiment, the virtual audio content data AUD-V2 (i.e., virtual drummer), AUD-V3 (i.e., virtual guitarist), AUD-V4 (i.e., virtual bassist) are selected for recording, while the audio content data AUD-V1 (i.e., virtual TV) is not selected for recording. Thus, the audio content data AUD-V2, AUD-V3, and AUD-V4 and the corresponding local metadata MD-V2, MD-V3, and MD-V4 are stored in memory 130, as shown in FIG.
[0094] In an alternative embodiment, instead of or in addition to storing the audio content data and corresponding local / absolute metadata and global metadata from the selected virtual objects individually in memory 130, CPU 180 additionally outputs spatialized audio generated by mixing spatialized audio data EFX-V2, EFX-V3, EFX-V4 corresponding only to the selected virtual objects AUD-V2, AUD-V3, and AUD-V4, applying global metadata MD-OUT to this mixed spatialized audio data, and obtaining spatialized audio including audio only from the selected virtual objects AUD-V2, AUD-V3, and AUD-V4. However, in this case, additional audio mixing functionality would need to be incorporated into CPU 180.
[0095] The DSP 182 is configured to process audio signals obtained from the microphone assembly 116 and output audio signals that preferentially represent sounds received by the microphone assembly 116 from a particular direction, in this case from the direction of each real object selected by the end user 50 via the object selection device 110. Because the position and / or orientation of the real object may move relative to the head 54 of the end user 50, real object tracking data may be received from the head / object tracking subsystem 120 such that any changes in the position and / or orientation of the real object relative to the head 54 of the end user 50 may be taken into account and the DSP 182 may dynamically modify the audio output to preferentially represent sounds received by the microphone assembly 116 from the direction of the relatively moving real object. For example, if the end user 50 moves their head 54 90 degrees counterclockwise relative to the orientation of the head 54 when a real object is selected, the preferential direction of the audio output from the DSP 182 may be dynamically shifted 90 degrees clockwise.
[0096] With reference to FIG. 14, the microphone elements 118 of the microphone assembly 116 take the form of a phased array of microphone elements (in this case microphone elements M1-Mn) configured to detect and convert ambient sound signals into audio signals, respectively. In the illustrated embodiment, the microphone elements 118 are digital in nature and thus convert the ambient sound signals into digital audio signals, in this case pulse density modulated (PDM) signals. Preferably, the microphone elements 118 are spaced apart from each other to maximize the directionality of the audio output. For example, as shown in FIG. 2, two of the microphone elements 118 may be mounted on each arm of the frame structure 106, although more than two microphone elements 118, such as four, may be mounted on each arm of the frame structure 106. Alternatively, the frame structure 106 may be designed such that the microphone elements 118 may be mounted on the front and back of the frame structure 106. In this manner, the array of microphone elements 118 may surround the head 54 of the end user 50, covering all directions of potential sources of sound.
[0097] The microphone assembly 116 further comprises a number of digital microphone interfaces (DMICs) 190 (in this case, one DMIC1-DMICn for each microphone element M), each configured to receive a respective digital audio signal from a corresponding microphone element 118, perform a digital filtering operation referred to as "decimation," and convert the digital audio signal from a PDM format to a more easily manipulable pulse code modulation (PCM). Each DMIC 190 also performs a fixed gain control on the digital audio signal.
[0098] The DSP 182 comprises a number of audio processing modules 200 each configured to process the digital audio signals output by the microphone assembly 116 and to output a directional audio signal AUD-R (one of the directional audio signals AUD-R1 to AUD-Rm) preferentially representing the sounds received by the microphone assembly 116 in the direction of a selected real object (one of R1 to Rm). The directional audio signals AUD-R1 to AUD-Rm output by the individual audio processing modules 200 are combined into a directional audio output AUD-OUT MIC, which preferentially represents the sounds originating from all selected real objects. In the illustrated embodiment, the DSP 182 creates one instance of the audio processing module 200 for each real object selected by the end user 50 via the object selection device 110.
[0099] To this end, each audio processing module 200 comprises processing parameters in the form of a plurality of delay elements 194 (in this case one delay element D1-Dn for each microphone element M), a plurality of gain elements 196 (in this case one gain element G1-Gn for each microphone element M), and a summer 198. The delay elements 194 each apply a delay factor to the amplified digital signal received from a corresponding gain amplifier 192 of the microphone assembly 116, and the gain elements 196 each apply a gain factor to the delayed digital signal. The summers 198(S) sum the gain adjusted and delayed signals to generate respective individual directional audio signals AUD-R.
[0100] The microphone elements 118 are spatially arranged, and the delay elements 194 and gain elements 196 of each audio processing module 200 are applied to the digital audio signals received from the microphone assemblies 116 in a manner that results in reception of ambient sound according to a directional polar pattern (i.e., sounds arriving from a particular angular direction or directions will be emphasized more than sounds arriving from other angular directions). The DSP 182 is configured to modify the directivity of the directional audio signals AUD-R1-AUD-Rm, and thus the combined directional audio output AUD-OUT MIC, by varying the delay coefficients of the delay elements 194 and the gain coefficients of the gain elements 196.
[0101] It can therefore be understood that the directivity of the audio output AUD-OUT MIC is modified based on the selected real object, e.g. the direction or directions from which sound is preferentially received may be set along the direction of the selected real object or source.
[0102] For example, referring to FIG. 15a, two specific directions D a and D. b Two real objects R along a and R b If is selected, the DSP 182 creates two instances of the audio processing module 200 and within each of these audio processing modules 200, the real object R a and R b Direction D a and D. b The receiver will select the individual delay and gain coefficients for all of the delay elements 194 and gain elements 196 in each audio processing module 200 such that a receive gain pattern is generated having two lobes that are aligned with the real object R relative to the head 54 of the end user 50. a and R b If the orientation of the real object R changes, a and R bThe particular direction of may change, in which case the DSP 182 determines whether the receive gain pattern is in the direction D, as shown in FIG. c and D. d 2. In this embodiment, the delay and gain elements 194 and 196 in each audio processing module 200 may be selected to have two lobes that are aligned with each other.
[0103] To facilitate such dynamic modification of the directivity of the audio output AUD-OUT MIC, different sets of delay / gain values and corresponding preferred directions may be stored in memory 130 for access by DSP 182. That is, DSP 182 matches the direction of each selected real object R with the closest directivity value stored in memory 130 and selects a corresponding set of delay / gain coefficients for that selected direction.
[0104] It should be noted that although the microphone elements 118 are described as digital, the microphone elements 118 may alternatively be analog. Additionally, while the delay elements 194, gain elements 196, and summers 198 are disclosed and illustrated as software components resident within the DSP 182, any one or more of the delay elements 194, gain elements 196, and summers 198 may comprise analog hardware components resident outside of, but under the control of, the DSP 182. However, the use of a software-based audio processing module 200 allows sounds from several distinct real objects to be received and processed preferentially at the same time.
[0105] Referring back to FIG. 12, the DSP 182 also receives voice data from the user microphone 122 and combines it with the directional audio output AUD-OUT MIC. In an optional embodiment, the DSP 182 is configured to perform acoustic echo cancellation (AEC) and noise suppression (NS) functions on sounds from the speaker 108 that originate from virtual objects. That is, the microphone assembly 116 may sense sounds emitted by the speaker 108 even if the direction from which the sound is preferentially received may not coincide with the speaker 108. To this end, the spatialized audio data output by the global special effects module 188 to the speaker 108 is also input into the DSP 182, which uses the spatialized audio data to suppress the resulting sounds (considered noise) output by the speaker 108 to the microphone assembly 116 and to cancel any echoes resulting from feedback from the speaker 108 to the microphone assembly 116.
[0106] Importantly, the DSP 182 also outputs a directional audio output AUD-OUT for storage in memory 130 (shown in FIG. 2) as audio content data. 13. The directional audio output AUD-OUT MIC is configured to transmit the MIC and the local metadata (e.g., the location and orientation of the real object from which the directional audio output AUD-OUT MIC originates) to the recorder 132. In the exemplary embodiment illustrated in FIG. 11, the local metadata corresponds to the real object AUD-R2 (i.e., the actual vocals). Thus, the directional audio output AUD-OUT MIC (which preferentially corresponds to the real object AUD-R2) and the corresponding local metadata MD-R2 are stored in the memory 130, as illustrated in FIG.
[0107] In an optional embodiment, the directional audio output AUD-OUT MIC (which may be spatialized) may be input into speaker 108 or other speakers for playback to end user 50. The directional audio output AUD-OUT MIC may be spatialized in the same manner as spatialized audio data originating from virtual sources appears to end user 50, such that sounds originate from the location of real objects to affect the clarity or realism of the sounds. That is, local metadata (e.g., the location and orientation of the real object from which the directional audio output AUD-OUT MIC preferentially originated) may be applied to the directional audio output AUD-OUT MIC to obtain the spatialized audio data.
[0108] In another optional embodiment, the sounds originating from the real object or even the virtual object selected by the end user 50 may be profiled. In particular, the DSP 182 may analyze and compare the characteristics of the sounds originating from the selected object with the characteristics of the sounds originating from other real objects to determine the type of the target sound. The DSP 182 may then include all the audio data originating from these real objects in the directional audio output AUD-OUT MIC for recording by the recorder 132 into the memory 130 (shown in FIG. 2) as desired. For example, if the end user 50 selects any of the music objects (AUD-V2, AUD-V3, AUD-V4, AUD-R2), the DSP 182 may control the microphone assembly 116 to preferentially sense all the music real objects.
[0109] In the illustrated embodiment, the DSP 182 continues to output the directional audio output AUD-OUT MIC to the recorder 130 for recording in memory 130 even if the real object 198 selected by the end user 50 moves out of the field of view of the display subsystem 102 (as indicated by real object tracking data received from the head / object tracking subsystem 120). In an alternative embodiment, the DSP 182 ceases outputting the directional audio output AUD-OUT MIC to the recorder 130 for recording in memory 130 as soon as the real object 198 selected by the end user 50 moves out of the field of view of the display subsystem 102, and resumes outputting the directional audio output AUD-OUT MIC to the recorder 130 for recording in memory 130 as soon as the real object 198 selected by the end user 50 moves back into the field of view of the display subsystem 102.
[0110] Similar to how the audio processor 128 (in the illustrated embodiment, the CPU 180 and the DSP 182) transmits audio content data (in the exemplary case, audio content data AUD-V2, AUD-V3, and AUD-V4, and AUD-MIC) and local metadata (in the exemplary case, MD-V2, MD-V3, MD-V4, and MD-R2) and global metadata (MD-OUT) resulting from selected virtual and real objects to the recorder 132 for storage in the memory 130, the video processor 126 may transmit video content data (in the exemplary case, video content data VID-V2, VID-V3, VID-V4, and VID-R2) resulting from the virtual and real objects, as illustrated in FIG. 13. In the case of virtual objects, the video processor 126 simply obtains the virtual objects from the 3D database 124 without further processing and transmits these virtual objects to the recorder 132 for storage in the memory 130. In the case of real objects, the video processor 126 may extract or "cut out" any of the selected real objects from the video obtained from the camera 112 and store these real objects as virtual objects in the memory 130. In the exemplary case illustrated in FIG. 11, the video for the actual vocalist R2 may be recorded as the virtual object VID-R2. In an optional embodiment, the video processor 126 transmits the entire video obtained from the camera 112 (including the video corresponding to the unselected virtual and real objects) to the recorder 132 for storage in the memory 130.
[0111] The player 134 is configured to play back the video and / or audio recorded in the memory 130 to a playback user 50' (shown in FIG. 16a), who may be the original end user 50 who recorded the video / audio or a third party user. The audio / video may be selectively played back by the player 134 in response to commands given by the playback user 50', e.g., voice commands via the user microphone 122. For example, the playback user 50' may use a "virtual audio on / off" command to turn virtual audio playback on or off, or a "display on / off" command to turn virtual video playback on or off, or a "real audio on / off" command to turn real audio playback on or off.
[0112] In the illustrated embodiment, the audio processor 128 reads the audio content data and metadata (corresponding to selected virtual and real objects) from the memory 130, renders spatialized audio from the audio content data and metadata, and communicates the spatialized audio to the player 134 for playback to the user 50' via the playback speakers 108. In alternative embodiments in which the mixed spatialized audio data (instead of the content and metadata) is stored, the player 134 may simply obtain the audio data from the memory 130 for playback to the playback user 50' without re-rendering or otherwise further processing the audio data.
[0113] Further, in the illustrated embodiment, the video processor 126 reads the video content data and metadata (corresponding to the selected virtual and real objects), renders the video from the video content data and metadata, and communicates the video to the player 134 for playback to the playback user 50' via the display subsystem 102 synchronized with the playback of the audio via the speaker 108. Optionally, if all of the video data captured by the camera 112 is stored, the player 134 may simply retrieve the video data from the memory 130 for playback to the playback user 50' without rendering or otherwise further processing the video data. The augmented reality system 10 may provide the playback user 50' with the option of either playing only the video corresponding to the selected virtual and real objects or playing the entire video captured by the camera 112.
[0114] In one embodiment, the current head pose of the playback user 50' is not taken into account during the playback of the video / audio. Instead, the video / audio is played back to the playback user 50' using the head pose originally detected during the recording of the video / audio data, which will be reflected in the local metadata stored in the memory 130 together with the audio / video content data, or if the mixed spatialized audio is recorded without metadata, the head pose will be reflected in the mixed spatialized audio stored in the memory 130. In this case, the playback user 50' will experience the video / audio in the same manner as the original end user 50 experienced the video / audio, except that only the audio and optionally only the video originating from the virtual and real objects selected by the original end user 50 will be played back. In this case, the playback user 50' will not be immersed in the augmented reality, since the head pose of the playback user 50' will be taken into account. Rather, the playback user 50' may experience the audio playback using a headset (so the audio will not be affected by the environment), or the playback user 50' may experience the audio playback in a quiet room.
[0115] In an alternative embodiment, the current head pose of the playback user 50' may be taken into account during the video / audio playback. In this case, the head pose of the playback user 50' during the recording of the video / audio does not need to be incorporated into the metadata stored in the memory 130 together with the video / audio content data, since the current head pose of the playback user 50' detected during the playback will be used to re-render the video / audio data. Instead, the absolute metadata stored in the memory 130 (e.g., the volume and absolute position and orientation of these virtual objects in the 3D scene, and the spatial audio surrounding each virtual object, including any virtual or real objects in the vicinity of the virtual source, room dimensions, wall / floor materials, etc.) will be localized to the head pose of the playback user 50' using the current head pose of the playback user 50' and then used to render the audio / video. Thus, the playback user 50' will be immersed in an augmented reality during the playback of the video / audio.
[0116] The playback user 50' may experience the augmented reality within the original spatial environment in which the video / audio was recorded (e.g., "same physical room"), or may experience the augmented reality within a new physical or virtual spatial environment (e.g., "different physical or virtual room").
[0117] If the augmented reality is experienced by a playback user 50' in the original spatial environment in which the video / audio was recorded, the absolute metadata associated with the selected objects does not need to be modified for accurate playback of the spatialized audio. In contrast, if the augmented reality is experienced by a playback user 50' in a new spatial environment, the absolute metadata associated with the objects may need to be modified for accurate rendering of the audio / video in the new spatial environment.
[0118] For example, in an exemplary embodiment, audio / video content from virtual objects AUD-V2 (i.e., virtual drummer), AUD-V3 (i.e., virtual guitarist), AUD-V4 (i.e., virtual bassist), and real objects (i.e., real vocalist) may be recorded in a small room 250, as illustrated in FIG. 16a. Previously recorded audio from virtual objects AUD-V2 (i.e., virtual drummer), AUD-V3 (i.e., virtual guitarist), AUD-V4 (i.e., virtual bassist), and real objects (i.e., real vocalist) may be played back in a concert hall 252, as illustrated in FIG. 16b. The augmented reality system 10 may reposition the objects anywhere within the concert hall 252, and absolute metadata may be generated or otherwise obtained, including the new position of each object within the concert hall 252 and the spatial audio surrounding each object within the concert hall 252. This absolute metadata is then localized using the current head pose of the playback user 50' and can then be used to render the audio and video within the concert hall 252 for playback to the playback user 50'.
[0119] Having described the arrangement and functionality of the augmented reality system 100, one method 300 of using the augmented reality system 100 to select at least one object and record audio and video from those selected objects will now be described with reference to FIG. 17.
[0120] Initially, the end user 50 persistently selects at least one object (e.g., real and / or virtual) in the spatial environment via the object selection device 110 (step 302). An object can be selected in the field of view 60 of the end user 50, for example, by moving a three-dimensional cursor 62 (shown in FIG. 5) within the field of view 60 of the end user 50 and selecting the object with the three-dimensional cursor 62. Alternatively, the object can be selected using hand gestures (shown in FIG. 6 or 7) or using voice commands. Multiple objects may be selected individually or globally, for example, by drawing a line 64 around the objects (shown in FIG. 8) or by defining an angular range 66 of the field of view 60 of the end user 50 (which may be less than the entire angular range of the field of view 60 of the end user 50) (shown in FIG. 9) and selecting all of the objects within the defined angular range 66 of the field of view 60 of the end user 50.
[0121] Next, audio and video content for all virtual objects in the spatial environment and absolute metadata associated with the virtual objects are obtained (step 304). Next, the current head pose of the end user 50 is tracked (step 306), and the absolute metadata is localized to the head 54 of the end user 50 using the current head pose data (step 308), and applied to the audio and video content of the virtual objects to obtain video data and spatialized audio data for all of the individual virtual objects (step 310). The spatialized audio data for all of the individual virtual objects in the 3D scene is mixed (step 312), and global metadata is applied to the mixed spatialized audio data to obtain final spatialized audio for all virtual objects in the 3D scene (step 314), which is then converted to sound for perception by the end user 50 (step 316). Next, the video data obtained in step 310 is converted to image frames for perception by the end user 50 (step 318). Next, the audio / video content and all associated metadata (both absolute and local metadata) for all virtual objects selected by the end user 50 in step 302 are recorded (step 320).
[0122] In parallel with steps 304-320, the position and / or orientation of the selected real objects relative to the head 54 of the end user 50 is tracked (step 322) and sounds originating from the selected real objects are preferentially sensed based on the tracked position and orientation of the real objects (step 324). Images of the selected real objects are then captured (step 326) and, optionally, converted into virtual video content. Audio content associated with the preferentially sensed sounds from the selected real objects and video content associated with the captured images of the selected real objects, as well as all associated metadata for each selected real object (location and orientation of the real objects), are then recorded (step 328).
[0123] One method 400 of playing previously recorded audio and video of at least one object using the augmented reality system 100 for a playback user 50' will now be described with respect to FIG. 18. Such audio and video may be pre-recorded as audio content data and video content data in the manner described in the method 300 of FIG. 17 above. The object may be real and / or virtual and may be persistently selected by the end user 50. In the exemplary method 400, the audio and video may be pre-recorded in an original spatial environment, such as a small room 250, as described with respect to FIGS. 16a and 16b, and played back in a new spatial environment different from the original spatial environment, such as a concert hall 252.
[0124] First, previously recorded audio and video content data are obtained (step 402). If the new spatial environment is at least partially virtual, additional virtual content (either audio or video) associated with the new spatial environment may also be obtained. The objects are then repositioned within the new spatial environment, which may respond to input from the playback user 50' (step 404). Absolute metadata corresponding to the objects positioned within the new spatial environment are then obtained (step 406), the head pose of the playback user 50' is tracked within the new spatial environment (step 408), and the absolute metadata is localized to the playback user 50' based on the tracked head pose of the playback user 50' (step 410). Next, audio and video are rendered from the retrieved audio and video content data based on the local metadata within the new spatial environment (step 412). The rendered audio and video are then converted into sound and image frames, respectively, for synchronous perception by the playback user 50' (step 414).
[0125] In the foregoing specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the invention. For example, the process flows described above are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the invention. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.
Claims
1. 1. A method for recreating audio-visual effects using spatialized audio in an augmented reality environment, comprising: identifying audio data captured within a first augmented reality environment and first metadata associated with the audio data, the audio data is perceived by a first user at a first user location within the first environment as emanating from an object within the first augmented reality environment; generating first localized metadata based at least in part on the second metadata by rendering the audio data into spatialized audio for the second user or the first user at the second user location within a second augmented reality environment by localizing the first metadata to the first user or the second user at a second user location; presenting the spatialized audio and at least one virtual object associated with the object to the first user or the second user simultaneously based at least in part on the first metadata and the second metadata; and Including, The method of claim 1, wherein the spatialized audio is perceived by the first or second user in the second augmented reality environment to be reflected and occluded or obstructed by a first real object at a first real object location in the second augmented reality environment.
2. capturing real visual data of at least a portion of a first real world environment within the first augmented reality environment, the real visual data comprising real image data or real video data for the first real world environment portion, the real image data comprising real image content data and real image metadata associated with the real image content data, and the real video data comprising real video content data and real video metadata associated with the real video content data; blending the real visual data with virtual visual data to obtain blended visual data, the virtual visual data comprising virtual image data or virtual video data for a first virtual environment portion within the first augmented reality environment; The method of claim 1 , comprising:
3. identifying the audio data includes selecting, in response to user input received at an augmented reality system, the object from a plurality of objects in the first augmented reality environment at an object selection device of the augmented reality system; The method of claim 1 , wherein the plurality of objects comprises at least one real object and at least one virtual object.
4. identifying the audio data further includes deselecting, in response to a separate user input, a first object selected with the object selection device of the augmented reality system; 4. The method of claim 3, wherein the object selection device includes at least one of a physical controller that controls a cursor displayed to the first user, an audio data capture device, a voice interpretation module that interprets one or more voice commands, a first visual data detection device that tracks gestures, a second visual data detection device that tracks eye position or orientation of the first user, or any combination thereof.
5. selecting the object from a plurality of objects in the first augmented reality environment; tracking a position or orientation of the object using an augmented reality system; preferentially sensing sounds emanating from the object based at least in part on the position or the orientation of the object; Further comprising: The method of claim 1 , wherein the audio data is recorded in a first orientation, at a first object location, or at a first object stance within the first augmented reality environment, with a preference for the object over another object.
6. continuously and preferentially detecting, in a position and orientation tracking device of the augmented reality system, a position and orientation or orientation of the first user or a portion of the first user relative to the object in the first augmented reality environment; updating the first metadata associated with the first user based at least in part on a result of continuously and preferentially detecting the position and the pose or orientation of the first user or a portion of the first user relative to the object to an updated first metadata; updating the first localized metadata based at least in part on the updated first metadata; and The method of claim 5 further comprising:
7. rendering the spatialized audio to a second user location relative to the first virtual object at the first virtual object location in the second augmented reality environment such that the spatialized audio is perceived by the first or second user at the second user location in the second augmented reality environment to emanate from a first virtual object at the first virtual object location in the second augmented reality environment; The method of claim 1 , wherein the audio data is captured as emanating from the object in the first augmented reality environment, and the object is rendered as the first virtual object in the second augmented reality environment.
8. capturing the audio data in the first environment using at least an audio processing module comprising a microphone, a delay element, and a gain element, and operating at least in part based on a polar pattern stored in an augmented reality system; capturing image or video content data using an image capture device of the augmented reality system; capturing image or video metadata associated with the image or video content data using a location or tracking device of the augmented reality system; determining audio metadata associated with said audio data by at least correlating said audio data with said image or video content data or said image or video metadata; The method of claim 1 further comprising:
9. Localizing the first metadata to the first user or the second user at the second user location includes: identifying relative positioning or orientation data relating to a position or orientation of the object relative to the first user location within the first environment or a first pose of the first user; capturing the audio data as perceived by the first user at the first user location by using at least microphone elements, delay elements, gain elements, and polar directivity patterns of an augmented reality system in the first environment; generating an audio signal that preferentially represents the audio data captured from the direction of the object based on the relative positioning or orientation data; identifying a position or orientation characteristic at the second user location within the second augmented reality environment; localizing the first metadata to the first user or the second user at the second user location by at least applying the relative positioning or orientation data to a head of the first user or the second user based at least in part on characteristics of the position or orientation at the second user location; The method of claim 1 , comprising:
10. 1. An augmented reality system, comprising: an audio processing module comprising an audio signal processor; an image processing module comprising a graphics processing unit and coupled to the audio processing module for generating audio-visual content for a first user; a virtual image generation module comprising at least a microprocessor and coupled to the audio processing module and the image processing module for presenting virtual content to the first user; Equipped with The augmented reality system is configured to perform a set of actions, the set of actions comprising: identifying audio data captured within a first augmented reality environment and first metadata associated with the audio data, the audio data is perceived by a first user at a first user location within the first environment as emanating from an object within the first augmented reality environment; generating first localized metadata based at least in part on the second metadata by rendering the audio data into spatialized audio for the second user or the first user at the second user location within a second augmented reality environment by localizing the first metadata to the first user or the second user at a second user location; presenting the spatialized audio and at least one virtual object associated with the object to the first user or the second user simultaneously based at least in part on the first metadata and the second metadata; and Including, The spatialized audio is perceived by the first or second user in the second augmented reality environment as being reflected and occluded or obstructed by a first real object at a first real object location in the second augmented reality environment.
11. further comprising an image or video capture device; The set of operations includes: capturing, at the image or video capture device, real visual data of at least a portion of a first real world environment within the first augmented reality environment, the real visual data comprising real image data or real video data for the first real world environment, the real image data comprising real image content data and real image metadata associated with the real image content data, and the real video data comprising real video content data and real video metadata associated with the real video content data; blending, in the image processing module, the real visual data with virtual visual data to become blended visual data, the virtual visual data comprising virtual image data or virtual video data for a first virtual environment portion within the first augmented reality environment, the virtual visual data being generated by the augmented reality system; The augmented reality system of claim 10 further comprising:
12. further comprising a user input module for receiving input from a user; identifying the audio data includes selecting, in response to user input received at the augmented reality system, the object from a plurality of objects in the first augmented reality environment at an object selection device of the augmented reality system; The augmented reality system of claim 10 , wherein the plurality of objects comprises at least one real object and at least one virtual object.
13. identifying the audio data further includes deselecting, in response to a separate user input, a first object selected with the object selection device of the augmented reality system; 13. The augmented reality system of claim 12, wherein the object selection device includes at least one of a physical controller that controls a cursor displayed to the first user, an audio data capture device, a voice interpretation module that interprets one or more voice commands, a first visual data detection device that tracks gestures, a second visual data detection device that tracks an eye position or orientation of the first user, or any combination thereof.
14. and a user input module and a position or orientation tracking module, the augmented reality system further configured to perform the set of actions, the set of actions comprising: selecting the object from a plurality of objects in the first augmented reality environment; tracking a position or orientation of the object using the augmented reality system; and preferentially sensing sounds emanating from the object based at least in part on the position or the orientation of the object; Including, The augmented reality system of claim 10 , wherein the audio data is recorded in a first direction, at a first object location, or at a first object stance within the first augmented reality environment, with a preference for the object over another object.
15. The set of actions performed by the augmented reality system includes: continuously and preferentially detecting, in a position and orientation tracking device of the augmented reality system, a position and orientation or orientation of the first user or a portion of the first user relative to the object in the first augmented reality environment; updating the first metadata associated with the first user based at least in part on a result of continuously and preferentially detecting the position and the pose or orientation of the first user or a portion of the first user relative to the object to an updated first metadata; updating the first localized metadata based at least in part on the updated first metadata; and The augmented reality system of claim 14 further comprising:
16. The set of actions performed by the augmented reality system includes: rendering the spatialized audio to a second user location relative to the first virtual object at the first virtual object location in the second augmented reality environment such that the spatialized audio is perceived by the first or second user at the second user location in the second augmented reality environment to emanate from a first virtual object at the first virtual object location in the second augmented reality environment; 11. The augmented reality system of claim 10, wherein the audio data is captured as emanating from the object in the first augmented reality environment, and the object is rendered by the augmented reality system as the first virtual object in the second augmented reality environment.
17. The set of actions performed by the augmented reality system includes: capturing the audio data in the first environment using at least an audio processing module comprising a microphone, a delay element, and a gain element, and operating at least in part based on a polar pattern stored in the augmented reality system; capturing image or video content data using an image capture device of the augmented reality system; capturing image or video metadata associated with the image or video content data using a location or tracking device of the augmented reality system; determining audio metadata associated with said audio data by at least correlating said audio data with said image or video content data or said image or video metadata; The augmented reality system of claim 10 further comprising:
18. The set of actions performed by the augmented reality system includes: identifying relative positioning or orientation data relating to a position or orientation of the object relative to the first user location within the first environment or a first pose of the first user; capturing the audio data as perceived by the first user at the first user location by using at least microphone elements, delay elements, gain elements, and polar directivity patterns of the augmented reality system in the first environment; generating an audio signal that preferentially represents the audio data captured from the direction of the object based on the relative positioning or orientation data; identifying a position or orientation characteristic at the second user location within the second augmented reality environment; localizing the first metadata to the first user or the second user at the second user location by at least applying the relative positioning or orientation data to a head of the first user or the second user based at least in part on characteristics of the position or orientation at the second user location; The augmented reality system of claim 10 further comprising:
Citation Information
Patent Citations
Method and unit for display control
JP2004258123A
Audio processing apparatus
JP2015019371A
Presence transmission system and presence reproduction apparatus
JP2016100677A
Systems and methods for augmented and virtual reality
JP2016522463A
Information processing system and information processing method
WO2015170520A1
