Method and apparatus for generating virtual or augmented reality presentations through 3D audio positioning

Abstract

An apparatus for generating a 3D presentation is described, wherein the 3D presentation (which may include a 3D movie, VR (virtual reality) and / or AR (augmented reality) content) is derived from original 3D content including an original audio portion and a visual portion, forming a plurality of scenes, each scene being associated with a respective one or more environments having respective audio characteristics. The original audio portions in at least some of the scenes are modified by the apparatus to conform to or compensate for the acoustic characteristics of the respective scene and / or the acoustic characteristics of the environment of the location in which the presentation is made.

Description

Background of the Invention

[0001] A. Technical field

[0002] The present disclosure relates to the generation and configuration of virtual reality or augmented reality presentations. More specifically, the present invention relates to a method and apparatus for enhancing a virtual reality and / or augmented reality presentation (hereinafter referred to as a "VR / AR presentation") by using 3D spatial information associated with at least some visual characters and other objects of interest that are part of an active viewing scene or outside of the field of view to locate associated audio characters / objects of interest (hereinafter referred to as "audio objects" or "audio objects of interest") in the 3D space of the VR / AR presentation. In addition, the apparatus and method also provide for enhancing audio objects using characteristics of the visual environment for VR presentations and characteristics of the actual environment for AR presentations.

[0003] B. Background Technology

[0004] In the past, 3D movies or other similar episodic audio / visual content were produced for analog cinema distribution or other relatively lower fidelity analog or digital transmission, storage, projection and / or display 3D formats, such as stereoscopic 3D. Advances in 3D encoding formats, rendering techniques, and digital signal processing have resulted in 3D movies or episodic visual content produced in higher quality 3D formats, such as stereoscopic HD 1920×1080p, 3D Blu-ray Disc, etc.

[0005] "Virtual reality" is a term used for various types of content that simulates immersion in a partial or full computer-generated and / or real-action three-dimensional world. Such content can include, for example, various video games and animated film content. Variants of these technologies are sometimes referred to as "augmented reality." In an augmented reality presentation, an actual 3D presentation of a user's current environment is "augmented" by the addition of one or more virtual objects or overlays. Augmented reality content can be as simple as textual "attention" information about objects or people visible around the user, or as complex as converting the entire appearance of the user's surroundings into a virtual environment that corresponds to the user's real environment. Advances in encoding formats, presentation techniques, motion tracking, position tracking, eye tracking, portable accelerometer and gyroscope output / input, and related signal processing have reached the point where virtual and augmented reality presentations can be displayed to users in real time.

[0006] Virtual reality (VR) and augmented reality (AR) have been implemented in various types of immersive video stereoscopic presentation technologies, including, for example, stereo VR head-mounted devices. As described above, 3D head-mounted devices and other 3D presentation devices immerse users in 3D scenes. The lenses in the head-mounted devices enable users to focus on a lightweight split display screen mounted in the head-mounted device and a few inches away from the user's eyes. In some types of head-mounted devices, different sides of the split display display show left and right stereoscopic views of the video content, while the user's peripheral view is blocked or partially unobstructed below the central field of view. In another type of head-mounted device, two separate displays are used to show different images to the user's left and right eyes, respectively. In another type of head-mounted device, the field of view of the display includes the entire field of view of each eye, including peripheral views. In another type of head-mounted device, in order to achieve AR or VR, a controllable small laser, mirror or lens is used to project the image on the user's retina. In either case, the head-mounted device enables the user to experience the displayed VR or AR content in a way that makes the user feel that he is immersed in a real scene. Furthermore, in the case of AR content, the user can experience the augmented content as if it were part of or placed in an augmented real scene. VR or AR content can also be presented to the viewer as a 360° picture, which can be presented on a standard screen with the image moving left or right and / or up and down, either automatically or under the control of the viewer.

[0007] Immersive AR / VR effects can be provided or enhanced by motion sensors in a head-mounted device (or elsewhere) that detect the movement of the user's head, and adjust the video display accordingly. By turning the head to the side, the user can see the VR or AR scene on the side; by turning the head up or down, the user can look up or down in the VR or AR scene. The head-mounted device (or other device) may also include a tracking sensor that detects the position of the user's head and / or body and adjusts the video display accordingly. By tilting or turning, the user can see the VR or AR scene from different perspectives. This responsiveness to head movement, head position, and body position greatly enhances the immersive effect that can be achieved with the head-mounted device. Therefore, the user can be provided with the impression of being placed in a VR scene or "immersed" in a VR scene. As used herein, "immersive" generally includes VR and AR presentations.

[0008] Immersive head mounted devices and other wearable immersive output devices are particularly useful for various types of games that involve user exploration of a modeled environment generated by a rendering engine while the user controls one or more virtual cameras or displays using head movements, position or orientation of the user's body, head, eyes, hands, fingers, feet or other body parts, and / or other input using sensors such as accelerometers, altimeters, GPS receivers, electronic tape measures, laser range finders, laser or sound digital measuring devices, gyroscope sensors, etc. In order to provide an immersive experience, the user needs to perceive a freedom of movement that is somewhat similar to human visual and auditory perception when interacting with reality.

[0009] The content generated for VR / AR presentation can use real-time rendering technology developed for various types of video games to provide this experience. The content can be designed as a three-dimensional computer model with defined boundaries and rules for rendering the content as a video signal. The content can be enhanced by stereoscopic technology to provide stereoscopic video output, sometimes referred to as "3D", and is associated with VR / AR presentation, which responds to the movement of the 3D head-mounted device or the movement of the head, eyes, hands, fingers, feet or other body parts (or body part appendages such as "magic wands" or golf clubs), and / or responds to other inputs such as the above sensors to manage the rendering process to produce the resulting digital VR / AR presentation and user experience. The user experience can be very similar to being placed or immersed in a rendered video game environment.

[0010] In other types of VR / AR presentations, the simulated 3D environment can be used primarily to tell the story, more like a traditional play or movie. In these types of presentations, added visual effects can enhance the depth and richness of the narrative elements or special effects of the story without giving the user full control (or arbitrary control) over the narrative itself. However, a rich mixed reality experience is provided that proceeds in different ways during each encounter (or viewing), as opposed to a book or movie in which a standard linear narrative with a set narrative or sequence of scenes with a single ending is presented. This experience depends on the viewer's direction - for example, which way of viewing - although this can obviously be influenced and guided by narrative clues, as well as some random elements that may be introduced by the software. As a result, the narrative is not linear or predictable at the beginning, but changes due to, for example, choices made by the viewer and other factors. In other words, as a result of the viewer's choices and other factors coordinated with the mixed reality environment, the presented narrative or story can evolve rapidly and hastily to produce tension and relaxation, surprises, linear or nonlinear progressions, turning points or endings. These considerations apply particularly to unscripted presentations, which in some sense have variable, dynamically changing sequences similar to a game or live real-life performance. It is especially important for these types of presentations to ensure that the audio and visual signals are as realistic as possible so that the presentation looks authentic and not fake or artificial.

[0011] In this application, the term "digital VR / AR presentation" is used to refer to video games, movies, episodes, or other audiovisual content that is recorded, produced, rendered, and / or otherwise generated in a digital format, or audiovisual content that is recorded, produced, rendered, or otherwise generated in a digital format to be overlaid on reality. The term also includes content produced in 2D, content produced in 2D and then post-produced from 2D to 3D, content produced natively in 3D, and content rendered from a 3D animation system.

[0012] When preparing a digital VR / AR presentation for distribution in a certain format or distribution channel, there may be relevant 3D visual information determined by the analysis software and / or operator on a frame-by-frame, frame-by-frame group, or scene-by-scene basis and recorded in a corresponding log. The traditional method of representing 3D depth information is via a z-axis depth map, which includes a single two-dimensional image with the same spatial resolution as the 3D image (e.g., 1920x1080 for HD video). Each pixel of the image contains a grayscale value corresponding to the depth of that particular pixel in the scene. For example, for an 8-bit data representation, a grayscale value of 256 (pure white) can represent the maximum positive 3D parallax (entering the screen), while a value of 0 (pure black) can represent the maximum negative parallax (off the screen). The values ​​can then be normalized based on the depth budget of the scene, for example, a value of 256 can represent a pixel 100 feet away from the observer, while a value of 0 can represent a pixel 10 feet away from the observer.

[0013] Another possible data representation of 3D depth information is a 3D depth volume, whereby each pixel in the 3D volume of the scene is represented by a specific value. Unlike the z-axis depth map, the 3D depth volume is not limited to a single grayscale value, but for each pixel, the color value (i.e., RGB value) of the specific pixel as well as the xyz coordinates of the pixel can be represented. Compared with using a 2D z-axis depth map, computer-generated 3D images or other 3D visual effects technologies can make it easier for them to create 3D depth volumes. This 3D representation of depth information can be used in future display systems including holographic projections. Other data representations can be used to represent depth information in a given scene, including but not limited to 2D disparity maps and eigenvectors.

[0014] As part of generating a VR / AR presentation, when preparing to position subtitles or other graphics in 3D space over a background video, a 3D spatial map of the visual content of a frame can be generated, or a 3D spatial map of an object of interest within a frame can be determined.

[0015] Some audio objects of interest may have visual counterparts on the screen that can be tracked spatially. For example, when an actor on the screen moves and speaks in a scene, his position can be tracked auditorily and visually. For example, there are visual object tracking software systems and software development kits (such as the SentiSight 3.0 toolkit of Neurotechnology, Vilnius, Latvia), which can detect and identify visual objects in a scene and identify their specific positions. Such a system can tolerate in-plane rotation, some out-of-plane rotation, and scale changes in a wide range. Such a system can also try to track occluded visual or audio objects (for example, up to 50%). If motion vectors are used to draw the trajectory of objects that are occluded to a greater extent or even completely occluded visually, object tracking can also identify the position of off-screen (off-screen, outside the screen) objects in the case of providing enough previous on-screen information or even in the case of pre-written sequence on-screen information. Other audio objects of interest may not be tracked, for example, actors who speak off-screen, or actors who speak when partially or completely occluded visually. In the latter case, an on-screen actor may gaze directly across and beyond the screen plane boundary at another off-screen actor with whom he is talking. Other audio objects of interest may not correspond to on-screen visual objects at all, depending on positioning or editing intent, e.g., the speech of an off-screen narrator may be essential to the presentation, but there may be no on-screen item corresponding to that speech.

[0016] However, in some cases, during preparation of a digital VR / AR presentation, its audio component, or a portion of the audio component associated with an audio object of interest, may not include clear 3D spatial awareness cues, either because those cues have been stripped out or otherwise lost, or because they were missing from the beginning. This problem is further compounded in real-time applications and environments such as video game rendering and live event broadcasting.

[0017] Just as there is a need to provide cues for 3D spatial awareness for the audio portion to enhance a digital VR / AR presentation, there is a need to include such cues in the audio portion of a digital VR / AR presentation in another format. However, currently prepared digital VR / AR presentations for publication in one format do not include effective conversion of the audio portion to ensure that 3D spatial awareness audio cues are present or preserved in a digital VR / AR presentation published in another format.

[0018] Therefore, there is a need for an effective scheme for utilizing audio 3D spatial awareness cues to optimize digital VR / AR presentation preparation. Furthermore, there is a need for an effective scheme for utilizing audio 3D spatial awareness cues in other formats or distribution formats to optimize additional digital VR / AR presentation conversions. In both cases, information collected in digital 3D video analysis is used as input to generate audio 3D spatial awareness cues to enhance the 3D audio-visual experience.

[0019] Another problem is that if other forms of digital VR / AR presentations are to be viewed in 2D, for example, if no digital VR / AR presentation system is available, i.e., no VR / AR related head mounted device and / or no 3D display, a separate 2D version of the audio portion without 3D spatial awareness cues can currently be distributed for viewing 2D content. Therefore, the data created in the process of encoding audio 3D spatial awareness cues can be saved and included in the digital VR / AR presentation distribution file so that down mixing of 3D to 2D can be managed downstream.

[0020] Audio and video largely create the VR / AR presentation and the resulting experience. {Not concerned here with so-called "4D" theatrical presentations, where scents and / or moving seats and / or water (e.g., "rain") dispensers, etc., are used to enhance an otherwise normal theatrical performance.} So, the VR / AR presentation will be enhanced, and therefore the user experience will be more enveloping and powerful if audio cues related to the location of objects of interest in the VR / AR presentation complement the video, because the associated audio cues emphasize the visual location of the objects of interest in real life, such as a fire truck racing through one's field of view, its siren at low amplitude and relatively low pitch when it is far away, then louder and higher pitch as it arrives, then fading in amplitude and pitch as it travels into the distance, with the apparent source of the sound rising upward as the fire truck exits the shot driving up a hill.

[0021] The format of the audio portion of the digital VR / AR presentation can vary in terms of generation, encoding, transmission, generation and / or presentation. Typical presentation formats of the audio portion can vary from mono to stereo to multi-channel, such as 5.1, 6.1, 7.1 or so-called "object-oriented" or "immersive" audio. Some of these audio formats include audio cues for depth perception, such as amplitude difference, phase difference, arrival time difference, level ratio of reverberation relative to direct sound source, tonal balance shift, shielding and / or surround or multi-channel directionality. These cues can be customized according to the video object spatial position data to enhance the presentation of the digital VR / AR presentation, so that the audio 3D spatial perception on the X, Y and Z axes complements the visual 3D spatial perception. In this way, if the 3D position of the visual object of interest and the associated audio is consistent, the digital VR / AR presentation looks and "feels" more real.

[0022] Therefore, it is desirable to develop methods and apparatus that not only provide a soundtrack that indicates the location of an object of interest in a VR / AR presentation, but also adjust the soundtrack to better match the environment in which the object is located to enhance the appeal and enjoyment of VR and AR content for a more immersive VR / AR presentation. Summary of the invention

[0023] An apparatus for generating a 3D presentation is described, wherein the 3D presentation (which may include a 3D movie, VR (virtual reality) and / or AR (augmented reality) content) is derived from original 3D content including an original audio portion and a visual portion, forming a plurality of scenes, each scene being associated with a corresponding environment having respective audio characteristics. The original audio portions in at least some of the scenes are modified by the apparatus to conform to or compensate for the acoustic characteristics of the corresponding scene and / or the environment of the location where the presentation occurs.

[0024] More specifically, an apparatus for generating a 3D presentation based on content having an original audio portion and a visual portion, the content comprising a plurality of scenes, each scene being associated with a corresponding environment having audio characteristics, the apparatus comprising: an audio processor receiving the audio portion and an environmental parameter indicating the corresponding environment, the audio processor being configured to modify the original audio portion into a modified audio portion based on the environmental parameter; an authoring tool receiving the modified audio portion and the original video portion and generating corresponding encoding coefficients; and an encoder receiving the original video portion and the modified audio portion and generating encoded content based on the encoding coefficients.

[0025] When the 3D presentation is presented in a cinema having cinema acoustic parameters, the apparatus comprises an acoustic memory storing the cinema acoustic parameters, and the audio processor is configured to receive the cinema acoustic parameters to generate the modified audio portion. In addition, an acoustic environment sensor adapted to sense the cinema acoustic parameters may be provided, and a signal sensed by the acoustic environment sensor is stored in the acoustic memory.

[0026] In one embodiment, the encoder is configured to generate VR (virtual reality) encoded content for presentation on a VR device, wherein the video portion defines at least a first virtual scene, the first virtual scene having a virtual environment including virtual acoustic environment characteristics, and wherein the audio processor is configured to receive the virtual acoustic environment characteristics and generate coefficients of an audio track defined for the virtual scene configuration.

[0027] In this embodiment, the apparatus may include an environment analyzer configured to analyze a VR environment defined in the first virtual scene, the environment analyzer generating the virtual environment acoustics for storage in the acoustic memory.

[0028] In one embodiment, the encoder is configured to generate AR (augmented reality) encoded content for presentation on an AR device based on the received content and content obtained from the current environment, wherein the video portion defines at least a first AR scene including virtual and real elements, and the audio processor is configured to receive virtual and real audio portions and acoustic characteristics of the first AR scene, and the audio processor generates coefficients describing an audio signal configured to match the acoustic characteristics of the first AR scene.

[0029] In one embodiment, an audio processor generates coefficients describing an audio signal including an ambient signal associated with the first AR scene.

[0030] In one embodiment, the device includes a real environment detector that detects a real environment associated with the viewer's current location and an acoustic environment detector configured to determine acoustic parameters of the real environment. For example, the real environment detector includes a GPS or a camera adapted to generate an image associated with the current location and an analyzer that analyzes the image to determine the acoustic parameters.

[0031] In one embodiment, the real environment detector comprises a sound generator generating a test signal and a microphone arranged to detect echoes originating from the sound detector, the acoustic environment detector being adapted to analyse the echoes.

[0032] In another aspect of the invention, a method for generating an encoded 3D presentation is described. The method comprises receiving 3D content having a plurality of 3D scenes and comprising audio and visual tracks; determining acoustic characteristics of one of a local environment in which the 3D presentation is presented and a virtual location of the 3D scene; modifying the audio track to generate a modified track based on the acoustic characteristics; encoding the 3D content using encoding coefficients that are at least partially dependent on the audio spatial parameters to generate an encoded 3D presentation, the encoded 3D presentation comprising a visual portion and an audio portion, the audio portion being derived from the modified audio track.

[0033] The 3D presentation may be presented in a cinema, and the step of determining comprises determining acoustic properties of the cinema.

[0034] The 3D presentation may be a VR presentation for a viewer using a personal VR device, and the determining step includes determining virtual acoustic characteristics of a virtual scene in the 3D presentation. The modifying the soundtrack step may include adding additional sounds to the soundtrack, the additional sounds being associated with an acoustic environment associated with the virtual scene.

[0035] The 3D presentation may be an AR presentation of a viewer using a personal AR device, the AR presentation comprising an AR scene having virtual and real scene elements. In this case, the determining step comprises determining virtual acoustic properties of the virtual scene elements. Alternatively, the determining step comprises determining real acoustic properties of the real scene elements.

[0036] The step of modifying the soundtrack may include adding additional sounds to the soundtrack, the additional sounds being associated with an acoustic environment associated with the virtual scene element.

[0037] The step of modifying the soundtrack may comprise adding additional sounds to the soundtrack, the additional sounds being associated with an acoustic environment associated with the real scene element. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1AA block diagram of an apparatus constructed according to the present invention is shown;

[0039] Figure 1B shows a block diagram of an audio processor;

[0040] Figure 2-4 Several plan views (on the XZ axis) of several 3D frames with several audio and visual objects of interest are shown;

[0041] Figure 5 A method of operating the apparatus of FIG. 1 is shown; and

[0042] Fig. 6A , 6B 6C show elements of the audio signal processor used in the block diagram of FIG. 1 for 3D presentation, V / R presentation, and A / R presentation, respectively. DETAILED DESCRIPTION

[0043] The present invention provides a solution to the above-mentioned problem. The solution proposed herein is an improvement of the device disclosed in the commonly assigned U.S. Patent No. 8,755,432 entitled "METHODAND APPARATUS FOR GENERATING 3D AUDIO POSITIONING USING DYNAMICALLY OPTIMIZED AUDIO 3D SPACE PERCEPTION CUES", the entire contents of which are incorporated herein, and it should be understood that the present invention can also be implemented using other devices.

[0044] FIG1 shows components of an apparatus for implementing the present invention. Content, such as uncompressed 3D content 60, is first presented to the apparatus for processing. In this application, the term "3D" content is used to generally refer to content that can be presented on a suitable 3D device, and may include 3D presentation, VR presentation, or AR presentation.

[0045] A 3D presentation is a presentation, such as a movie, that includes multiple 3D images presented in a predetermined order, without interaction or input from the viewer, and is accompanied by a suitable 3D soundtrack. For example, a digital 3D movie may be a sequence of 3D frames having the same native resolution as the source material (e.g., 1920×1080p). In other words, a 3D movie may be a feature film shot with live actors, or may be an animated film with virtual characters, or a hybrid, or live action supplemented by CGI effects. Importantly, the 3D presentation is displayed to a single viewer, as presently contemplated, or to multiple viewers simultaneously, for example, in a movie theater with the final audio portion presented to the viewers through the theater's audio speaker system.

[0046] VR presentation refers to a presentation (such as an interactive game) in which various scenes are played in an order determined by: viewer actions, including eye tracking or focus, gestures or other body / limb movements; positioning of an appendage (such as a "magic wand" or golden club), object or facial recognition software, or motion tracking or depth sensing input provided by appropriate VR equipment, such as a head-mounted display (HMD) with audio output for stereo or multi-channel audio and a screen that presents 3D images to the user. The audio portion may be provided through headphones or earbuds. The 3D images are presented in an order determined by the viewer, and a suitable 3D soundtrack is generated to match the visual order. The 3D images may be real images of actual objects, scenes, etc., or may be virtual images generated using well-known microprocessor-based techniques. VR presentation may be an interactive game in which various scenes are played in an order determined by: viewer actions, including eye tracking or focus, gestures or other body / limb movements; positioning of an appendage (such as a "magic wand" or golden club), object or facial recognition software, or motion tracking or depth sensing input provided by a device in the head-mounted display (HMD). The final video portion of the VR presentation is shown on the screen of the VR device (not shown), and the audio portion is presented through standard audio equipment, such as headphones or earbuds (not shown).

[0047] An AR presentation is similar to a VR presentation and includes an image of the real-time object that the viewer is viewing and which image is combined with other 3D images that are superimposed or otherwise combined with the real-time image. The other 3D images can be, for example, real 3D images that were previously taken or computer-generated images. Real objects such as walls or other obstacles are detected using various sensors or optical recognition elements. Proximity or changes to surrounding objects are determined by accelerometers, altimeters, GPS receivers, electronic tape measures, laser rangefinders, laser or sound digital measuring devices, or gyroscope sensors. Likewise, an AR presentation includes audio and video portions, where the audio portion is played through a headset or similar device and the video portion is presented on a screen.

[0048] Typically, in each 3D frame, a plane and / or a visual object of interest corresponds to an audio object of interest. For example, in one instance, an actor on the screen can speak a main dialogue. In this case, the audio object of interest-the main dialogue and the actor have a 3D spatial position. In another case, the audio object of interest may not correspond to the visual object of interest. For example, an on-screen or off-screen narrator or commentator can provide an edited dialogue of a car race. Various vehicles in the race can quickly appear from the background, roar closer and closer, and then zoom in past the off-screen viewer. In this case, the visual object of interest is a car, and there are two groups of audio objects of interest: the sound of the edited dialogue and the amplified car. In this case, in order to make the scene look and feel dramatic, the sound of the car race and the edited dialogue should be provided as an audio object of interest. During the scene, the car sound (e.g., audio object) and the visual image (e.g., visual object) get closer and closer to the viewer but do not cover up the edited dialogue. In another case, for example, in a video game, visual objects can pop in and out of the scene in a pseudo-random manner, and various sound effects can be associated with some of these objects, or audio objects at random intervals can be generated. Importantly, as discussed in more detail below, the soundtracks generated herein not only include standard audio elements such as speech, sound effects, etc., but also modify these elements and add new elements that are consistent with the characteristics of the virtual environment of the scene being played and, in some cases, the physical environment of the viewer.

[0049] As described above, 3D content may include a Z-axis depth map indicating the relative distances of various visual objects (or planes of interest, such as one plane for A / R real-world objects and one position for A / R overlays) of the corresponding 3D frame. Traditionally, a 3D Z-axis depth map consists of a two-dimensional image with the overall outlines and surface features of all objects of interest. Objects and object surfaces at various positions along the Z axis are represented on the two-dimensional image by their respective outlines and surface shadows. Each outline includes one or more surfaces that are shaded. More specifically, each outline and each surface of the outline (if more than one) is shaded in a grayscale corresponding to the relative position of the corresponding object and its surface. Therefore, the 3D Z-axis depth map provides information about the relative positions of various objects of interest and their surfaces relative to each other and relative to specific reference points such as the screen and / or the viewer. Of course, as described above, other means of indicating the position of audio or visual objects in 3D space can be used, and the present invention is not limited to any of these means.

[0050] Reference now Figure 1AIn the system according to the invention, the encoder 10 receives a 3D movie, for example in the form of an uncompressed video master 60. The audio / video encoder 10 generates from the master a first 3D encoded content that can be stored, recorded or distributed to a client via various distribution channels. The first 3D encoded content may or may not have information that provides audio 3D spatial perception cues relative to 3D spatial visual objects or any other audio objects of interest. It does not have audio 3D information that defines the acoustic environment in which the master is to be played and / or the virtual environment in which the scene takes place.

[0051] like Figure 1A As shown, the apparatus for implementing the present invention includes a 3D video encoder 10, an audio processor 21, a video processor 23 and an authoring tool 22. 3D content 60 is presented to a 3D video encoder for processing 3D content. The output of the encoder 10 is referred to as first 3D encoded content.

[0052] The video processor 23 receives a 3D video tracking map, such as the above-mentioned 3D Z-axis map that tracks the position of the visual object of interest for each frame or group of frames of the encoded content. The 3D tracking map can be derived from the log 20, or obtained by analyzing the first 3D encoded content using the analyzer 30. In addition, the video processor 23 can also receive manual controls for controlling the position of some visual objects. In addition, new visual content can also be provided to the video processor in the A / R instance, for example, according to the static or changing proximity of surrounding objects determined by object recognition or facial recognition software or motion tracking or depth sensing devices included in the HMD (e.g., computer vision, accelerometers, altimeters, GPS receivers, electronic tape measures, laser rangefinders, laser or sound digital measurement devices, or gyroscopic sensors). New visual content may be provided to the video processor in the form of subtitles and / or other additional graphical elements (as more fully described in commonly assigned co-pending application Ser. No. 13 / 205,720, filed Aug. 9, 2011, entitled “METHOD AND APPARATUS FOR GENERATING ENCODED CONTENT USING DYNAMICALLY OPTIMIZED CONVERSTION FOR 3D MOVIES,” which is incorporated herein by reference). The visual processor generates visual parameters that are provided to the authoring tool 22. The authoring tool may operate in real time, e.g., for video gaming, A / R, or V / R rendering purposes, or perform its functions prior to rendering, e.g., for 3D presentation.

[0053] The audio processor 21 receives the original audio signal (e.g., an audio track from an uncompressed video master) and / or other sources. For example, additional audio content that was not in the original uncompressed video master may be added. The audio processor 21 receives other information, such as the 3D positions of various visual objects, manual and / or automatic selection signals that associate certain audio tracks with visual objects, or define audio objects (if the audio track is not associated with a visual object, or is associated with an off-screen visual object). Importantly, the audio processor 21 also receives other environmental parameters indicative of the corresponding environment, as described in more detail below. The audio processor 21 then generates audio parameters indicating the location of the audio content, such as an audio object of interest in the relevant frame. Figure 1B As shown in more detail in , the audio processor 21 includes a comparator 100 that receives the 3D video and audio tracking diagram. Audio sources (i.e., audio objects of interest) can be located in 2D and 3D space, i.e., along the X, Y, and Z axes. For various editing or other reasons, the audio source positions on the X, Y, and Z axes may be important. Various standards or methods can be used to generate and / or identify the positions of audio objects, as discussed in more detail below. Once these positions are selected, the authoring tool 22 analyzes the audio object parameters, and for each frame or a group of frames, the positions of the audio objects are assigned and then appropriate authoring or reauthoring control signals are generated for processing the audio sources in the 3D frames with prompts to locate them at specified positions.

[0054] For AR, an evolving visual tracking map may be generated based on motion tracking or depth sensing devices included in the HMD, such as computer vision, object or facial recognition software, accelerometers, altimeters, GPS receivers, electronic tape measures, laser rangefinders, laser or sound digital measurement devices, or gyroscope sensors. The tracking map may be used to generate AR overlays or other blends of AR objects or graphics with real-world objects, including audio positioning.

[0055] In some cases, the audio and visual positions of the object of interest correspond, and 3D spatial information of the audio and visual objects is recorded so that the audio object can be mapped onto the visual object of interest. In one embodiment of the invention, only the depth position (e.g., the position along the Z axis) is provided. In another embodiment, the position of the audio object in the 3D volume (e.g., along all three axes X, Y, and Z) is provided separately from the position of any visual object. Reference Figure 1A, a comparator 100 receives the audio and visual tracking graphs and determines whether they are convergent. If there is a single visual object and a single audio, this determination can be simple. However, if there are multiple visual and multiple audio objects, each audio object can be assigned to each visual object using manual control or other means. Once this identification is completed, automatic tracking software is provided to track the audio and visual objects consistently by component 103.

[0056] In other cases, the audio and visual objects of interest diverge, or there is no available information indicating whether the audio object is mapped to a specific visual object of interest. In this case, a manual operator or an automatic analyzer running automatic object audio tracking software (if necessary) represented by component 102 can select and track the audio object of interest "on the screen" (as well as any visual objects A, B, C or D). This tracking information can be represented as, for example, a Z-axis depth map (if only depth perception is required), or it can be represented in full 3D space along the X, Y and Z axes. If there is no audio object of interest on the screen or the audio object is occluded (e.g., an actor speaking from behind a closed door), or an off-screen virtual object has a known or mapped position, the tracking map is extended to also provide information for occluded or off-screen audio objects, including audio objects that are set in the same vertical plane as the viewer or even behind the viewer.

[0057] Figure 2-4 Some simple examples are provided in plan view, showing various audio and visual objects of interest in the scene. Figure 2 A plan view of a stage with four performers A, B, C, D as seen by a viewer V is shown. A, B, C and D can be singers and / or musicians, and they can remain stationary or move around the stage. Initially, the video portion of the received 3D movie shows the position of each performer. The audio portion consists of a mixed soundtrack or several tracks with audio signals from each performer, in which case there is essentially no audio depth perception or any other audio cue to spatially bind the tracks to the performers. The audio portion also lacks any acoustic properties of the desired environment, for example, the environment in which the corresponding scene takes place.

[0058] If there are no separate tracks, the original tracks are analyzed and manipulated by an audio processor (comparator 100 or component 102) to generate audio parameters defining a virtual track representing the sound from each performer. The tracks are then associated with each performer. This can be done manually by an operator or automatically, for example, by processing the video portion to identify the four performers using a facial recognition algorithm, by identifying their instruments, or other means.

[0059] Once the position of the performer is identified or associated with the corresponding soundtrack, audio parameters representing the audio object of interest are generated by the audio processor 21. In a preferred embodiment, these parameters represent the depth perception cues of some or all soundtracks. If necessary, the positions of some performers are emphasized in some scenes, and the positions of other performers can be emphasized at other times, for example, based on the relative loudness of the track or other editing standards. Importantly, the audio processor also receives environmental parameters defining the acoustic characteristics of the corresponding environment, and modulates each soundtrack accordingly. Therefore, if the scene occurs in a forest, the soundtrack is modified so that the viewer who watches and listens to the program not only hears the soundtrack in the way of hearing the sound in the forest, but also hears additional sounds, such as birdsong, bee buzzing, etc. These sound effects (the term "sound effect" is used in this article to refer to the modification of sound and the addition of new sound) therefore provide a true 3D experience to the viewer.

[0060] As described above, if the performers move around, their positions are tracked and the audio object positions corresponding to the audio objects of interest are recorded so that the audio tracks corresponding to these objects can be processed with appropriate audio position cues to locate them in 3D space. The quality of the audio tracks is also modified based on the position of the audio objects.

[0061] Figure 3 A plan view of different scenes is shown where two actors A and B are talking and move along paths E and F so that actor A ends up at position A1 and actor B ends up at position B1. At one point in time, one or both of actors A and B are positioned "off-stage" behind a viewer V. All positions are tracked and / or plotted from the video portion object tracking map and the corresponding audio object tracks are processed with 3D spatial position cues by component 103 to correspond to these positions. The resulting audio track also includes modifications to make it more realistic based on the acoustic environment of the location where the scene is taking place.

[0062] Figure 4 Shown with Figure 3 Here is a floor plan of a scene somewhat similar to the one in . In this scene, actor B is initially behind wall W (or other obstacle), but can still be heard while he speaks. Actor A is visible, and is talking to B. During the conversation, actor B moves near wall W and forward to position B1. At the same time, or later, actor A walks through doorway D so that he is no longer visible, but can move along an arbitrary path P, and still continues to talk. As the two actors move around and talk, their positions are constantly tracked and mapped, so appropriate audio 3D spatial cues can be generated even when the actors are out of sight, such as off-stage. (Obviously, Figure 4 Not drawn to scale because viewer V is typically further away from the stage. )

[0063] Figure 5 A flow chart of an exemplary method used by the apparatus of FIG. 1 is shown. As initially discussed, a 3D movie of a known format is received and its audio and video portions are encoded in step 110. In step 120, audio and video objects of interest (e.g., in a scene) are detected. This step may be performed by an operator or automatically. The spatial positions of these objects are then compared. If they match (step 130), they are converted to a new format using an appropriate encoding scheme (step 140) and then distributed over an appropriate channel (step 150). If there is no match in the spatial position, or if there is not enough information to detect the spatial position of the audio object, then in step 160, the spatial position is identified by the audio processor 21.

[0064] In step 170, it is checked whether the audio object of interest is associated with a visible object. If so, in step 180, the position of the corresponding visual object (actor A, B) is continuously tracked (step 180) and the corresponding audio prompt is generated (step 190). If the audio object does not correspond to the visual object (e.g., actor B in its original position or actor A in position A1), a different calculation is performed using specific rules applicable to occluded, off-screen or other hidden objects (step 200). For example, the movement of the hidden object (e.g., an actor moving behind a wall) can be interpolated and used to generate the corresponding audio object. At this time, additional sound effects are also obtained based on the previously received environmental parameters. In step 210, the audio and visual parameters are provided to the authoring tool 22. For example, a positional prompt (one-dimensional, two-dimensional or three-dimensional) can be added to the audio part. Alternatively, a separate field is provided, in which the audio prompt is presented as an audio 3D position map, table or other convenient form. The new sound effect is also provided to the authoring tool so that the coefficients generated in step 220 also define the sound track that has been modified according to the corresponding acoustic environment.

[0065] These steps are implemented by the apparatus of Fig. 1 by encoding the audio portion using appropriate 3D spatial audio perception cues. During the initial encoding performed by the audio / video encoder 10, various parameters (such as a Z-axis depth map or other markers for visual objects of interest) and the audio track are stored in a log 20. The encoder 10 generates a first 3D encoded content for distribution, storage, recording, etc.

[0066] Then, possibly at a future date, content in a different, possibly enhanced 3D format is desired, which includes 3D audio perception cues. For this purpose, information from the log 20 is provided to the audio processor 21. Alternatively, the first 3D encoded content (or 3D uncompressed digital master - if still available) is analyzed by the analyzer 30, and the resulting 3D tracking map (e.g., Z-axis depth map) for the visual objects of interest is obtained, as well as tracking information indicating the position of the audio objects of interest (if available).

[0067] The audio processor 21 then uses the information obtained from the log 20 and / or the analyzer 30, as well as additional control signals and environmental parameters generated manually or automatically, and generates a set of audio parameters defining the positions of various audio objects of interest. These audio parameters are provided to the authoring device 22. The authoring device also receives corresponding video information about the positions of visual objects or each frame or group of frames from the video processor 23.

[0068] The audio parameters are used by the authoring tool 22 to generate parameters (including 3D positional awareness cues) for the way audio objects of interest are perceived, which can be processed according to one or more audio object tracking map parameters to add cues such as amplitude differences, phase differences, arrival time differences, level ratios of reverberation relative to direct sound sources, tonal balance shifts, masking and / or surround or multi-channel directionality. These parameters are customized for the audio objects of interest to correspond with the visual objects of interest where appropriate for editing, thereby providing an enhanced 3D audio-visual experience. Operator controls allow manual override of the otherwise automatic process so that editorial emphasis or de-emphasis of audio depth perception cues can be appropriately managed, i.e., when visual and audio objects of interest do not correspond, or when there are no on-screen audio objects of interest, for example, the voice of an omniscient, disembodied narrator is delivering important dialogue, as described above.

[0069] In the examples given above, various scenes are described as real scenes with real actors speaking real dialogues. Of course, these scenes need not be based on actual actions and characters, but can be a mixture of real and virtual actions and characters, or purely virtual actions. The term "virtual action or character" is used to describe cartoon characters or other virtual objects or actions (visual and audio) that are animated or generated by a video game or other similar rendering device.

[0070] Importantly, the audio processor may include an ambient processor for modifying the original audio track to take into account local ambient elements. Fig. 6A, an audio processor 21A including an ambient processor 200 for processing 3D presentations in a movie theater is shown. The processor 200 receives local environmental characteristics from a memory that stores information about the acoustics of the corresponding movie theater. The processor 200 analyzes the original soundtrack and modifies them as needed to compensate for the acoustic effects of the movie theater. For example, the sound of a wall of water, a whispered conversation, or a passing fire truck sounds different for viewers sitting in different theaters or at different locations in the theater, in either case, with reference to different acoustic environmental characteristics. The ambient processor 200 compensates for this effect so that the viewer hears a more realistic soundtrack - that is, a soundtrack that is closer to the original intention of the video director.

[0071] Figure 6B Another audio processor 21B designed for VR presentation is shown. During this presentation, the user is controlling the sequence and environment of the action from one scene to another. So, for example, in one scene, the action may take place in an open field, while the next scene may take place in a cave. To achieve a realistic effect, the audio processor 21B includes an environment processor 210, a virtual environment detector 212, and an acoustic memory 214. The detector 212 detects the appropriate visual environment for a particular scene based on the video information it receives. The video information may be provided by the video processor 23 in FIG. 1 . Once the appropriate environment is detected, the appropriate acoustic characteristics of the environment are obtained from the acoustic memory 214. The processor 210 then uses this information to modify the original soundtrack. The resulting parameters include or represent the parameters discussed above and in Figure 5 The audio depth perception cues discussed in steps 190 and 200 of .

[0072] Figure 6C Details of a sound processor 21C for an AR presentation are shown. For this presentation, both audio and video information are combined with real audio and video signals from the viewer's actual environment to produce an augmented reality. For this purpose, the original soundtrack is fed to a corresponding synthesizer 222. The synthesizer 222 also receives real live audio signals from the viewer's actual environment via a microphone 220. The combined soundtrack is provided to an environment processor 224.

[0073] like Figure 6B As in FIG. 2 , the virtual video information is provided to a virtual environment detector 226. The environment detector 226 detects the virtual environment from the video signal and provides the information to an acoustic memory 228.

[0074] In addition, a real environment detector 230 is used to detect the real environment of the viewer. For this purpose, the detector 230 is connected to a camera 232. The detector 230 analyzes the image from the camera to determine, for example, whether the viewer is in a park or his bedroom. This information is provided to the memory 228 so that the appropriate acoustic characteristics are presented to the processor 224.

[0075] Alternatively, in place of camera 232, a geo-locator such as GPS 236 is provided to determine the viewer's environment based on its location.

[0076] In yet another alternative, a sound generator 238 is used to generate the sound pattern. The resulting echoes are sensed by the microphone 220 or by another microphone to determine the current environment of the viewer.

[0077] It should be understood that Fig. 6A , 6B The three environmental processors shown in 6C are preferably combined with the elements discussed above and shown in Figures 1, 1A, etc.

[0078] The authoring tool 22 generates a set of encoder coefficients corresponding to the audio and visual objects for each frame or group of frames.

[0079] In one embodiment of the present invention, the second audio / video encoder 70 may, but need not be, similar or even identical to the encoder 10 shown in FIG. 1 to receive digital content in any format including any of the above formats based on the 3D uncompressed video master 60. The encoder 70 analyzes the received digital 3D movie and automatically determines its format, or the device may receive user input or some other signal indicating the format of the received digital 3D content. The encoder 70 also receives information specifying the desired output format, or is preprogrammed to convert the digital 3D content into a corresponding second encoded 3D content of a preselected different format (e.g., Blu-ray disc, satellite broadcast, Nintendo 3DS, etc.). For this purpose, the encoder 70 uses a plurality of encoding coefficients stored in the memory 70B. Some encoding coefficients may be common to several different output formats, while other encoding coefficients are unique to each format. Importantly, according to the present invention, the contents of the memory 70B are changed by changing one or more predetermined coefficients with parameters from the authoring tool 22. In some cases, the parameters from the tool 22 may be incompatible with the encoder 70. In this case, the parameters from the tool 22 are first provided to a mapping processing element 24, which maps the parameters into a format compatible with the encoder 70. The mapped parameters replace some or all of the coefficients in the memory 70B.

[0080] It should be emphasized that in the above description, cues for audio depth perception are primarily discussed as indicating the 3D position of an audio object of interest in at least one, or two, or all three dimensions (ie, X, Y, and Z).

[0081] In one embodiment, the authoring tool 70 not only modifies the audio signal with depth perception cues; it also creates metadata containing the relevant inverse transforms so that a downstream signal processor (not shown) can derive the original (or close to the original if the process is lossy) unoptimized digital 3D movie audio stream for 2D viewing. The metadata is stored with the second encoded 3D content.

[0082] In one embodiment of the invention, the format of the second encoded 3D content is very different from the format of the first encoded 3D content, for example in terms of the intended environment. For example, the first encoded 3D content may be formatted for a digital 3D movie distribution channel intended for a 70-foot cinema exhibition, while the second encoded 3D content is targeted for a Nintendo 3DS 3.5-inch handheld screen. Authoring tool settings are selected accordingly to conform to different digital 3D movie distribution channels or formats for other system reasons. (Details of such transformations are described in commonly assigned co-pending provisional application serial number 61 / 533,777, filed on September 12, 2011, entitled “METHODS FOR CONTROLLINGSCENE, CAMERA AND VIEWING PARAMETERS FOR ALTERING PERCEPTION OF 3D IMAGERY”; and in application serial number 61 / 491,157, filed on May 28, 2011, entitled “METHODS FOR CONTROLLING SCENE, CAMERA AND VIEWING PARAMETERS FOR ALTERING PERCEPTION OF 3D IMAGERY”).

[0083] Importantly, the position of the audio object of interest must be adjusted accordingly.

[0084] Those skilled in the art will appreciate that, although in the embodiment shown in FIG. 1 , the second audio / video encoder 70 receives the original uncompressed video master or the first 3D encoded content as input, the present invention is not limited thereto, and in fact, the encoder 70 receives any other content as input and cooperates with the appropriate components of FIG. 1 (e.g., the analyzer 30, the audio and visual processors 21 and 23, and the authoring tool 22) to generate a corresponding 3D encoded output, which includes information indicating the position information of one or more sounds specific to one or more frames (in the form of audio objects or as hints). As described above, this information can be coupled or bound to the position of any visual object of interest, or can be completely independent of the position of any visual object of interest.

[0085] The present system and method provides a means to efficiently provide enhanced encoded 3D content through dynamic processing by using information from the log 20 or analyzer 30 to identify or generate audio objects of interest.

[0086] Many modifications may be made to the present invention without departing from the scope of the invention as defined in the appended claims.

Claims

1. An apparatus for generating a virtual reality (VR) presentation or an augmented reality (AR) presentation to a viewer based on input content having an original audio portion and a video portion, the content including a scene associated with a corresponding location environment having acoustic characteristics, the apparatus comprising: a virtual environment detector configured to output a first signal indicative of a detected specific location environment of the scene, the detection being based on the video portion of the content, wherein the scene is a virtual scene associated with a virtual location environment having virtual acoustic characteristics; a real environment detector configured to output a second signal indicative of a detected real environment associated with a current position of the viewer; an acoustic memory having acoustic characteristics associated with at least one environment stored therein, the acoustic memory being configured to output selected acoustic characteristics of the location-specific environment of the virtual scene identified by the first signal and to output local environment characteristics of the real environment identified by the second signal; an audio processor comprising an environment processor, the environment processor receiving the selected acoustic characteristics of the virtual scene and the local environment characteristics of the current real environment of the viewer, analyzing the original audio portion, the audio processor being configured to modify the original audio portion into a modified audio portion based on the selected acoustic characteristics of the specific location environment of the virtual scene and the local environment characteristics of the real environment to compensate for the acoustic characteristics of the virtual scene and the current real environment of the viewer; as well as An encoder is configured to receive the original video portion and the modified audio portion and generate the presentation as an encoded output.

2. An apparatus according to claim 1, wherein the encoder is configured to generate AR (augmented reality) encoded content based on a combination of (i) received content having the virtual scene and (ii) real content obtained from the viewer's current real environment, wherein the encoded content defines an AR scene including virtual elements and real elements. The apparatus of claim 1 , wherein the real environment detector comprises a GPS.

4. The apparatus of claim 1 , further comprising a camera adapted to generate an image associated with the current location of the viewer, the real environment detector being configured to analyze the image to detect a real environment depicted in the image from the camera.

5. The apparatus of claim 1, wherein the real environment detector comprises a sound generator generating a test signal and a microphone arranged to detect an echo originating from the sound detector, the real environment detector being adapted to analyze the echo.

6. A method for generating an encoded 3D presentation to a viewer, comprising: Receiving 3D content having a 3D scene, the 3D scene having audio content and video content, the 3D scene being associated with a corresponding virtual location having an acoustic environment; Detecting the virtual position of the 3D scene based on the video content; determining acoustic characteristics of the detected virtual location in the 3D scene; receiving real content including a video of a real environment of the viewer at a viewer location having real elements; Receiving local environment characteristics of the viewer's current real environment; analyzing the audio content, modifying the audio content to generate modified audio content based on the detected acoustic characteristics of the virtual location and local environment characteristics of the current real environment to compensate for the acoustic characteristics of the 3D scene and the current real environment of the viewer; as well as The 3D content is encoded to generate an encoded 3D presentation, the encoded 3D presentation comprising a visual portion and an audio portion, the audio portion being derived from the modified audio content.

7. The method of claim 6, wherein the step of modifying the audio content comprises adding additional sounds to the audio content, the additional sounds being associated with the acoustic environment associated with the 3D scene.

8. The method according to claim 6, further comprising the steps of: determining real acoustic properties of the real element; as well as Additional sounds are added to the audio element, the additional sounds being associated with an acoustic environment associated with the real element.

Images