Apparatus, method and non-transitory medium for enhanced 3D audio authoring and rendering

Abstract

Improved tools for editing and presenting audio playback data are proposed. Some of these editing tools enable the widespread use of audio playback data in various playback environments. Audio playback data can be edited by generating metadata for audio objects. Metadata can be generated with reference to speaker locations. During the presentation process, audio playback data can be reproduced according to the playback speaker layout of a specific playback environment.

Description

Technical Field

[0001] This disclosure relates to the editing and presentation of audio playback data. In particular, it relates to the editing and presentation of audio playback data for playback environments such as theater sound playback systems. Prior Technology

[0002] Since the introduction of sound into film in 1927, steadily developing technologies have been employed to extract the artistic meaning from film recordings and reproduce them in theatrical settings. In the 1930s, synchronized sound on the magnetic tape gave way to variable areas of sound on film, a situation further improved in the 1940s with the introduction of multiple simultaneous recording and controlled playback (using tone control to move sound), and further enhanced by theatrical auditory considerations and improved speaker design. In the 1950s and 1960s, film magnetic tapes allowed for multi-channel recording and playback in cinemas, with high-quality cinemas employing surround sound and up to five screen channels.

[0003] In the 1970s, Dolby introduced noise reduction in both post-production and film by combining three screen channels and a mono surround channel using cost-effective encoding and distribution methods. The quality of theatrical sound was further improved in the 1980s through Dolby Spectral Recording (SR) noise reduction and certification programs such as THX. During the 1990s, Dolby brought digital sound to theaters in a 5.1 channel configuration, offering separate left, center, and right screen channels, left and right surround arrays, and a subwoofer channel for low-frequency effects. The Dolby Surround 7.1, introduced in 2010, increased the number of surround channels by dividing the existing left and right surround channels into four "regions."

[0004] With the increase in the number of channels and the shift in speaker layout from planar two-dimensional (2D) arrays to three-dimensional (3D) arrays including height, positioning and rendering sound effects has become increasingly difficult. Improved audio editing and rendering methods will be in high demand. Summary of the Invention

[0005] Some aspects of the subject matter disclosed herein can be implemented using tools for editing and presenting audio playback data. Such editing tools enable the audio playback data to be widely used in various playback environments. Based on some of the above implementations, audio playback data can be edited by generating metadata for audio objects. Metadata can be generated with reference to speaker locations. During the presentation process, the audio playback data can be reproduced according to the playback speaker layout of a specific playback environment.

[0006] Some embodiments described herein propose an apparatus including an interface system and a logic system. The logic system is configurable to receive, via the interface system, audio playback data including one or more audio objects and associated metadata and playback environment data. The playback environment data may include indications of a plurality of playback speakers in the playback environment and indications of the position of each playback speaker within the playback environment. The logic system may present the audio objects as one or more speaker feedback signals based on at least a portion of the associated metadata and playback environment data, wherein each speaker feedback signal corresponds to at least one playback speaker within the playback environment. The logic system may be configured to calculate speaker gain corresponding to a virtual speaker position.

[0007] The regenerated environment can be, for example, a theater sound system environment. The regenerated environment may have a Dolby 5.1 surround sound configuration, a Dolby 7.1 surround sound configuration, or a Hamasaki 22.2 surround sound configuration. Regenerated environment data may include regenerated speaker layout data indicating the locations of regenerated speaker areas. The regenerated environment data may include regenerated speaker area layout data, indicating multiple regenerated speaker areas and multiple regenerated speaker locations corresponding to those areas.

[0008] Metadata may include information for mapping the position of an audio object to a single reproducing loudspeaker location. Presentation may include generating a set gain based on one or more of a desired audio object position, a distance from the desired audio object position to a reference position, the velocity of an audio object, or an audio object content type. Metadata may include data for confining the position of an audio object to a one-dimensional curve or a two-dimensional surface. Metadata may include track data for an audio object.

[0009] Presentation may include imposing restrictions on speaker areas. For example, the device may include a user input system. According to some embodiments, presentation may include using screen-based spatial balance control based on screen-based spatial balance control data received from the user input system.

[0010] The device may include a display system. The logic system may be configured to control the display system to display a dynamic three-dimensional view of the regenerated environment.

[0011] Rendering may include controlling the unfolding of audio objects in one or more dimensions in three dimensions. Rendering may include dynamic object changes in response to speaker load. Rendering may include mapping the location of audio objects onto a plane of a speaker array in the reproducing environment.

[0012] The device may include one or more non-transitory storage media, such as memory devices in a memory system. Memory devices may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, one or more hard disks, etc. The interface system may include an interface between the logic system and one or more such memory devices. The interface system may also include a network interface.

[0013] Metadata may include speaker region limitation metadata. The logic system may be configured to attenuate the selected speaker feedback signal by performing the following operations: calculating multiple first gains, which include contributions from the selected speaker; calculating multiple second gains, which do not include contributions from the selected speaker; and mixing the first and second gains. The logic system may be configured to determine whether to apply a localization rule to an audio object location or map an audio object location to a single speaker location. The logic system may be configured to smooth the transition in speaker gain when transitioning from mapping an audio object location from a first single speaker location to a second single speaker location. The logic system may be configured to smooth the transition in speaker gain when transitioning between mapping an audio object location to a single speaker location and applying a localization rule to the audio object location. The logic system may be configured to calculate the speaker gain for the audio object location along a one-dimensional curve between virtual speaker locations.

[0014] Some of the methods described herein include receiving audio playback data comprising one or more audio objects and associated metadata, and receiving playback environment data, including indications of a plurality of playback speakers in the playback environment. The playback environment data may include indications of the position of each playback speaker within the playback environment. The methods may include representing the audio objects as one or more speaker feedback signals based on at least a portion of the associated metadata. Each speaker feedback signal may correspond to at least one of the playback speakers within the playback environment. The playback environment may be a theater sound system environment.

[0015] Presentation may include generating a set gain based on one or more of the following: a desired audio object location, a distance from the desired audio object location to a reference location, the velocity of an audio object, or the content type of an audio object. Metadata may include data used to constrain the location of an audio object to a one-dimensional curve or a two-dimensional surface. Presentation may include imposing constraints on speaker regions.

[0016] Some implementations may be displayed on one or more non-transitory media having software stored thereon. The software may include multiple instructions for controlling one or more devices to perform the following operations: receiving audio playback data containing one or more audio objects and associated metadata; receiving playback environment data, which includes indications of a plurality of playback speakers in the playback environment and indications of the position of each playback speaker in the playback environment; and presenting the audio objects as one or more speaker feedback signals based on at least a portion of the associated metadata. Each speaker feedback signal may correspond to at least one of the playback speakers in the playback environment. The playback environment may, for example, be a theater sound system environment.

[0017] Presentation may include generating a set gain based on one or more of the following: a desired audio object location, a distance from the desired audio object location to a reference location, the velocity of an audio object, or an audio object content type. Metadata may include data used to constrain the location of an audio object to a one-dimensional curve or a two-dimensional surface. Presentation may include imposing constraints on multiple speaker regions. Presentation may include dynamic object smearing changes in response to speaker load.

[0018] Alternative devices and apparatuses are described herein. Some such devices may include an interface system, a user input system, and a logic system. The logic system may be configured to receive audio data via the interface system, receive the position of an audio object via the user input system or the interface system, and determine a position of the audio object in a three-dimensional space. The determination may include limiting the position to a one-dimensional curve or a two-dimensional surface in the three-dimensional space. The logic system may be configured to generate metadata about the audio object based on at least a portion of the user input received via the user input system, the metadata including data indicating the position of the audio object in three-dimensional space.

[0019] Metadata may include track data, which indicates a time-varying position of an audio object in three-dimensional space. The logic system may be configured to calculate the track data based on user input received via a user input system. Track data may include a set of positions in three-dimensional space at multiple time points. Track data may include an initial position, velocity data, and acceleration data. Track data may include an initial position and an equation defining the position and corresponding time in three-dimensional space.

[0020] The device may include a display system. The logic system may be configured to control the display system to display an audio object track based on track data.

[0021] The logic system can be configured to generate speaker area restriction metadata based on user input received via a user input system. The speaker area restriction metadata may include data for disabling selected speakers. The logic system can also be configured to generate speaker area restriction metadata by mapping the location of audio objects to a single speaker.

[0022] The device may include a sound reproduction system. The logic system may be configured to control the sound reproduction system based on at least a portion of the metadata.

[0023] The position of an audio object can be constrained to a one-dimensional curve. The logic system can be further configured to generate a virtual speaker position along the one-dimensional curve.

[0024] Alternative methods are described here. Some of these methods include receiving audio data, receiving the position of an audio object, and determining a position of the audio object in a three-dimensional space. The determination may include limiting the position to a one-dimensional curve or a two-dimensional surface within the three-dimensional space. The method may include generating metadata about the audio object based on at least a portion of user input.

[0025] Metadata may include data indicating the position of an audio object in three-dimensional space. Metadata may include track data, which indicates a time-varying position of an audio object in three-dimensional space. Generating metadata may include, for example, generating speaker region limiting metadata based on user input. Speaker region limiting metadata may include data for disabling selected speakers.

[0026] The position of audio objects can be constrained to a one-dimensional curve. The method further includes generating virtual speaker positions along a one-dimensional curve.

[0027] Other embodiments of this disclosure can be implemented on one or more non-transitory media having software stored thereon. The software may include multiple instructions for controlling one or more devices to perform the following operations: receiving audio data; receiving the position of an audio object; and determining a position of the audio object in a three-dimensional space. The determination may include limiting the position to a one-dimensional curve or a two-dimensional surface within the three-dimensional space. The software may include instructions for controlling one or more devices to generate metadata about the audio object. The metadata may be generated based on at least a portion of user input.

[0028] Metadata may include data indicating the position of an audio object in three-dimensional space. Metadata may include track data, which indicates a time-varying position of an audio object in three-dimensional space. Generating metadata may include, for example, generating speaker region limiting metadata based on user input. Speaker region limiting metadata may include data for disabling selected speakers.

[0029] The position of an audio object can be constrained to a one-dimensional curve. The software may include instructions for controlling one or more devices to generate virtual speaker positions along the one-dimensional curve.

[0030] One or more practical details of the main body described in this specification will be set forth in the accompanying drawings and the description below. Other features, appearances, and advantages will become apparent from the description, illustrations, and claims. Please note that the relative dimensions in the following illustrations may not be drawn to scale. Simple Explanation of the Diagram

[0031] Figure 1 shows an example of a regeneration environment with a Dolby surround 5.1 configuration. Figure 2 shows an example of a regeneration environment with a Dolby surround 7.1 configuration. Figure 3 shows an example of a regenerated environment with a Hamasaki 22.2 surround sound configuration. Figure 4A shows an example of a graphical user interface (GUI) depicting speaker areas at different heights within a virtual recreated environment. Figure 4B shows another example of a regenerative environment. Figures 5A-5C show examples of speaker responses corresponding to an audio object, wherein the audio object has a position on a two-dimensional surface that is confined to three-dimensional space. Figures 5D and 5E show an example of an audio object that can be confined to a two-dimensional surface. Figure 6A is a flowchart illustrating an example of the process of confining the position of an audio object to a two-dimensional surface. Figure 6B is a flowchart outlining an example of the process of mapping the location of an audio object to a single speaker location or a single speaker area. Figure 7 is a flowchart outlining the process of creating and using a virtual speaker. Figures 8A-8C show examples of virtual speakers mapped to line endpoints and corresponding speaker responses. Figures 9A-9C show examples of using a virtual rope to move an audio object. Figure 10A is a flowchart outlining the process of using a virtual rope to move an audio object. Figure 10B is a flowchart outlining another process of using a virtual rope to move an audio object. Figures 10C-10E show examples of the process described in Figure 10B. Figure 11 shows an example of imposing speaker area restrictions in a virtual regeneration environment. Figure 12 is a flowchart outlining some examples of applying the loudspeaker area restriction rule. Figures 13A and 13B show examples of GUIs that can switch between two-dimensional and three-dimensional views of a virtual regeneration environment. Figures 13C-13E show a combination of two-dimensional and three-dimensional depictions of the regenerated environment. Figure 14A is a flowchart outlining the process of controlling a device to present a GUI as shown in Figures 13C-13E. Figure 14B is a flowchart outlining the process of presenting audio objects for use in a regenerating environment. Figure 15A shows an instance of an audio object and the width of an associated audio object in a virtual regeneration environment. Figure 15B shows an example of a data chart corresponding to the distribution of audio object widths shown in Figure 15A. Figure 16 is a flowchart outlining the process of applying changes to audio objects. Figures 17A and 17B show examples of audio objects positioned in a three-dimensional virtual recreated environment. Figure 18 shows an example of a region that conforms to the location method. Figures 19A-19D show examples of using near-field and far-field localization techniques for audio objects in different locations. Figure 20 shows the speaker area of ​​the regenerated environment that can be used in the process of controlling spatial offset on the screen. Figure 21 is a block diagram showing an example of setting up, editing, and / or presenting device components. Figure 22A is a block diagram showing some of the components that can be used to generate audio content. Figure 22B is a block diagram showing some of the components that can be used to replay audio in a playback environment. The same reference numbers and names in the various figures refer to the same components. Implementation

[0032] The following description focuses on certain implementations to illustrate some innovative aspects of this disclosure and examples within the text that can implement these innovative aspects. However, the teachings herein can be applied in various different ways. For example, although the various implementations describe specific regeneration environments, the teachings herein can be broadly applied to other known regeneration environments, as well as regeneration environments that may be proposed in the future. Similarly, this document presents examples of graphical user interfaces (GUIs), while others present examples of speaker locations, speaker areas, etc., and the inventors will carefully consider other implementations. Furthermore, the implementations described can be implemented using various editing and / or presentation tools, which can be implemented using various hardware, software, and firmware. Therefore, the teachings of this disclosure are not intended to be limited to the implementations shown in the figures and / or described herein, but rather have a wide range of applications.

[0033] Figure 1 shows an example of a playback environment with a Dolby 5.1 surround sound configuration. While Dolby 5.1 surround sound was developed in the 1990s, this configuration is still widely used in theater sound systems. Projector 105 can be configured to project video images, such as those about a movie, onto screen 150. Audio playback data can be synchronized with the video images and processed by audio processor 110. Power amplifier 115 provides speaker feedback signals to the speakers of playback environment 100.

[0034] The Dolby Surround 5.1 configuration includes a left surround array 120 and a right surround array 125, each driven by a single channel array. The Dolby Surround 5.1 configuration also includes separate channels for the left screen channel 130, the center screen channel 135, and the right screen channel 140. The separate channel for the subwoofer 145 is for low-frequency effect (LFE).

[0035] In 2010, Dolby introduced Dolby Surround 7.1 to enhance digital theater audio. Figure 2 shows an example of a playback environment with a Dolby Surround 7.1 configuration. A digital projector 205 is configured to receive digital video data and project the video image onto a screen 150. Audio playback data can be processed by an audio processor 210. A power amplifier 215 provides speaker feedback signals to the speakers of the playback environment 200.

[0036] The Dolby Surround 7.1 configuration includes a left surround array 220 and a right surround array 225, each driven by a single channel. Like the Dolby Surround 5.1, the Dolby Surround 7.1 configuration includes separate channels for the left screen channel 230, the center screen channel 235, the right screen channel 240, and the subwoofer 245. However, Dolby Surround 7.1 increases the number of surround channels by dividing the left and right surround channels of Dolby Surround 5.1 into four zones (in addition to the left surround array 220 and the right surround array 225, the separate channels also include those for the left rear surround speaker 224 and the right rear surround speaker 226). Increasing the number of surround zones within the reproduction environment 200 significantly improves sound localization.

[0037] In an effort to create a more virtual environment, some recreated environments may be equipped with an increased number of speakers driven by an increased number of channels. Furthermore, some recreated environments may include speakers deployed at different heights, some above the seating area.

[0038] Figure 3 shows an example of a playback environment with a Hamasaki 22.2 surround sound configuration. The Hamasaki 22.2 was developed at NHK Science and Technology Research Laboratories in Japan as a surround sound element for ultra-high-definition televisions. The Hamasaki 22.2 provides 24 speaker channels, which can be used to drive speakers arranged in three layers. The upper speaker layer 310 of the playback environment 300 can be driven by 9 channels. The middle speaker layer 320 can be driven by 10 channels. The lower speaker layer 330 can be driven by 5 channels, two of which are for subwoofers 345a and 345b.

[0039] Therefore, the modern trend is not only to include more speakers and more channels, but also speakers at different heights. As the number of channels increases and speaker layouts shift from 2D arrays to 3D arrays, the task of positioning and presenting sound becomes increasingly difficult.

[0040] This disclosure presents various tools and related user interfaces that add functionality to 3D audio sound effects systems and / or reduce editing complexity.

[0041] Figure 4A shows an example of a graphical user interface (GUI) depicting speaker areas at different heights within a virtual playback environment. The GUI 400 can be displayed on a display device, for example, based on instructions from a logic system, signals received from a user input device, etc. Some such devices are described below with reference to Figure 21.

[0042] When used herein to refer to a virtual playback environment such as virtual playback environment 404, the term "speaker region" generally refers to a logical construct that may or may not correspond one-to-one with the playback speakers of an actual playback environment. For example, "speaker region location" may or may not correspond to a specific playback speaker location in a theater playback environment. Instead, the term "speaker region location" may generally refer to a region within the virtual playback environment. In some implementations, a speaker region in the virtual playback environment may correspond to a virtual speaker, for example, by using virtualization technology such as Dolby Headphone™ (sometimes called Mobile Surround™), which uses a set of two-channel stereo headphones to generate an instant virtual surround sound environment. In GUI 400, there are 7 speaker regions 402a at a first height and 2 speaker regions 402b at a second height, forming a total of 9 speaker regions in virtual playback environment 404. In this example, speaker regions 1-3 are in the front region 405 of virtual playback environment 404. The front area 405 may correspond, for example, to the area where the screen 150 is located in a theater playback environment, the area where the television screen is located in a home, and so on.

[0043] Here, speaker region 4 typically corresponds to the speaker in left region 410, and speaker region 5 corresponds to the speaker in right region 415 of virtual playback environment 404. Speaker region 6 corresponds to left rear region 412, and speaker region 7 corresponds to right rear region 414 of virtual playback environment 404. Speaker region 8 corresponds to the speaker in upper region 420a, and speaker region 9 corresponds to the speaker in upper region 420b, which may be the virtual ceiling region of virtual ceiling 520 as shown in Figures 5D and 5E. Therefore, as described in more detail below, the locations of speaker regions 1-9 shown in Figure 4A may or may not correspond to the locations of playback speakers in the actual playback environment. Furthermore, other implementations may include more or fewer speaker regions and / or heights.

[0044] In the various implementations described herein, a user interface such as GUI 400 may be used as part of the editing and / or rendering tools. In some implementations, the editing and / or rendering tools may be implemented via software stored in one or more non-transitory media. The editing and / or rendering tools may be implemented via software, firmware, etc. (such as the logic system and other devices described below with reference to Figure 21). In some editing implementations, associative editing tools may be used to generate metadata for associating audio data. The metadata may include, for example, data indicating the position and / or track of an audio object in three-dimensional space, speaker area limitation data, etc. The metadata may be generated with respect to the speaker area 402 of the virtual playback environment 404, rather than to a specific speaker layout of the actual playback environment. The rendering tool may receive the audio data and associated metadata and may calculate the audio gain and speaker feedback signal for the playback environment. The aforementioned audio gain and speaker feedback signal may be calculated according to an amplitude localization procedure that generates the perception of sound from position P in the playback environment. For example, the loudspeaker feedback signal can be provided to the regenerating loudspeakers 1 to N in the regenerating environment according to the following equation: xi(t) = gix(t), i = 1, ..., N (Equation 1)

[0045] In Equation 1, xi(t) represents the speaker feedback signal to be applied to speaker i, gi represents the gain factor of the corresponding channel, x(t) represents the audio signal, and t represents time. The gain factor can be determined, for example, according to the amplitude localization method described in paragraph 2, pages 3-4 of V. Pulkki, Compensating Displacement of Amplitude - Panned Virtual Sources (Audio Engineering Society (AES) International Conference on Virtual, Synthetic and Entertainment Audio), which is incorporated herein by reference. In some implementations, the gain may be frequency-dependent. In some implementations, a time delay can be introduced by replacing x(t) with x(t-Δt).

[0046] In some implementations, the audio reproduction data generated for speaker region 402 can be mapped to speaker locations in various playback environments (such as a Dolby surround 5.1 configuration, a Dolby surround 7.1 configuration, a Hamasaki 22.2 configuration, or other configurations). For example, referring to Figure 2, the presentation tool can map the audio reproduction data for speaker regions 4 and 5 to the left surround array 220 and right surround array 225 in a playback environment with a Dolby surround 7.1 configuration. The audio reproduction data for speaker regions 1, 2, and 3 can be mapped to the left screen channel 230, right screen channel 240, and center screen channel 235, respectively. The audio reproduction data for speaker regions 6 and 7 can be mapped to the left rear surround speaker 224 and right rear surround speaker 226.

[0047] Figure 4B shows an example of another playback environment. In some implementations, the rendering tool can map the audio playback data for speaker regions 1, 2, and 3 to the corresponding screen speakers 455 of the playback environment 450. The rendering tool can map the audio playback data for speaker regions 4 and 5 to the left surround array 460 and the right surround array 465, and can map the audio playback data for speaker regions 8 and 9 to the upper left speaker 470a and the upper right speaker 470b. The audio playback data for speaker regions 6 and 7 can be mapped to the left rear surround speaker 480a and the right rear surround speaker 480b.

[0048] In some editing practices, editing tools can be used to generate metadata for audio objects. As used herein, the term "audio object" can refer to a string of audio data and associated metadata. Metadata generally indicates the object's 3D position, rendering constraints, and content type (e.g., dialogue, effects, etc.). Depending on the implementation, metadata may include other types of data, such as width data, gain data, track data, etc. Some audio objects can be static, while others are movable. Audio object details can be edited or rendered based on associated metadata, which, among other things, can indicate the audio object's position at a specific point in three-dimensional space. When monitoring or replaying audio objects in a playback environment, the audio objects can be rendered based on position metadata that exists in the playback environment, rather than being output to predetermined physical channels, as is the case with traditional channel-based systems such as Dolby 5.1 and Dolby 7.1.

[0049] This section describes various editing and rendering tools for a GUI that is essentially the same as GUI 400. However, various other user interfaces (including but not limited to GUIs) can be used in conjunction with these editing and rendering tools. Some of these tools can simplify the editing process by imposing various types of restrictions. Some implementations will now be illustrated with reference to Figure 5A, etc.

[0050] Figures 5A-5C show examples of speaker responses corresponding to an audio object, where the audio object has a position confined to a two-dimensional surface in three-dimensional space (hemisphere in this example). In these examples, the renderer has calculated the speaker responses, assuming a nine-speaker configuration, with each speaker corresponding to one of the speaker regions 1-9. However, as mentioned elsewhere, there may typically be a one-to-one mapping between speaker regions in the virtual playback environment and the playback speakers in the playback environment. Referring first to Figure 5A, audio object 505 is shown in the front left portion of the virtual playback environment 404. Therefore, the speaker corresponding to speaker region 1 indicates high gain, while the speakers corresponding to speaker regions 3 and 4 indicate medium gain.

[0051] In this example, the location of audio object 505 can be changed by placing cursor 510 on audio object 505 and "dragging" audio object 505 to the desired location on the x,y plane of virtual playback environment 404. When the object is dragged toward the center of the playback environment, it is also mapped onto the surface of the hemisphere and its height increases. Here, the increase in the height of audio object 505 is indicated by increasing the diameter of the circle (representing audio object 505), as shown in Figures 5B and 5C. As audio object 505 is dragged to the top center of virtual playback environment 404, audio object 505 appears to become larger and larger. Alternatively or additionally, the height of audio object 505 can be indicated by changing color, brightness, numerical height indication, etc. When audio object 505 is positioned at the top center of virtual playback environment 404, as shown in Figure 5C, the speakers corresponding to speaker areas 8 and 9 indicate a large gain, while other speakers indicate a small or no gain.

[0052] In this implementation, the position of the audio object 505 is constrained to a two-dimensional surface, such as a spherical surface, elliptical surface, conical surface, cylindrical surface, wedge-shaped surface, etc. Figures 5D and 5E show examples of two-dimensional surfaces to which the audio object can be constrained. Figures 5D and 5E are cross-sectional views through the virtual playback environment 404, with the front region 405 shown on the left. In Figures 5D and 5E, the y-values ​​of the y-z axis increase towards the front region 405 of the virtual playback environment 404 to maintain consistency with the x-y axis orientation shown in Figures 5A-5C.

[0053] In the example shown in Figure 5D, the two-dimensional surface 515a is part of an ellipsoid. In the example shown in Figure 5E, the two-dimensional surface 515b is part of a wedge. However, the shape, orientation, and position of the two-dimensional surface 515 shown in Figures 5D and 5E are merely examples. In alternative implementations, at least a portion of the two-dimensional surface 515 may extend outside the virtual recreated environment 404. In some of the above implementations, the two-dimensional surface 515 may extend above the virtual ceiling 520. Therefore, the three-dimensional space within the extension of the two-dimensional surface 515 is not necessarily as large as the volume of the virtual recreated environment 404. In other implementations, audio objects may be limited to one-dimensional features, such as curves, straight lines, etc.

[0054] Figure 6A is a flowchart outlining an example of a process for constraining the position of an audio object to a two-dimensional surface. As with other flowcharts presented herein, the operations of process 600 are not necessarily performed in the order shown. Furthermore, process 600 (and other processes presented herein) may include more or fewer operations than those indicated and / or described in the figures. In this example, blocks 605 to 622 are performed by an editing tool, while blocks 624 to 630 are performed by a rendering tool. The editing tool and the rendering tool may be implemented in a single device or more than one device. Although Figure 6A (and other flowcharts presented herein) may give the impression that the editing and rendering processes are performed sequentially, in many implementations, the editing and rendering processes are performed substantially simultaneously. The editing and rendering processes may be interactive. For example, the results of editing operations may be sent to the rendering tool, and a user who can perform further editing based on these results can obtain corresponding results from the rendering tool.

[0055] In block 605, an instruction is received that the position of the audio object should be limited to a two-dimensional surface. This instruction may be received, for example, by the logic system of a device configured to provide editing and / or presentation tools. As in other implementations described herein, the logic system may operate according to instructions from software stored on non-transitory media, or according to firmware, etc. The instruction may be a signal from a user input device (such as a touchscreen, mouse, trackball, gesture recognition device, etc.) in response to user input.

[0056] In block 607, which is not essential, audio data is received. Block 607 is not essential in this example, as the audio data can also come directly to the presenter from another source (e.g., a mixer) that is time-synchronized with the metadata editing tool. In some of the above implementations, there may be an inherent mechanism to combine each audio stream with its corresponding incoming metadata stream to form an audio object. For example, the metadata stream may contain an identifier for the audio object, which represents, for example, a value from 1 to N. If the presenter is equipped with audio inputs also numbered from 1 to N, the presenter tool may automatically assume that the audio object consists of a metadata stream identified by a value (e.g., 1) and the audio data received on the first audio input. Similarly, any metadata stream identified as the number 2 can form an object with audio received on the second audio input channel. In some implementations, the audio and metadata may be pre-packaged by the editing tool to form an audio object, and the audio object may be provided to the presenter tool, for example, by transmission over a network as a TCP / IP packet.

[0057] In an alternative implementation, the editing tool can transmit only metadata over a network, while the presentation tool can receive audio from another source (e.g., via Pulse Code Modulation (PCM) streaming, via analog audio, etc.). In this type of implementation, the presentation tool can be configured to group audio data and metadata to form audio objects. The audio data can be received by the logic system, for example, via an interface. The interface can be, for example, a network interface, an audio interface (e.g., an interface configured to communicate via the AES3 standard developed by the Audio Engineering Society and the European Broadcasting Union (also known as AES / EBU), via the Multichannel Audio Digital Interface (MADI) protocol, via analog signals, etc.), or an interface between the logic system and a memory device. In this example, the data received by the presenter includes at least one audio object.

[0058] In block 610, the (x, y) or (x, y, z) coordinates of the audio object's position are received. Block 610 may, for example, include receiving the initial position of the audio object. Block 610 may also include receiving an indication that the user has positioned or repositioned the audio object, as described above with respect to figures 5A-5C. In block 615, the coordinates of the audio object are mapped onto a two-dimensional surface. The two-dimensional surface may be similar to one of those described with respect to figures 5D and 5E, or it may be a different two-dimensional surface. In this example, each point in the xy plane will be mapped to a single z value, so block 615 includes mapping the x and y coordinates received in block 610 to z values. In other implementations, different mapping procedures and / or coordinate systems may be used. The audio object may be displayed (block 620) at the (x, y, z) location determined in block 615. Audio data and metadata, including the (x, y, z) regions mapped in block 615, can be stored in block 621. The audio data and metadata can be transferred to the rendering tool (block 622). In some implementations, metadata can be transferred continuously while editing operations are in progress, for example, when locating, limiting, or displaying audio objects in GUI 400.

[0059] In block 623, a decision is made as to whether the editing process should continue. For example, the editing process can end once the user interface receives an input indicating that the user no longer wants to restrict the position of the audio object to a two-dimensional surface (block 625). Otherwise, the editing process can continue, for example, by returning to block 607 or block 610. In some implementations, the presentation operation can continue regardless of whether the editing process continues. In some implementations, the audio object can be recorded to a disk on the editing platform and then replayed from a dedicated sound processor or a theater server connected to a sound processor (such as a sound processor similar to sound processor 210 in Figure 2) for presentation.

[0060] In some implementations, the rendering tool can be software running on a device configured to provide editing functionality. In other implementations, the rendering tool can be located on a different device. The type of communication protocol used for communication between the editing tool and the rendering tool can vary depending on whether both tools run on the same device or whether they communicate via a network.

[0061] In block 626, the rendering tool receives audio data and metadata (including the (x, y, z) positions determined in block 615). In an alternative implementation, the rendering tool can receive the audio data and metadata separately and treat them as audio objects through an inherent mechanism. As mentioned above, for example, the metadata stream may contain audio object identification codes (e.g., 1, 2, 3, etc.) and may be respectively attached to the first, second, and third audio inputs (i.e., digital or analog audio connections) on the rendering system to form audio objects that can be rendered to the speakers.

[0062] During the presentation operation of process 600 (and other presentation operations described herein), a positioning gain equation can be applied based on the layout of the reproducing loudspeakers in a specific reproduction environment. Therefore, the logic system of the presentation tool can receive reproduction environment data, which includes indications of multiple reproducing loudspeakers in the reproduction environment and indications of the position of each reproducing loudspeaker within the reproduction environment. This data can be received, for example, by accessing a data structure stored in memory accessible to the logic system, or via an interface system.

[0063] In this example, the positional gain equation is applied to the (x, y, z) location to determine the gain value (block 628) applied to the audio data (block 630). In some implementations, the audio data, already adjusted to reflect the gain value, can be reproduced by a playback speaker, such as the speaker (or other speaker) of a headset configured to communicate with the logic system of the presentation tool. In some implementations, the playback speaker location can correspond to the speaker location in a virtual playback environment (virtual playback environment 404 as described above). The corresponding speaker response can be displayed on a display device, for example, as shown in Figures 5A-5C.

[0064] In block 635, a decision is made as to whether the process should continue. For example, the process can end once the user interface receives input indicating that the user no longer wants to continue presenting the process (block 640). Otherwise, the process can continue, for example, by returning to block 626. If the logic system receives an instruction from the user that they want to return to the corresponding editing process, process 600 can return to block 607 or block 610.

[0065] Other implementations may include imposing various other types of restrictions and generating other types of restriction metadata for audio objects. Figure 6B is a flowchart outlining an example of a process for mapping an audio object location to a single speaker location. This process may also be referred to herein as a "snapshot". In block 655, an indication is received that the audio object location can be snapshotted to a single speaker location or a single speaker area. In this example, when appropriate, the audio object location is indicated to be snapshotted to a single speaker location. The indication may, for example, be received by the logic system of a device configured to provide editing tools. The indication may conform to input received from a user input device. However, the indication may also conform to the type of audio object (e.g., as a gunshot sound, a sound effect, etc.) and / or the width of the audio object. For example, information about the type and / or width may be received as metadata for the audio object. In such an implementation, block 657 may occur before block 655.

[0066] In block 656, audio data is received. In block 657, the coordinates of the audio object's position are received. In this example, the audio object's position is displayed based on the coordinates received in block 657 (block 658). In block 659, metadata including the audio object's coordinates and a snapshot flag (indicating snapshot functionality) is stored. The audio data and metadata are then sent from the editing tool to the rendering tool (block 660).

[0067] In block 662, it is determined whether the editing process should continue. For example, the editing process can end once the user interface receives an input indicating that the user no longer wants to snapshot the audio object's position to the speaker area (block 663). Otherwise, the editing process can continue, for example, by returning to block 665. In some implementations, the presentation operation can continue regardless of whether the editing process continues.

[0068] In block 664, the presentation tool receives audio data and metadata transmitted by the editing tool. In block 665, it is determined (e.g., by a logic system) whether to snapshot the audio object location to the speaker location. This determination can be based on the distance between at least a portion of the audio object location and the nearest reproducing speaker location in the playback environment.

[0069] In this example, if it is decided to snapshot the audio object position to a speaker location in block 665, then in block 670, the audio object position will be mapped to a speaker location, which is typically the location closest to the expected (x, y, z) position received by the audio object. In this case, the gain of the audio data reproduced by the speaker location will be 1.0, while the gain of the audio data reproduced by other speakers will be zero. In an alternative implementation, the audio object position can be mapped to a group of speaker locations in block 670.

[0070] For example, referring again to Figure 4B, block 670 may include a snapshot of the position of an audio object onto one of the upper left speakers 470a. Alternatively, block 670 may include a snapshot of the position of an audio object onto a single speaker and neighboring speakers, such as one or two neighboring speakers. Thus, the corresponding metadata can be applied to a small group of reproducing speakers and / or individual reproducing speakers.

[0071] However, if determining the position of an audio object in square 665 would not result in a direct view of the speaker area, for example, if it would cause a significant difference between the position and the original expected position of the object, then a positioning rule (square 675) will be applied. Positioning rules can be applied based on the position of the audio object, as well as other characteristics of the audio object (such as width, volume, etc.).

[0072] The gain data determined in block 675 can be applied to the audio data in block 681, and the result can be stored. In some implementations, the generated audio data can be reproduced by a speaker configured to communicate with the logic system. If it is determined in block 685 that process 650 will continue, process 650 can return to block 664 to continue the presentation operation. Alternatively, process 650 can return to block 655 to restart the editing operation.

[0073] Process 650 may include various types of smoothing operations. For example, the logic system may be configured to smooth the transition in gain applied to the audio data when the location of an audio object changes from a first single speaker location to a second single speaker location. Referring again to Figure 4B, if the location of an audio object is initially mapped to one of the upper left speakers 470a and then to one of the right rear surround speakers 480b, the logic system may be configured to smooth the transition between speakers so that the audio object does not appear to "jump" abruptly from one speaker (or speaker area) to another. In some implementations, smoothing may be implemented based on a cross-fading ratio parameter.

[0074] In some implementations, the logic system can be configured to smooth the transition in gain applied to the audio data when shifting between mapping the position of an audio object to the position of a single speaker and applying a positioning rule to the position of the audio object. For example, if it is later determined in block 665 that the position of the audio object has moved to a position determined to be too far from the nearest speaker, a positioning rule can be applied to the position of the audio object in block 675. However, when transitioning from snapshot to positioning (or vice versa), the logic system can be configured to smooth the transition in gain applied to the audio data. The process can end in block 690, for example, once corresponding input is received from the user interface.

[0075] Some alternative implementations may include creating logical constraints. In some cases, for example, during a specific positioning operation, the mixer may want more explicit control over the speaker array being used. Some implementations allow the user to create a one- or two-dimensional "logical mapping" between the speaker array and the positioning interface.

[0076] Figure 7 is a flowchart outlining the process of creating and using a virtual speaker. Figures 8A-8C show examples of virtual speakers mapped to line endpoints and corresponding speaker responses. Referring first to process 700 in Figure 7, an instruction to create a virtual speaker is received in block 705. This instruction can be received, for example, via the logic system of an editing device and can correspond to input received from a user input device.

[0077] In block 710, an instruction for the location of the virtual speaker is received. For example, referring to Figure 8A, the user can use a user input device to position the cursor 510 at the location of the virtual speaker 805a and select that location, for example, by clicking with the mouse. In block 715, a decision is made (e.g., based on user input) to select an additional virtual speaker in this example. The process returns to block 710, and in this example, the user selects the location of the virtual speaker 805b displayed in Figure 8A.

[0078] In this example, the user only wants to establish two virtual speaker locations. Therefore, in block 715, it is decided (e.g., based on user input) that no additional virtual speakers will be selected. As shown in Figure 8A, a polyline 810 connecting the positions of virtual speakers 805a and 805b can be displayed. In some implementations, the position of audio object 505 will be constrained to polyline 810. In some implementations, the position of audio object 505 can be constrained to a parametric curve. For example, a set of control points can be provided based on user input, and a curve fitting algorithm such as a spline can be used to determine the parametric curve. In block 725, an indication of the audio object's position along polyline 810 is received. In some of the above implementations, the position will be indicated as a scalar value between zero and one. In block 725, the (x, y, z) coordinates of the audio object and the polyline defined by the virtual speakers can be displayed. It can display audio data and associated metadata, including the obtained scalar position and the (x,y,z) coordinates of the virtual speaker (block 727). Here, in block 728, the audio data and metadata can be sent to the rendering tool through an appropriate communication protocol.

[0079] In block 729, a decision is made as to whether the editing process should continue. If not, process 700 can either end based on user input (block 730) or continue the presentation operation. However, as mentioned above, in many implementations, at least some presentation operations can be performed concurrently with editing operations.

[0080] In block 732, the presentation tool receives audio data and metadata. In block 735, the gain to be applied to the audio data is calculated for each virtual speaker location. Figure 8B shows the speaker response at the location of virtual speaker 805a. Figure 8C shows the speaker response at the location of virtual speaker 805b. In this example, as in many other instances described herein, the speaker response refers to the regenerated speaker with a location corresponding to the speaker area shown in GUI 400. Here, virtual speakers 805a and 805b, and line 810, are positioned on a plane not close to the regenerated speakers with locations corresponding to speaker areas 8 and 9. Therefore, figures 8B and 8C indicate that no gain is applied to these speakers.

[0081] When the user moves the audio object 505 to other positions along line 810, the logic system will calculate the cross-fading (block 740) corresponding to these positions, for example, based on the scalar position parameters of the audio object. In some implementations, pairwise positioning rules (e.g., energy conservation sine or dynamic laws) can be used to mix the gain of the audio data to be applied to the position of the virtual speaker 805a with the gain of the audio data to be applied to the position of the virtual speaker 805b.

[0082] In block 742, a decision can then be made (e.g., based on user input) whether to continue process 700. The user can, for example, suggest (e.g., via a GUI) whether to continue the presentation operation or return to the editing operation. If it is decided that process 700 will not continue, the process ends (block 745).

[0083] When positioning fast-moving audio objects (e.g., audio objects equivalent to cars, jets, etc.), it can be difficult to edit smooth tracks if the user selects the audio object's position point by point. The lack of smoothness in the audio object track can affect the perceived sound image. Therefore, some editing practices presented here apply a low-pass filter to the audio object's position to smooth the generated positioning gain. Alternative editing practices apply a low-pass filter to the gain used for the audio data.

[0084] Other editing implementations allow users to simulate grabbing, dragging, throwing audio objects, or similar interactions with audio objects. Some of these implementations may include simulations of the application of physical laws, such as sets of laws used to describe applications of velocity, acceleration, momentum, kinetic energy, and force.

[0085] Figures 9A-9C show an example of using a virtual rope to drag an audio object. In Figure 9A, a virtual rope 905 is formed between the audio object 505 and the cursor 510. In this example, the virtual rope 905 has a virtual spring constant. In some such implementations, the virtual spring constant can be selected based on user input.

[0086] Figure 9B shows the audio object 505 and cursor 510 at a subsequent moment, after the user has moved cursor 510 toward speaker area 3. The user can use a mouse, joystick, trackball, gesture detection device, or other type of user input device to move cursor 510. The virtual cord 905 has been extended, and the audio object 505 has moved closer to speaker area 8. The audio object 505 is approximately the same size in Figures 9A and 9B, which means that (in this example) the height of the audio object 505 has not changed substantially.

[0087] Figure 9C shows the audio object 505 and cursor 510 at a later time, after the user has moved the cursor to the vicinity of speaker area 9. The virtual rope 905 has been further extended. The audio object 505 has been moved downwards, as shown by reducing the size of the audio object 505. The audio object 505 has moved in a smooth arc. This example demonstrates a potential advantage of the above implementation: the audio object 505 can move in a smoother trajectory compared to if the user simply selects the position of the audio object 505 point by point.

[0088] Figure 10A is a flowchart outlining the process of moving an audio object using a virtual rope. Process 1000 begins with block 1005, where audio data is received. In block 1007, an instruction is received to attach a virtual rope between the audio object and the cursor. The instruction can be received by the logic system of the editing device and can conform to input received from the user input device. Referring to Figure 9A, for example, the user can position the cursor 510 on the audio object 505 and then instruct via the user input device or GUI that a virtual rope 905 should be formed between the cursor 510 and the audio object 505. Cursor and object position data can be received (block 1010).

[0089] In this example, when the cursor 510 moves, the logic system can calculate the cursor velocity and / or acceleration data based on the cursor position data (block 1015). The position and / or track data of the audio object 505 can be calculated based on the virtual spring constant of the virtual rope 905 and the cursor position, velocity, and acceleration data. Some implementations of this type may include assigning a virtual mass to the audio object 505 (block 1020). For example, if the cursor 510 moves at a relatively constant speed, the virtual rope 905 may not extend and can pull the audio object 505 at a relatively constant speed. If the cursor 510 accelerates, the virtual rope 905 can extend and a corresponding force can be applied to the audio object 505 by means of the virtual rope 905. There may be a time delay between the acceleration of the cursor 510 and the force applied by the virtual rope 905. In alternative implementations, the position and / or track of the audio object 505 can be determined in different ways, such as by not specifying a virtual spring constant for the virtual rope 905, by applying the laws of friction and / or inertia to the audio object 505, and so on.

[0090] The discrete position and / or track of audio object 505, along with a cursor 510 (block 1025), can be displayed. In this example, the logic system samples the position of the audio object at time intervals (block 1030). In some such implementations, the user can determine the time interval used for sampling. Audio object location and / or track metadata, etc., can be stored (block 1034).

[0091] In block 1036, decide whether this editing mode should continue. If the user wishes to continue, the process can be resumed, for example, by returning to block 1005 or block 1010. Otherwise, process 1000 can end (block 1040).

[0092] Figure 10B is a flowchart outlining another process for moving an audio object using a virtual rope. Figures 10C-10E show examples of the process described in Figure 10B. Referring first to Figure 10B, process 1050 begins with block 1055, where audio data is received. In block 1057, an instruction is received to attach a virtual rope between the audio object and the cursor. The instruction can be received by the logic system of the editing device and can conform to input received from a user input device. Referring to Figure 10C, for example, a user can position the cursor 510 on the audio object 505 and then instruct via a user input device or GUI that a virtual rope 905 should be formed between the cursor 510 and the audio object 505.

[0093] In block 1060, cursor and audio object position data can be received. In block 1062, the logic system can receive (e.g., via a user input device or GUI) an indication that the audio object 505 should remain at a specified position (e.g., the position indicated by cursor 510). In block 1065, the logic device receives an indication that cursor 510 has moved to a new position, which may be displayed along with the position of the audio object 505 (block 1067). Referring to figure 10D, for example, cursor 510 has moved from the left to the right side of the virtual playback environment 404. However, the audio object 505 remains at the same position indicated in figure 10C. Therefore, the virtual rope 905 has substantially been extended.

[0094] In block 1069, the logic system receives an instruction that audio object 505 will be released (e.g., via a user input device or GUI). The logic system can calculate the resulting audio object position and / or track data, which can be displayed (block 1075). The resulting display may resemble that shown in Figure 10E, showing audio object 505 moving smoothly and quickly through virtual playback environment 404. The logic system can store audio object location and / or track metadata in the memory system (block 1080).

[0095] In block 1085, it is determined whether the editing process 1050 will continue. If the logic system receives an instruction from the user to continue, the process can continue. For example, process 1050 can continue by returning to block 1055 or block 1060. Otherwise, the editing tool can send the audio data and metadata to the rendering tool (block 1090), after which process 1050 can end (block 1095).

[0096] To optimize the realism of perceived movement of audio objects, it is desirable to allow users of editing (or rendering) tools to select a subset of speakers in the playback environment and restrict the effective speaker combinations to within the selected subset. In some implementations, speaker regions and / or groups of speaker regions can be designated as disabled or enabled during editing or rendering operations. For example, referring to Figure 4A, the speaker regions of front region 405, left region 410, right region 415, and / or upper region 420 can be controlled as a group. The speaker regions of the rear region, including speaker regions 6 and 7 (and, in other implementations, one or more other speaker regions located between speaker regions 6 and 7), can also be controlled as a group. The user interface can be configured to dynamically enable or disable all speakers corresponding to a specific speaker region or a region comprising multiple speaker regions.

[0097] In some implementations, the logic system of the editing device (or presentation device) can be configured to generate speaker region restriction metadata based on user input received through a user input system. The speaker region restriction metadata may include information used to disable selected speaker regions. Some such implementations will now be illustrated with reference to Figures 11 and 12.

[0098] Figure 11 illustrates an example of imposing speaker area restrictions in a virtual playback environment. In some implementations of this type, the user can select speaker areas by clicking on a representative image in a GUI (such as GUI 400) using a user input device such as a mouse. Here, the user has disabled speaker areas 4 and 5 on the sides of the virtual playback environment 404. Speaker areas 4 and 5 may correspond to most (or all) of the speakers in an actual playback environment (such as a theater sound system environment). In this example, the user has also restricted the position of audio object 505 to the position along line 1105. By disabling most or all of the speakers along the side walls, the area from screen 150 to the back of the virtual playback environment 404 is restricted from using side speakers. This can create an enhanced sense of motion from front to back for a large audience area, especially for audience members seated near the playback speakers corresponding to speaker areas 4 and 5.

[0099] In some practices, speaker region limiting can be implemented in all re-rendering modes. For example, speaker region limiting can be implemented when a small number of regions are available for rendering, such as when rendering a Dolby surround 7.1 or 5.1 configuration that exposes only 7 or 5 regions. Speaker region limiting can also be implemented when more regions are available for rendering. In itself, speaker region limiting can also be seen as a method of manipulating re-rendering, providing a non-blind solution to the traditional "up-mix / down-mix" process.

[0100] Figure 12 is a flowchart outlining some examples of applying speaker zone restriction rules. Process 1200 begins with block 1205, where one or more instructions are received to apply speaker zone restriction rules. Instructions may be received by the logic system of an editing or presentation device and may correspond to input received from a user input device. For example, an instruction may correspond to a user's selection of one or more speaker zones to be undone. In some implementations, block 1205 may include receiving an instruction on what type of speaker zone restriction rule should be applied, as described below.

[0101] In block 1207, the editing tool receives audio data. Audio object position data can be received, for example, based on input from the user of the editing tool (block 1210) and displayed (block 1215). In this example, the position data is (x, y, z) coordinates. Here, the valid and invalid speaker regions used for the selected speaker region restriction rules are also displayed in block 1215. In block 1220, the audio data and associated metadata are stored. In this example, the metadata includes audio object position and speaker region restriction metadata, which may include speaker region identification flags.

[0102] In some implementations, speaker region limitation metadata can instruct the presentation tool to use positioning equations to calculate gain in binary form, for example, by treating all speakers in the selected (disabled) speaker region as "off" and all other speaker regions as "on". The logic system can be configured to generate speaker region limitation metadata that includes data used to disable the selected speaker region.

[0103] In alternative implementations, loudspeaker region limiting metadata can instruct the rendering tool to use a positioning equation to calculate a mixed gain that includes levels of contribution from loudspeakers in disabled loudspeaker regions. For example, a logic system can be configured to generate loudspeaker region limiting metadata that instructs the rendering tool to attenuate selected loudspeaker regions by performing the following operations: calculating multiple first gains that include contributions from the selected (disabled) loudspeaker regions; calculating multiple second gains that do not include contributions from the selected loudspeaker regions; and mixing the first and second gains. In some implementations, a bias can be applied to the first and / or second gains (e.g., from a selected minimum to a selected maximum) to allow for a range of potential contributions from the selected loudspeaker regions.

[0104] In this example, in block 1225, the editing tool transmits audio data and metadata to the rendering tool. The logic system can then decide whether the editing process should continue (block 1227). If the logic system receives an instruction from the user to continue, the editing process can continue. Otherwise, the editing process can end (block 1229). In some implementations, the rendering operation can continue based on user input.

[0105] Audio objects, including audio data and metadata generated by the editing tools, are received by the rendering tool in block 1230. In this example, position data for a specific audio object is received in block 1235. The rendering tool's logic system can use positioning equations based on speaker area limitation laws to calculate the gain for the audio object's position data.

[0106] In block 1245, the calculated gain is applied to the audio data. The logic system can store the gain, audio object location, and speaker area limitation data in the memory system. In some implementations, the audio data can be reproduced by the speaker system. The corresponding speaker response can be displayed on a monitor in some implementations.

[0107] In block 1248, a decision is made as to whether process 1200 should continue. If the logic system receives an instruction from the user to continue, the process can continue. For example, the presentation process can continue by returning to block 1230 or block 1235. If the system receives an instruction from the user to return to the corresponding editing process, the process can return to block 1207 or block 1210. Otherwise, process 1200 can end (block 1250).

[0108] Positioning and rendering audio objects in a 3D virtual playback environment is becoming increasingly difficult. Part of the challenge lies in representing the virtual playback environment within the GUI. Some of the editing and rendering implementations presented here allow users to switch between 2D and 3D screen space positioning. This functionality helps maintain the accuracy of audio object positioning while providing a user-friendly GUI.

[0109] Figures 13A and 13B show an example of a GUI that can switch between two-dimensional and three-dimensional views in a virtual playback environment. Referring first to Figure 13A, GUI 400 depicts image 1305 on the screen. In this example, image 1305 is a saber-toothed tiger. In the top view of the virtual playback environment 404, the user can immediately see that the audio object 505 is close to speaker area 1. For example, the height can be inferred from the size, color, or other attributes of the audio object 505. However, the relationship of position to image 1305 may be difficult to determine in this view.

[0110] In this example, GUI 400 can appear to rotate dynamically around an axis such as axis 1310. Figure 13B shows GUI 1300 after the rotation process. In this view, the user can see image 1305 more clearly and use information from image 1305 to more accurately position audio object 505. In this example, the audio object corresponds to the sound the saber-toothed tiger is facing. The ability to switch between the top view and the screen view of the virtual playback environment 404 allows the user to immediately and accurately select the appropriate height for audio object 505 using information from the material on the screen.

[0111] Various other convenient GUIs for editing and / or presentation are presented herein. Figures 13C-13E show a combination of two-dimensional and three-dimensional depictions of the playback environment. Referring first to Figure 13C, the top view of the virtual playback environment 404 is depicted in the left region of GUI 400. GUI 400 also includes a three-dimensional depiction 1345 of the virtual (or actual) playback environment. The region 1350 of the three-dimensional depiction 1345 corresponds to the screen 150 of GUI 400. The position of the audio object 505, especially its height, can be clearly seen in the three-dimensional depiction 1345. In this example, the width of the audio object 505 is also displayed in the three-dimensional depiction 1345.

[0112] Speaker layout 1320 depicts speaker locations 1324 to 1340, each indicating the gain corresponding to the position of audio object 505 in the virtual playback environment 404. In some implementations, speaker layout 1320 may, for example, represent the playback speaker locations of an actual playback environment (such as a Dolby surround 5.1 configuration, a Dolby surround 7.1 configuration, a Dolby 7.1 configuration with speakers at higher elevations, etc.). When the logic system receives an indication of the position of audio object 505 in the virtual playback environment 404, the logic system may be configured to map this position to the gain for speaker locations 1324 to 1340 of speaker layout 1320, for example, by means of the amplitude positioning procedure described above. For example, in Figure 13C, speaker locations 1325, 1335, and 1337 each have a color change, indicating the gain corresponding to the position of audio object 505.

[0113] Referring now to Figure 13D, the audio object has been moved to a position behind screen 150. For example, a user can move the audio object 505 by placing a cursor in GUI 400 on it and dragging it to the new position. This new position is also shown in the 3D rendering 1345, where it has been rotated to the new orientation. The response of speaker layout 1320 can essentially appear in the same way in Figures 13C and 13D. However, in the actual GUI, speaker locations 1325, 1335, and 1337 can have different appearances (such as different brightness or color) to indicate the corresponding gain differences caused by the new position of audio object 505.

[0114] Referring now to Figure 13E, audio object 505 has rapidly moved to the rear right position of the virtual regeneration environment 404. At the moment shown in Figure 13E, speaker location 1326 reflects the current position of audio object 505, while speaker locations 1325 and 1337 still reflect the previous position of audio object 505.

[0115] Figure 14A is a flowchart outlining the process of controlling a device to present a GUI as shown in Figures 13C-13E. Process 1400 begins at block 1405, where one or more instructions are received to display audio object locations, speaker locations, and playback speaker locations for the playback environment. Speaker locations may correspond to virtual playback environments and / or actual playback environments, as shown in Figures 13C-13E, for example. Instructions may be received by the logic system of the presentation and / or editing device and may conform to input received from a user input device. For example, instructions may conform to the user's selection of playback environment configuration.

[0116] In block 1407, audio data is received. In block 1410, audio object position data and width are received, for example, based on user input. In block 1415, the audio object, speaker area location, and regenerating speaker location are displayed. The audio object position can be displayed in two-dimensional and / or three-dimensional views, as shown in Figures 13C-13E, for example. Width data is not only used for the presentation of the audio object but also affects how the audio object is displayed (see the depiction of audio object 505 in the three-dimensional depiction 1345 of Figures 13C-13E).

[0117] Audio data and associated metadata can be recorded (block 1420). In block 1425, the editing tool transmits the audio data and metadata to the rendering tool. The logic system can then decide (block 1427) whether the editing process should continue. If the logic system receives an instruction from the user that they want to continue, the editing process can continue (e.g., by returning to block 1405). Otherwise, the editing process can end (block 1429).

[0118] Audio objects, including audio data and metadata generated by the editing tools, are received by the rendering tool in block 1430. In this example, position data for a specific audio object is received in block 1435. The rendering tool's logic system can use a positioning equation based on the width metadata to calculate the gain used for the audio object's position data.

[0119] In some implementations, the logic system can map speaker regions to the reproducing speakers in the reproduction environment. For example, the logic system can access a data structure that includes speaker regions and corresponding reproducing speaker locations. Further details and examples are illustrated below with reference to Figure 14B.

[0120] In some implementations, for example, a positioning equation (block 1440) can be applied by a logic system based on the position, width, and / or other information (such as the speaker location in the reproducing environment). In block 1445, the audio data is processed based on the gain obtained in block 1440. If necessary, at least some of the generated audio data can be stored along with the corresponding audio object position data and other metadata received from the editing tool. The speaker can reproduce the audio data.

[0121] The logic system can then decide (block 1448) whether process 1400 should continue. If, for example, the logic system receives an instruction from the user that they want to continue, process 1400 can continue. Otherwise, process 1400 can end (block 1449).

[0122] Figure 14B is a flowchart outlining the process of presenting audio objects for a playback environment. Process 1450 begins at block 1455, where one or more instructions are received to present audio objects for the playback environment. Instructions may be received by the logic system of the presentation device and may correspond to input received from a user input device. For example, the instructions may correspond to the user's selection of playback environment configuration.

[0123] In block 1457, audio playback data (including one or more audio objects and associated metadata) is received. Playback environment data can be received in block 1460. The playback environment data may include indications of multiple playback speakers in the playback environment and indications of the location of each playback speaker within the playback environment. The playback environment may be a theater sound system environment, a home theater environment, etc. In some implementations, the playback environment data may include playback speaker area layout data, indicating multiple playback speaker areas and multiple playback speaker locations corresponding to the speaker areas.

[0124] The regenerated environment can be displayed in block 1465. In some implementations, the regenerated environment can be displayed in a manner similar to speaker layout 1320 shown in Figures 13C-13E.

[0125] In block 1470, an audio object can be presented as one or more speaker feedback signals for a playback environment. In some implementations, the metadata associated with the audio object can be edited as described above, such that the metadata may include gain data corresponding to speaker regions (e.g., speaker regions 1-9 corresponding to GUI 400). The logic system can map speaker regions to playback speakers in the playback environment. For example, the logic system can access a data structure stored in memory that includes speaker regions and corresponding playback speaker locations. The presentation device can have various of the above-described data structures, each corresponding to a different speaker configuration. In some implementations, the presentation device can have the above-described data structures for various standard playback environment configurations (such as Dolby surround 5.1 configuration, Dolby surround 7.1 configuration, and / or Hamasaki 22.2 surround sound configuration).

[0126] In some implementations, the metadata for audio objects may include additional information from the editing process. For example, metadata may include speaker limitation data. Metadata may include information for mapping the position of an audio object to a single-reproducer speaker location or region. Metadata may include data that limits the position of an audio object to a one-dimensional curve or a two-dimensional surface. Metadata may include track data for the audio object. Metadata may include identifiers regarding the content type (e.g., dialogue, music, or effects).

[0127] Therefore, the presentation process may include the use of metadata, such as imposing restrictions on speaker regions. In some such implementations, the presentation device may provide the user with the option to modify the restrictions indicated by the metadata, such as modifying speaker restrictions and re-presenting accordingly. Presentation may include generating a set gain based on one or more of the following: the desired audio object location, the distance from the desired audio object location to a reference location, the speed of the audio object, or the audio object content type. The corresponding response of the reproduced speaker may be displayed (block 1475). In some implementations, the logic system may control the speaker to reproduce sound corresponding to the result of the presentation process.

[0128] In block 1480, the logic system can determine whether process 1450 should continue. If, for example, the logic system receives an instruction from the user that they want to continue, process 1450 can continue. For example, process 1450 can continue by returning to block 1457 or block 1460. Otherwise, process 1450 can end (block 1485).

[0129] Expanding and source width control are features of some existing surround sound editing / presentation systems. In this disclosure, "expanding" refers to distributing the same signal across multiple speakers to blur the sound image. "Width" refers to removing the association between the output signal and each channel for source width control. Width can be an additional scalar value controlling the amount of deassociation applied to the feedback signal from each speaker.

[0130] Some of the implementations described herein present 3D axis-guided unfolding control. An implementation of this type will now be illustrated with reference to Figures 15A and 15B. Figure 15A shows an example of an audio object and its associated width in a virtual playback environment. Here, GUI 400 displays an ellipsoid 1505 enlarged around the audio object 505, indicating the width of the audio object. The width of the audio object may be indicated by the audio object metadata and / or received based on user input. In this example, the x and y dimensions of ellipsoid 1505 are different, but in other implementations, these dimensions may be the same. The z dimension of ellipsoid 1505 is not shown in Figure 15A.

[0131] Figure 15B shows an example of a distribution data chart corresponding to the width of the audio object shown in Figure 15A. The distribution can be represented as a three-dimensional vector parameter. In this example, the distribution data chart 1507 is controlled independently along three dimensions, for example, based on user input. The gain along the x and y axes is represented in Figure 15B by the respective heights of curves 1510 and 1520. The gain for each sample 1512 is also indicated by the size of the corresponding circle 1515 within the distribution data chart 1507. The response of the speaker 1510 is indicated by the gray shading in Figure 15B.

[0132] In some implementations, the distribution data chart 1507 can be implemented by integrating each axis separately. According to some implementations, when positioning, the minimum distribution value can be automatically set as a function of speaker placement to avoid timbre mismatch. Alternatively or additionally, the minimum distribution value can be automatically set as a function of the velocity of the positioned audio object, causing the object to become more spatially distributed as the velocity of the audio object increases, much like the blurring of a rapidly moving image in a moving picture.

[0133] When using audio object-based audio rendering implementations (as described herein), a significant amount of audio tracks and accompanying metadata (including, but not limited to, metadata indicating the position of audio objects in three-dimensional space) may be transmitted unmixed to the playback environment. Real-time rendering tools can use the aforementioned metadata and information about the playback environment to calculate speaker feedback signals to optimize the playback of each audio object.

[0134] When a large number of audio objects are simultaneously mixed into the speaker output, load occurs in the digital domain (e.g., when a digital signal is clipped before analog conversion) or in the analog domain when a reproducing speaker replays an amplified analog signal. Both situations can lead to undesirable auditory distortion. Load in the analog domain can also damage reproducing speakers.

[0135] Therefore, some of the practices described herein involve "smearing variations" in response to the load on the regenerator speakers. In some practices, when the audio object is represented by a specific distribution data chart, the energy remains a fixed overall energy across an increasing number of neighboring regenerator speakers. For example, if the energy for the audio object is not uniformly distributed across N regenerator speakers, a gain of 1 / sqrt(N) can be contributed to the output of each regenerator speaker. This method provides additional mixing "margin" and can mitigate or prevent regenerator speaker distortion (such as clipping).

[0136] To illustrate with numerical examples, assume that a speaker will clip if it receives an input greater than 1.0. Suppose two objects are instructed to be mixed into speaker A, one at level 1.0 and the other at level 0.25. Without smearing variations, the total mix level in speaker A is 1.25, and clipping occurs. However, if the first object is smeared with another speaker B, then (according to some implementations) each speaker will receive 0.707 of the object, creating additional "spare space" in speaker A to mix the extra object. The second object can then be safely mixed into speaker A without clipping, because the mix level for speaker A will be 0.707 + 0.25 = 0.957.

[0137] In some implementations, during the editing phase, each audio object can be mixed with a specific mixing gain to a subset (or all) of the speaker regions. This allows for the creation of a dynamic list of all objects contributing to each speaker. In some implementations, this list can be ordered by decreasing energy levels, for example, using the product of the original root mean square (RMS) levels of the signal multiplied by the mixing gain. In other implementations, the list can be ordered according to other criteria, such as the relative importance assigned to the audio objects.

[0138] During the presentation process, if a load is detected at the output of a particular regenerating speaker, the energy of the audio object can be distributed across several regenerating speakers. For example, the energy of the audio object can be distributed using a width or distribution factor, which is proportional to the load and the relative contribution of each audio object to a particular regenerating speaker. If the same audio object contributes to several load regenerating speakers, its width or distribution factor can be further increased in some implementations and applied to the presentation frame of the next audio data.

[0139] Generally, a hard limiter clips any value above a threshold. In the example above, if a speaker receives a mixed element at level 1.25 and only allows a maximum level of 1.0, the element will be "hard-limited" to 1.0. A soft limiter begins applying limitation before reaching an absolute threshold to provide a smoother, more satisfying sound. Soft limiters can also use a "look-forward" feature to predict when future clipping will occur, smoothly reducing gain before clipping happens, thus avoiding clipping altogether.

[0140] The various "smearing variations" implemented here can be used in conjunction with hard or soft limiters to limit auditory distortion while avoiding a decrease in spatial accuracy / clarity. When opposing overall deployment or using limiters individually, smearing variations can selectively pick out loud objects or objects of specific content types. These implementations can be controlled by a mixer. For example, if the speaker region limitation metadata for audio objects indicates that a subset of reproducing speakers should not be used, the presentation device can apply the corresponding speaker region limitation rules in addition to implementing smearing variations.

[0141] Figure 16 is a flowchart outlining the process of applying smear changes to an audio object. Process 1600 begins at block 1605, where one or more instructions are received to initiate the audio object smear change function. Instructions may be received by the logic system of the presentation device and may correspond to input received from a user input device. In some implementations, the instructions may include the user's selection of a playback environment configuration. In alternative implementations, the user may pre-select a playback environment configuration.

[0142] In block 1607, audio playback data (including one or more audio objects and associated metadata) is received. In some implementations, the metadata may include, for example, speaker area limitation metadata as described above. In this example, in block 1610, audio object location, time, and deployment data are analyzed from the audio playback data (or received otherwise, for example, through input from a user interface).

[0143] The playback speaker response is determined by using a positioning equation for audio object data (as described above) to configure the playback environment (block 1612). Block 1615 displays the audio object position and the playback speaker response (block 1615). The playback speaker response can also be configured to communicate with the logic system for speaker playback.

[0144] In block 1620, the logic system determines whether a load is detected on any regenerating speaker in the regenerating environment. If so, the audio object smearing variation law described above can be applied until no load is detected (block 1625). In block 1630, the audio data output can be stored (if desired) and output to the regenerating speaker.

[0145] In block 1635, the logic system can determine whether process 1600 should continue. If, for example, the logic system receives an instruction from the user that they want to continue, process 1600 can continue. For example, process 1600 can continue by returning to block 1607 or block 1610. Otherwise, process 1600 can end (block 1640).

[0146] Some implementations propose extended positioning gain equations that can be used to image the position of audio objects in a three-dimensional control room. Some examples will now be illustrated with reference to Figures 17A and 17B. Figures 17A and 17B show examples of audio objects positioned in a three-dimensional virtual playback environment. Referring first to Figure 17A, the position of audio object 505 can be seen within the virtual playback environment 404. In this example, speaker regions 1-7 are located on the same plane, while speaker regions 8 and 9 are located on another plane, as shown in Figure 17B. However, the number of speaker regions, planes, etc., is merely illustrative; the concepts described herein can be extended to different numbers of speaker regions (or individual speakers) and more than two height planes.

[0147] In this example, the height parameter "z", ranging from zero to 1, maps the position of the audio object to the height plane. In this example, the value z=0 corresponds to the base plane including speaker regions 1-7, while the value z=1 corresponds to the upper plane including speaker regions 8 and 9. The value e between zero and 1 corresponds to the blending of the sound image produced by speakers used only on the base plane and the sound image produced by speakers used only on the upper plane.

[0148] In the example shown in Figure 17B, the height parameter for audio object 505 has a value of 0.6. Therefore, in one implementation, a first sound image can be generated using the positioning equation for the base plane based on the (x, y) coordinates of audio object 505 in the base plane. A second sound image can be generated using the positioning equation for the upper plane based on the (x, y) coordinates of audio object 505 in the upper plane. Based on the planes adjacent to audio object 505, the first and second sound images can be combined to generate a resulting sound image. The energy or amplitude conservation function of height z can be utilized. For example, assuming that the range of z can be from zero to one, the gain value of the first sound image can be multiplied by Cos(z * π / 2) and the gain value of the second sound image can be multiplied by sin(z * π / 2), such that the sum of their squares is 1 (energy conservation).

[0149] Other implementations described herein may include calculating gain based on two or more positioning techniques and generating ensemble gain based on one or more parameters. Parameters may include one or more of the following: the desired audio object position; the distance from the desired audio object position to a reference position; the speed or rate of the audio object; or the audio object content type.

[0150] Figure 18 will now be used to illustrate some of these practices. Figure 18 shows examples of regions that conform to different positioning methods. The size, shape, and extent of these regions are merely examples. In this example, the near-field positioning method is applicable to audio objects located within region 1805, while the far-field positioning method is applicable to audio objects located within region 1815 (outside region 1810).

[0151] Figures 19A-19D illustrate examples of applying near-field and far-field localization techniques to audio objects in different locations. Referring first to Figure 19A, the audio object is essentially outside the virtual reproduction environment 1900. This location corresponds to region 1815 in Figure 18. Therefore, one or more far-field localization methods will be used in this example. In some implementations, the far-field localization method is based on the Vector Basis Amplitude Localization (VBAP) equation, which is generally known to those skilled in the art. For example, the far-field localization method may be based on the VBAP equation described in paragraph 2.3, page 4 of V. Pulkki, Compensating Displacement of Amplitude-Panned Virtual Sources (AES International Conference on Virtual, Synthetic and Entertainment Audio), which is incorporated herein by reference. In alternative implementations, other methods can be used to localize far-field and near-field audio objects, including methods that synthesize corresponding auditory plane or spherical waveforms. Related methods are described in D. de Vries, Wave Field Synthesis (AES Monograph 1999), which is incorporated herein by reference.

[0152] Referring now to Figure 19B, the audio object is located within the virtual reproduction environment 1900. This location corresponds to region 1805 in Figure 18. Therefore, one or more near-field localization methods will be employed in this example. Some of the aforementioned near-field localization methods will utilize speaker regions surrounding the audio object 505 within the virtual reproduction environment 1900.

[0153] In some implementations, near-field localization methods may include "dual-balance" localization and combining two sets of gains. In the example shown in Figure 19B, the first set of gains corresponds to the front / back balance between two sets of speaker regions surrounding the audio object 505 along the y-axis. The corresponding response includes all speaker regions of the virtual reproduction environment 1900, except for speaker regions 1915 and 1960.

[0154] In the example shown in Figure 19C, the second set of gains corresponds to the left / right balance between two sets of speaker regions surrounding the audio object 505 along the x-axis. The corresponding response includes speaker regions 1905 to 1925. Figure 19D shows the result of combining the responses shown in Figures 19B and 19C.

[0155] When an audio object enters or leaves the virtual reproduction environment 1900, it may be desirable to blend different localization methods. Therefore, a blend of the gains calculated using near-field and far-field localization methods will be applied to the audio object located in region 1810 (see Figure 18). In some implementations, pairwise localization rules (e.g., energy conservation sine or dynamic laws) can be used to blend the gains calculated using near-field and far-field localization methods. In alternative implementations, the pairwise localization rule may be amplitude conservation rather than energy conservation, such that the sum equals one rather than the sum of squares equals one. It is also possible to blend generated processed signals, for example, by processing the audio signals independently using two localization methods and cross-fading the two generated audio signals.

[0156] It may be desirable to propose a mechanism that allows content creators and / or content re-renderers to easily fine-tune different re-renders for specific editing tracks. In the context of moving image blending, it is important to consider the concept of screen spatial energy balance. In some cases, the automatic re-rendering of a specific audio track (or "disc") will result in different screen spatial balances depending on the number of re-rendering speakers in the re-rendering environment. According to some implementations, screen spatial offset can be controlled based on metadata generated during the editing process. According to alternative implementations, screen spatial offset can be controlled only at the rendering end (i.e., under the control of the content re-renderer) and not reflected in the metadata.

[0157] Therefore, some of the implementations described herein propose one or more forms of screen-to-space offset control. In some such implementations, screen-to-space offset can be implemented as a scaling operation. For example, a scaling operation may include scaling the original intended track of an audio object along a front-to-back direction and / or scaling the speaker position used in the renderer to determine positioning gain. In some such implementations, screen-to-space offset control can be a variable value between zero and a maximum value (e.g., 1). The degree of variation can be controlled, for example, by a GUI, virtual or physical sliders, knobs, etc.

[0158] Alternatively or additionally, screen spatial offset control can be implemented using some form of speaker region limitation. Figure 20 illustrates the speaker regions of the replay environment that can be used during screen spatial offset control. In this example, a front speaker region 2005 and a rear speaker region 2010 (or 2015) can be established. The screen spatial offset can be adjusted as a function of the selected speaker regions. In some such implementations, the screen spatial offset can be implemented as a scaling operation between the front speaker region 2005 and the rear speaker region 2010 (or 2015). In alternative implementations, the screen spatial offset can be implemented in a binary form, for example, by allowing the user to select front offset, rear offset, or no offset. The offset setting for each case can conform to a predetermined (usually non-zero) offset degree for the front speaker region 2005 and the rear speaker region 2010 (or 2015). Essentially, the above implementations can provide three presets for screen spatial offset control, instead of (or additionally) continuous value scaling operations.

[0159] According to some implementations of this type, two additional logical speaker regions can be generated in the editing GUI (e.g., 400) by dividing the sidewalls into front and rear sidewalls. In some implementations, the two additional logical speaker regions correspond to the left wall / left surround sound area and the right wall / right surround sound area of ​​the renderer. Depending on the user's selection of these two logical speaker regions as valid, the rendering tool applies a default scaling factor (e.g., as described above) to the Dolby 5.1 or Dolby 7.1 configuration when rendering. The rendering tool can also apply the aforementioned default scaling factor when rendering in playback environments that do not support defining these two additional logical regions, for example, because their physical speaker configuration on the sidewalls only has one physical speaker.

[0160] Figure 21 is a block diagram illustrating an example of components for setting up an editing and / or presentation device. In this example, device 2100 includes interface system 2105. Interface system 2105 may include a network interface, such as a wireless network interface. Alternatively or additionally, interface system 2105 may include a Universal Serial Bus (USB) interface or other such interfaces.

[0161] Device 2100 includes logic system 2110. Logic system 2110 may include a processor, such as a general-purpose single-chip or multi-chip processor. Logic system 2110 may include a digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, or discrete hardware components, or combinations thereof. Logic system 2110 may be configured to control other components of device 2100. Although interfaces are not shown between the components of device 2100 in Figure 21, logic system 2110 may be equipped with interfaces for communicating with other components. Other components may or may not be configured to communicate with each other.

[0162] Logic system 2110 may be configured to perform audio editing and / or presentation functions, including but not limited to the types of audio editing and / or presentation functions described herein. In some such implementations, logic system 2110 may be configured to (at least in part) operate one or more non-transitory media based on stored software. Non-transitory media may include memory associated with logic system 2110, such as random access memory (RAM) and / or read-only memory (ROM). Non-transitory media may include the memory of memory system 2115. Memory system 2115 may include one or more suitable types of non-transitory storage media, such as flash memory, hard disks, etc.

[0163] The display system 2130 may include one or more suitable types of displays depending on the performance of the device 2100. For example, the display system 2130 may include a liquid crystal display, a plasma display, a bistable display, etc.

[0164] User input system 2135 may include one or more devices configured to accept input from a user. In some implementations, user input system 2135 may include a touch screen overlaid on the display of display system 2130. User input system 2135 may include a mouse, trackball, gesture detection system, joystick, one or more GUIs and / or menus displayed on display system 2130, buttons, keyboard, switches, etc. In some implementations, user input system 2135 may include a loudspeaker 2125 through which a user can provide voice commands to device 2100. Logic system may be configured to recognize voice and use it to control at least some operations of device 2100 based on the aforementioned voice commands.

[0165] The power system 2140 may include one or more suitable energy storage devices, such as nickel-cadmium batteries or lithium batteries. The power system 2140 may be configured to receive power from a power outlet.

[0166] Figure 22A is a block diagram showing some components that can be used to generate audio content. System 2200 can be used, for example, to generate audio content in a mixing studio and / or during the mixing stage. In this example, system 2200 includes audio and metadata editing tools 2205 and rendering tools 2210. In this implementation, audio and metadata editing tools 2205 and rendering tools 2210 respectively include audio connection interfaces 2207 and 2212, which can be configured to communicate via AES / EBU, MADI, analog, etc. Audio and metadata editing tools 2205 and rendering tools 2210 respectively include network interfaces 2209 and 2217, which can be configured to transmit and receive metadata via TCP / IP or other suitable protocols. Interface 2220 is configured to output audio data to speakers.

[0167] System 2200 may include, for example, an existing editing system, such as the Pro Tools™ system, executing a metadata generation tool (i.e., a camcorder as described herein) as a plug-in. The camcorder may also run on a standalone computer system (e.g., a PC or mixer) connected to the rendering tool 2210, or it may run on the same physical device as the rendering tool 2210. In subsequent examples, the camcorder and the rendering device will use locale connectivity, for example, through shared memory. The camcorder GUI can also be remotely controlled from a tablet, laptop, etc. The rendering tool 2210 may include a rendering system, which includes an audio processor configured to execute the rendering software. The rendering system may include, for example, a personal computer, laptop, etc., which includes an interface for audio input / output and appropriate logic.

[0168] Figure 22B is a block diagram representing some of the components that can be used to replay audio in a playback environment (such as a movie theater). System 2250 in this example includes a theater server 2255 and a presentation system 2260. Theater server 2255 and presentation system 2260 respectively include network interfaces 2257 and 2262, which are configured to transmit and receive audio objects via TCP / IP or any other suitable protocol. Interface 2264 is configured to output audio data to speakers.

[0169] Those skilled in the art will readily understand the various modifications to the practices described in this disclosure. The general principles defined herein can be applied to other practices without departing from the spirit and scope of this disclosure. Therefore, the scope of the patent application is not intended to be limited to the practices shown herein, but rather to the broadest range consistent with the disclosure, principles, and novel features described herein.

[0170] 100: Regenerative Environment 105: Projector 110: Audio Processor 115: Power Amplifier 120: Left Surround Array 125: Right-Surround Array 130: Left Screen Audio Channel 135: Central Screen Audio Channel 140: Right Screen Audio Channel 145: Subwoofer 150: Screen 200: Regenerative Environment 205: Digital Projector 210: Audio Processor 215: Power Amplifier 220: Left-side surround array 224: Left rear surround speaker 225: Right-side surround array 226: Right rear surround speaker 230: Left Screen Audio Channel 235: Central Screen Audio Channel 240: Right Screen Audio Channel 245: Subwoofer 300: Recycling Environment 310: Upper speaker layer 320: Intermediate speaker layer 330: Lower speaker layer 345a: Subwoofer 345b: Subwoofer 400: Graphical User Interface 402a: Speaker Area 402b: Speaker area 404: Virtual Regeneration Environment 405: Front Area 410: Left region 412: Left rear area 414: Right rear area 415: Right Region 420a: Upper Region 420b: Upper Region 450: Recycling Environment 455: Screen Speaker 460: Left-side surround array 465: Right-side surround array 470a: Top left speaker 470b: Top right speaker 480a: Left rear surround speaker 480b: Right rear surround speaker 505: Audio Objects 510: Vernier 515a: Two-dimensional surface 515b: Two-dimensional surface 520: Virtual Ceiling 805a: Virtual Speaker 805b: Virtual Speaker 810: Broken Line 905: Virtual Rope 1105: Line 1-9: Speaker Area 1300: Graphical User Interface 1305: Image 1310: Shaft 1320: Speaker Layout 1324-1340: Speaker location 1345: Three-dimensional depiction 1350: Area 1505: Ellipsoid 1507: Distribution Data Chart 1510: Curve 1520: Curve 1512: Sample 1515: Circle 1805: Region 1810: Region 1815: Region 1900: Virtual Regeneration Environment 1905-1960: Loudspeaker Area 2005: Front speaker area 2010: Rear speaker area 2015: Rear speaker area 2100: Device 2105: Interface System 2110: Logical System 2115: Memory System 2120: Speaker 2125: Megaphone 2130: Display System 2135: User Input System 2140: Power System 2200: System 2205: Audio and Metadata Editing Tools 2210: Presentation Tools 2207: Audio Connection Interface 2212: Audio Connection Interface 2209: Network Interface 2217: Network Interface 2220: Interface 2250: System 2255: Theater Server 2260: Presentation System 2257: Network Interface 2262: Network Interface 2264: Interface

Claims

1. A method for presentation, comprising: receiving audio playback data including one or more audio objects and metadata associated with each of the one or more audio objects; receiving playback environment data including an indication of the number of playback loudspeakers in the playback environment and an indication of the location of each playback loudspeaker in the playback environment; and presenting the audio objects as one or more loudspeaker feedback signals by applying an amplitude localization procedure to each audio object, wherein the amplitude localization procedure includes: determining a scaling factor corresponding to each playback loudspeaker for the audio object based at least in part on the metadata associated with each audio object and the location of each playback loudspeaker in the playback environment, and wherein each loudspeaker feedback signal corresponds to at least one of the playback loudspeakers in the playback environment; wherein the metadata associated with each audio object includes audio object coordinates indicating the expected playback location of the audio object in the playback environment, and includes a snapshot flag indicating whether the amplitude localization procedure should present the audio object as a single loudspeaker feedback signal or apply a localization rule to present the audio object as a plurality of loudspeaker feedback signals.

2. The method as described in request item 1, wherein: The snapshot flag indicates whether the amplitude localization procedure should present the audio object as a single speaker feedback signal; and the amplitude localization procedure presents the audio object as a speaker feedback signal corresponding to the reproducing speaker at the expected reproduction position closest to the audio object.

3. The method as described in request item 1, wherein: The snapshot flag indicates whether the amplitude positioning procedure should present the audio object as a single speaker feedback signal when the distance between the expected reproduction location of the audio object and the reproducing speaker closest to the expected reproduction location of the audio object exceeds a critical value. The amplitude localization process overwrites the snapshot flag and uses a localization rule to present the audio object as multiple speaker feedback signals.

4. The method as described in request item 2, wherein: The metadata is time-varying; the coordinates of the audio object indicate that the expected playback location of the audio object in the playback environment is different at the first time point and the second time point; At that first point in time, the reproducing speaker that is closest to the intended reproduction position of the audio object corresponds to the first reproducing speaker; At the second time point, the regenerating speaker closest to the intended regeneration position of the audio object corresponds to the second regenerating speaker; and the amplitude positioning procedure smoothly transitions between presenting the audio object as a first speaker feedback signal corresponding to the first regenerating speaker and presenting the audio object as a second speaker feedback signal corresponding to the second regenerating speaker.

5. The method as described in request item 1, wherein: The metadata is time-varying; at a first time point, the snapshot flag indicates that the amplitude localization procedure should present the audio object as a single speaker feedback signal; at a second time point, the snapshot flag indicates that the amplitude localization procedure should apply a localization rule to present the audio object as multiple speaker feedback signals; and the amplitude localization procedure smoothly transitions between presenting the audio object as a speaker feedback signal corresponding to the regenerated speaker closest to the expected regeneration position of the audio object and applying a localization rule to present the audio object as multiple speaker feedback signals.

6. A presentation device comprising: an interface system; and a logic system configured to: receive audio playback data via the interface system, the data comprising one or more audio objects and metadata associated with each of the one or more audio objects; receive playback environment data via the interface system, the data comprising an indication of the number of playback speakers in the playback environment and an indication of the location of each playback speaker in the playback environment; and present the audio objects as one or more speaker feedback signals by applying an amplitude localization procedure to each audio object, wherein the amplitude localization procedure comprises: determining a scaling factor corresponding to each playback speaker for the audio object based at least in part on the metadata associated with each audio object and the location of each playback speaker in the playback environment, and wherein each speaker feedback signal corresponds to at least one of the playback speakers in the playback environment; The metadata associated with each audio object includes audio object coordinates, which indicate the expected playback position of the audio object within the playback environment, and snapshot flags, which indicate whether the amplitude localization procedure should present the audio object as a single speaker feedback signal or apply a localization rule to present the audio object as multiple speaker feedback signals.

7. The device as claimed in claim 6, wherein: The snapshot flag indicates whether the amplitude localization procedure should present the audio object as a single speaker feedback signal; and the amplitude localization procedure presents the audio object as a speaker feedback signal corresponding to the reproducing speaker at the expected reproduction position closest to the audio object.

8. The device as claimed in claim 6, wherein: The snapshot flag indicates whether the amplitude positioning procedure should present the audio object as a single speaker feedback signal when the distance between the expected reproduction location of the audio object and the reproducing speaker closest to the expected reproduction location of the audio object exceeds a critical value. The amplitude localization process overwrites the snapshot flag and uses a localization rule to present the audio object as multiple speaker feedback signals.

9. The device as claimed in claim 6, wherein: The metadata is time-varying; the coordinates of the audio object indicate that the expected playback location of the audio object in the playback environment is different at the first time point and the second time point; At that first point in time, the reproducing speaker that is closest to the intended reproduction position of the audio object corresponds to the first reproducing speaker; At the second time point, the regenerating speaker closest to the intended regeneration position of the audio object corresponds to the second regenerating speaker; and the amplitude positioning procedure smoothly transitions between presenting the audio object as a first speaker feedback signal corresponding to the first regenerating speaker and presenting the audio object as a second speaker feedback signal corresponding to the second regenerating speaker.

10. The device as claimed in claim 6, wherein: The metadata is time-varying; at a first time point, the snapshot flag indicates that the amplitude localization procedure should present the audio object as a single speaker feedback signal; at a second time point, the snapshot flag indicates that the amplitude localization procedure should apply a localization rule to present the audio object as multiple speaker feedback signals; and the amplitude localization procedure smoothly transitions between presenting the audio object as a speaker feedback signal corresponding to the regenerated speaker closest to the expected regeneration position of the audio object and applying a localization rule to present the audio object as multiple speaker feedback signals.

11. A non-transitory computer-readable medium having software stored thereon, the software including instructions for causing one or more processors to perform the following operations: receiving audio playback data comprising one or more audio objects and metadata associated with each of the one or more audio objects; receiving playback environment data comprising an indication of the number of playback loudspeakers in the playback environment and an indication of the location of each playback loudspeaker in the playback environment; and presenting the audio objects as one or more loudspeaker feedback signals by applying an amplitude localization procedure to each audio object, wherein the amplitude localization procedure comprises: determining a scaling factor corresponding to each playback loudspeaker for the audio object based at least in part on the metadata associated with each audio object and the location of each playback loudspeaker in the playback environment, and wherein each loudspeaker feedback signal corresponds to at least one of the playback loudspeakers in the playback environment; The metadata associated with each audio object includes audio object coordinates, which indicate the expected playback position of the audio object within the playback environment, and snapshot flags, which indicate whether the amplitude localization procedure should present the audio object as a single speaker feedback signal or apply a localization rule to present the audio object as multiple speaker feedback signals.

Images